Contextual text lookup for images

Abstract

The subject technology provides contextual text lookup for an image. When an electronic device receives a request to perform a lookup or search for text in an image displayed at the electronic device, the electronic device may obtain one or more search results based on the text itself and based on contextual information derived by the electronic device from the image. In one or more implementations, application information associated with an application displaying the image may also be used as contextual metadata to enhance the results of the search for the text from the image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 336,987, filed on April 29, 2022, entitled “Contextual Text Lookup for Images,” the disclosure of which is hereby incorporated herein in its entirety. Technical Field

[0003] This specification generally relates to machine learning, including, for example, using machine learning for contextual text lookup for images. Background Art

[0004] Conventional search engines are configured to perform searches for text strings that are typically entered by a user into a browser application at an end user's device. For example, a user of an electronic device viewing a product name may open a browser application, type the product name into the browser application, and submit the typed text to a search engine for search via the browser application. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following figures.

[0006] Figure 1 An example network environment is illustrated in accordance with one or more implementations of the subject technology.

[0007] Figure 2 An example system is illustrated in accordance with one or more implementations of the subject technology.

[0008] Figure 3 An example of an electronic device displaying an image according to one or more implementations of the subject technology is illustrated.

[0009] Figure 4 An example of an electronic device displaying context-based search results for image text in accordance with one or more implementations of the subject technology is illustrated.

[0010] Figure 5 A flow chart illustrating an example process that may be performed by an electronic device for contextual text lookup for an image in accordance with one or more implementations of the subject technology.

[0011] Figure 6A flow chart illustrating another example process that may be performed by an electronic device for contextual text lookup for an image in accordance with one or more implementations of the subject technology.

[0012] Figure 7 A flow chart illustrating an example process that may be performed by a server for contextual text lookup for an image in accordance with one or more implementations of the subject technology.

[0013] Figure 8 Example electronic systems are illustrated that can be used to implement various aspects of the subject technology according to one or more implementations. DETAILED DESCRIPTION

[0014] The detailed description set forth below is intended to be a description of various configurations of the subject technology and is not intended to represent the only configuration by which the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0015] An electronic device is provided that can recognize text in an image displayed by an electronic device and provide a user with an option to select or otherwise interact with the recognized text in the image. In one or more specific implementations, a user can request to search (e.g., find) some or all of the text in the text of the displayed image.

[0016] According to one or more specific implementations, the subject technology provides improved text lookup for text identified in an image displayed by an electronic device by using other contextual information in and / or associated with the image to enhance search results. For example, the contextual information can be derived from the image and can include other text in the image or on the screen, object types of objects in the image or on the screen, one or more embeddings of portions of the image, application information of an application displaying the image, location information of the image and / or the device displaying the image, or any other information that can be extracted from the image or derived from the image.

[0017] According to various implementations, the context information may be used to locally rank the server-provided search results at the device displaying the image, or the context information may be transmitted to the server along with the selected text to enhance the search results from the server. In one or more implementations, the server-provided search results may also be displayed along with locally generated dictionary results (e.g., dictionary entries for words in the selected text).

[0018] Figure 1 An example network environment 100 is illustrated according to one or more implementations of the subject technology. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0019] The network environment 100 includes a computing device 110 (also referred to herein as an electronic device) and a server 120. A network 106 may communicatively couple (directly or indirectly) the computing device 110 and / or the server 120. In one or more specific implementations, the network 106 may be an interconnected network that may include the Internet or devices communicatively coupled to the Internet. For purposes of explanation, the network environment 100 is described in detail in FIG. Figure 1 1 and 120; however, the network environment 100 may include any number of electronic devices and any number of servers.

[0020] The computing device 110 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, a peripheral device (e.g., a digital camera, a headset), a tablet device, a wearable device such as a watch, a band, etc. Figure 1 In the present invention, by way of example, computing device 110 is depicted as a smart phone. Electronic device 110 may be Figure 2 and / or Figure 8 The system discussed, and / or may include all or a portion of the system.

[0021] In one or more specific implementations, computing device 110 may provide a system for training a machine learning model using training data, wherein the trained machine learning model is subsequently deployed locally on computing device 110. In addition, computing device 110 may provide one or more frameworks for training machine learning models and / or developing applications using such machine learning models. In an example, computing device 110 may be a user device (e.g., a smart phone, a tablet device, a laptop computer, a desktop computer, a wearable electronic device, etc.) that displays an image including selectable and / or searchable text. In one or more specific implementations as described herein, computing device 110 may communicate with server 120 (e.g., a backend server or a search server), such as to obtain search results (e.g., context-based search results) for text in an image displayed by computing device 110.

[0022] In one implementation, the server 120 may train one or more machine learning models for deployment to client electronic devices (e.g., computing device 110). In other implementations, the server 120 may provide a system for training a machine learning model using training data, where the trained machine learning model is then deployed locally on the server 120. The machine learning model that may be deployed on the server 120 and / or computing device 110 may then execute one or more machine learning algorithms. In one or more implementations, the server 120 may provide a cloud service that utilizes the trained machine learning model and improves over time. The server 120 may be a server described below with respect to Figure 2 and / or Figure 8 The system discussed, and / or may include all or a portion of the system.

[0023] Figure 2 An example system 200 is illustrated in accordance with one or more specific implementations of the subject technology. In an example, the system 200 can be implemented in a computing device, such as computing device 110. In another example, the system 200 can be implemented in a single device or in multiple devices in a distributed manner, and the specific implementation of the system will be apparent to those skilled in the art.

[0024] In an example, system 200 may include processor 202, memory 204 (memory device), and communication unit 210. Memory 204 may store data 206 and one or more machine learning models 208. In an example, system 200 may include or be communicatively coupled to storage device 212. Thus, storage device 212 may be an internal storage device or an external storage device. Figure 2 In the example of FIG. 2 , the system 200 includes one or more cameras 211, a display 214, and one or more sensors 216. The camera 211 is operable to capture an image and may be mounted on Figure 1 The display 214 may be operable to display images captured by the camera 211 and / or received from another device or system and stored in the storage device 212 or the memory 204. The displayed images may include captured still images, real-time preview image frames from the camera 211, video frames, or any other digital images. The sensor 216 may include a position sensor (e.g., a satellite positioning system sensor), a motion sensor (e.g., an inertial sensor), and / or a depth sensor (e.g., a stereo camera, a LIDAR sensor, a radar sensor, a time-of-flight sensor, etc.).

[0025] In an example, the processor 202 can be a single processing unit or multiple processing units. The processor 202 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. In other capabilities, the processor 202 is configured to retrieve and execute computer-readable instructions and data stored in the memory 204.

[0026] Memory 204 may include any non-transitory computer-readable medium known in the art, including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and / or non-volatile memory, such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disks, optical disks, and magnetic tapes.

[0027] Data 206 may represent, among other things, a repository of data processed, received, and generated by one or more processors, such as processor 202. One or more of the aforementioned components of system 200 may transmit or receive data, for example, using one or more input / output ports and one or more communication units.

[0028] In an example, the machine learning model 208 may include one or more of a machine learning-based model and an artificial intelligence-based model, such as, for example, a neural network or any other model and / or machine learning architecture. In an example, the machine learning model 208 may be trained using training data (e.g., training data included in the data 206 or other data) and may be implemented by the processor 202 to perform one or more of the operations described herein.

[0029] In an example, communication unit 210 may include one or more hardware units that enable wired or wireless communications between processor 202 and processors of other computing devices.

[0030] In an example, an image may be displayed by a computing device 110 implementing system 200. The image may be stored in storage 212, memory 204, and / or may be received from a remote device or server. The image may be displayed in an image viewing application of computing device 110. In another example, the image may be displayed by a browser application, a social media application, a digital media player application that can display the image. In another example, computing device 110 may display a real-time preview of a field of view as captured by a camera of computing device 110. In accordance with a specific implementation of the subject technology, processor 202 may be configured to obtain an image being displayed by an application running on computing device 110. For example, processor 202 may determine that an image is being displayed or will be displayed by an application running at a computing device, and may provide the image and / or a portion thereof to one or more machine learning models in machine learning model 208.

[0031] The machine learning models 208 may be trained to identify one or more elements of interest in text and / or images. For example, one or more of the machine learning models may be configured to recognize text in an image displayed on the display 214. For example, one or more of the machine learning models 208 may receive an image as input and then output object types for objects in the image and / or may output one or more embeddings of portions of the image. The processor 202 may also obtain other contextual information for the image, such as application information indicating an application that is displaying the image, and / or location information of the image and / or the computing device 110 and / or the system 200.

[0032] In one or more specific implementations, the processor 202 (e.g., using the machine learning model 208) may identify elements of interest in the image. As an example, a smart camera model may be implemented to detect any text in the image (if present), an object detector may be implemented for identifying and / or classifying objects present in the image, and / or a gating model (also referred to herein as a coarse classification model) may be implemented to classify objects present in the image. In yet another example, a scene classification model may be implemented to detect and classify the overall scene depicted in the image. Thus, by implementing and / or executing the machine learning model 208, the processor 202 may derive contextual information from the image, such as by determining various types of elements of interest in the image and / or extracting features, information, and / or other signals from the image.

[0033] Figure 3An example is illustrated in which computing device 110 displays image 300. In this example, image 300 is displayed by an application running on computing device 110, and an application identifier (ID) 308 for the display application is displayed together with the image on display 302 of computing device 110. For example, in one or more specific implementations, Figure 3 The display 302 may be Figure 2 A specific implementation of the display 214 of the system 200. Figure 3 In the example of , additional information is also displayed on display 302 of computing device 110 along with image 300. In this example, the additional information includes application controls 310 for the application displaying image 300, current time 304, and location indicator 306 (e.g., such as to indicate that the current location of computing device 110 is known based on sensor data from sensor 216). For example, application control 310 may be a virtual button or other interactive feature of the application displaying image 300 (e.g., a control button or interactive feature for controlling a browser application, an image display application, a social media application, a camera application, a media playback application, etc.).

[0034] exist Figure 3 In the example of , image 300 includes image text 312, image text 318, and objects such as foreground object 314 and background object 316. As described herein, for example Figure 2 As discussed, computing device 110 may identify object types (e.g., classifications) of objects (e.g., foreground objects 314 and / or background objects 316) in image 300. Computing device 110 may also recognize image text 312 and image text 318, and modify the display of image 300 to make image text 312 and / or image text 318 selectable and / or searchable. For example, once computing device 110 makes image text 312 and image text 318 selectable, a user may tap or touch the location of the displayed text in image 300, causing a selection tool or highlighter to appear for selecting the text displayed in image 300. In this example, once image text 312 and / or image text 318 are highlighted or otherwise selected, the user may tap or “right-click” again on the selected text to appear options such as a search or find option, causing computing device 110 to obtain search results for the selected text.

[0035] As an example, in one or more implementations, a user of computing device 110 may use a finger, cursor, or other input mechanism to interact with image 300 to select image text 312, and may initiate a search for the selected image text 312. In another example, the user may see image text 312 in a displayed image 300 (or otherwise be provided with information indicating the presence of image text 312), and use voice input to a virtual assistant application running on computing device 110 to request a search for image text 312 included in the displayed image 300.

[0036] In one illustrative example, image 300 may be an image of a storefront, and image 300 may include image text 312 indicating the name of the store. However, a search for only the image text indicating the name of the store may return search results that are not relevant to the store. For example, an image of a restaurant named "Butterfly" may be displayed on a user's smartphone, and the user may request a search for the text "Butterfly" displayed in image 300. However, since "Butterfly" is a term not typically associated with restaurants, the search results may not be relevant to the desired search results for the text from the image.

[0037] In one or more specific implementations, the subject technology provides improved text lookup or search for text identified in an image by using other contextual information in the image to enhance search results. For example, contextual information may include other text (e.g., unselected and / or unsearched text, such as image text 318) in image 300 or elsewhere on display 302 (e.g., text associated with application ID 308, and / or text associated with application control 310), object type of objects in image 300 or on display 302 (e.g., foreground object 314 and / or background object 316), one or more embeddings of portions of image 300, application information for an application used to display image 300 (e.g., application identifier 308 or application type), location information of image 300 and / or computing device 110, and the like.

[0038] In the previous example of an image of a restaurant named "Butterfly," computing device 110 may identify one or more objects in the image, such as food plates, menus, tables and chairs, doors or windows, or other objects indicative of a restaurant, may identify relative depths of objects in the image, relative distances between objects and / or text in the image, may identify other text in the image (e.g., another word such as "restaurant," "bistro," "cafe," etc.) Computing device 110 may then initiate an enhanced search for the image text "Butterfly" by including some or all of the derived contextual information in the search request and / or in the ranking of search results obtained without the contextual information.

[0039] It should be understood that the example in which image 300 is an image of a storefront from a restaurant is merely illustrative. In various specific implementations, image 300 can be any stored or real-time preview image that includes text in the image context. As another illustrative example, image 300 can be a rendered user interface of a media playback application, image text 312 can be the song title of a song played back by the media playback application, foreground object 314 can be an album cover art image, and image text 318 can be an artist name and / or album title. In this example, the search for song names can be enhanced by using contextual information (such as album title, artist name and / or album art) derived from image 300. In one or more use cases, information associated with the application displaying image 300 can also be useful contextual information for enhancing the search for image text 312. For example, in an example in which image 300 includes a rendered user interface of a media playback application, the search for the selected text corresponding to the song title can be enhanced by including information indicating the media playback application in the contextual information of the notification search.

[0040] This example of image text 312 being a song title may particularly illustrate the enhancement provided by including contextual information in a text search when considering that a song title may be "Butterfly" (just as a restaurant name may be "Butterfly"). In this example, by including contextual information indicating a media playback application or indicating an embedding of an artist name, album name, cover art image, and / or an object type displayed in an album cover art in the search request and / or in the sorting or re-ranking of search results, the search results obtained may be related to the song "Butterfly" rather than to the restaurant "Butterfly" or the insect "Butterfly".

[0041] In one or more specific implementations, depth information may also be derived from image 300 and used as contextual information for searching for image text 312. Depth information may be obtained using a depth sensor (e.g., depth sensor 216) of computing device 110 while capturing image 300, or may be derived from image 300 itself (e.g., using computer vision and / or other machine learning techniques to identify the relative depths of objects in the image). As an example, depth information derived for and / or from image 300 may be used to determine foreground object 314 as a foreground object and / or background object 316 as a background object. In one or more specific implementations, foreground objects and / or objects closer to the image text 312 being searched for may be weighted more heavily in assisting the search for image text 312 than background objects or objects that are relatively far from the image text 312 being searched for in image 300.

[0042] According to various implementations, the context information may be used to locally rank server search results for the selected text at computing device 110, and / or some or all of the context information may be transmitted to a server (e.g., a search server such as server 120) along with image text 312 to enhance search results from the server. In one or more implementations, server-obtained search results may also be displayed along with locally generated dictionary results.

[0043] For example, Figure 4 An example is illustrated in which search results for image text 312 are presented by computing device 110. Figure 4 In the example of , the search results for the image text 312 include local results 400, such as locally generated dictionary results obtained by searching the image text 312 in a local dictionary stored at the computing device 110 and obtaining dictionary definitions, synonyms and antonyms, or other dictionary entries for the image text 312 (e.g., dictionary entries obtained without using contextual information). Figure 4 In the example of , the search results of the image text 312 also include context-based server results 402. As discussed in further detail herein, the context-based server results 402 may be obtained by: sending a request to a server such as Figure 1 The search results may be obtained by providing the image text 312 and contextual information of the image 300 to a server 120 and receiving search results generated based on both the image text 312 and the contextual information, or by providing only the image text 312 to a search server, receiving text-only based search results from the server, and sorting or re-ranking the text-only based search results using the contextual information (e.g., moving more relevant search results to the top of a presented list of search results).

[0044] Figure 5A flowchart of an example process 500 for performing contextual text lookup for text in an image according to one or more specific implementations is illustrated. For purposes of explanation, this document primarily refers to Figure 1 Process 500 is described with reference to computing device 110 of the embodiment of the present invention. However, process 500 is not limited to computing device 110, and one or more blocks (or operations) of process 500 may be performed by one or more other components and / or other suitable devices. Further for the purpose of explanation, the blocks of process 500 are described herein as occurring sequentially or linearly. However, multiple blocks of process 500 may occur in parallel. In addition, the blocks of process 500 do not have to be performed in the order shown, and / or one or more blocks of process 500 do not have to be performed and / or may be replaced by other operations.

[0045] At block 502, an electronic device (e.g., computing device 110) that is displaying an image (e.g., image 300) may receive a request to perform a search for text in the image (e.g., image text 312). For example, receiving the request may include receiving a selection of text in the image (e.g., a user selection of the text) by the electronic device while the image is being displayed. As another example, receiving the request may include receiving a request from a user via voice input to an application and / or to a voice assistance application running on the device.

[0046] In one or more implementations, the image is a flat image (e.g., an array of pixel values ​​without metadata indicating the content of the image), and process 500 also includes, before receiving the request: detecting, by the electronic device, text in the flat image while the flat image is displayed; and modifying the display of the flat image to display the text as selectable text. In one or more implementations, process 500 may also include obtaining the image from a memory of the electronic device, from a memory of a remote device, or from a camera of the electronic device, and displaying the image using the electronic device (e.g., using a remote device such as Figure 3 In one or more other implementations, the electronic device may detect text in a planar image in response to receiving a user interaction with the image (such as attempting to select text in the planar image).

[0047] At block 504, the electronic device may derive contextual information from the image including the text. In one or more implementations as discussed herein, deriving the contextual information may include determining, by the electronic device (e.g., by providing the image as input to one or more machine learning models in machine learning models 208), labels of objects in the image (e.g., foreground objects 314, background objects 316, and / or any other image objects). In various implementations, the contextual information may be derived from the image prior to receiving a search request or in response to receiving a search request.

[0048] In one or more implementations as discussed herein, deriving contextual information may include obtaining, by the electronic device (e.g., by providing the image as input to one or more of the machine learning models 208), an embedding of a region of interest in the image. In one or more implementations as discussed herein, deriving contextual information may include obtaining, by the electronic device (e.g., by providing the image as input to one or more of the machine learning models 208), unselected text and / or unsearched text (e.g., image text 318 or other text not interacted with by a user in conjunction with a search request) from the image. In one or more implementations as discussed herein, deriving contextual information may include determining, by the electronic device (e.g., by providing the image as input to one or more of the machine learning models 208 and / or by obtaining location information from a location sensor or process at the electronic device), a location associated with the image (e.g., such as the location where the image was captured from location metadata for the image and / or by identifying location-specific information in the image (e.g., a street sign)). In one or more specific implementations as discussed herein, deriving contextual information may include obtaining, by the electronic device (e.g., by providing the image as input to one or more machine learning models in machine learning model 208), depth information associated with the image (e.g., from depth metadata captured using one or more depth sensors (e.g., a depth sensor of sensor 216) when the image was obtained, and / or by inferring relative depths of objects in the image from the image itself). For example, deriving contextual information may include identifying and / or distinguishing one or more foreground objects (e.g., foreground object 314) from one or more background objects (e.g., background object 316).

[0049] At block 506 , the electronic device may obtain one or more search results based on the text and context information in response to the request.

[0050] In one or more specific implementations, obtaining one or more search results includes providing text from an electronic device to a server (e.g., server 120), receiving a ranked set of search results from the server at the electronic device, and re-ranking the ranked set of search results based on contextual information to generate one or more search results for output by the electronic device. For example, the ranked set of search results can be a set of search results that is ranked and / or ordered based on a relevance to the text determined by the server. However, as described herein, the relevance to the text alone may not be consistent with the relevance to the user's desired information about the searched text from the displayed image. Therefore, in one or more specific implementations, contextual information derived from the image can be used to re-rank and / or re-order the set of search results received from the server to place the search results that are most relevant to one or more contextual aspects of the searched text and image at the top of the displayed / output set of search results.

[0051] In one or more specific implementations, obtaining one or more search results may include providing text and context information from the electronic device to a server (e.g., server 120), and receiving one or more search results from the server. In these specific implementations, one or more search results from the server may have been ranked and / or ordered for relevance based on the context information (e.g., without having to perform context-based re-ranking at the electronic device). As discussed herein, in one or more specific implementations, the context information may include application information of an application that displays an image (e.g., an application identifier such as application ID 308, application text associated with an application control such as application control 310, and / or any other information indicating a particular application or application type). In these specific implementations, obtaining one or more search results may include obtaining, by the electronic device, application information of an application that displays an image; and obtaining one or more search results based on the text, context information, and application information.

[0052] In one or more embodiments, when a search request is received from an interface separate from the display displaying the image (e.g., via voice input, such as if a user speaks a request to search image text 312 to computing device 110 while image 300 is displayed on display 302), computing device 110 may determine that the requested search is related to and / or the same as some or all of the text displays in the image, and then, based on that determination, use contextual information derived from the image to obtain search results for the requested search.

[0053] At block 508, the electronic device may provide one or more search results for output by the electronic device. For example, providing one or more search results for output may include providing one or more search results for display on a display of the electronic device (e.g., display 302) (e.g., as described herein in conjunction with Figure 4 described).

[0054] Figure 6 A flowchart of another example process 600 for performing contextual text lookup for text in an image according to one or more specific implementations is illustrated. Figure 1 Process 600 is described with reference to computing device 110 of the embodiment of the present invention. However, process 600 is not limited to electronic device 110, and one or more blocks (or operations) of process 600 may be performed by one or more other components and / or other suitable devices. Further for the purpose of explanation, the blocks of process 600 are described herein as occurring sequentially or linearly. However, multiple blocks of process 600 may occur in parallel. In addition, the blocks of process 600 do not have to be performed in the order shown, and / or one or more blocks of process 600 do not have to be performed and / or may be replaced by other operations.

[0055] At block 602, an electronic device (e.g., computing device 110) may receive a request to perform a search for text (e.g., image text 312) in an image (e.g., image 300) displayed by an application running on the electronic device. For example, receiving the request may include receiving, by the electronic device, a selection of text in the image while displaying the image (e.g., a user selection of the text). As another example, receiving the request may include receiving a request from a user via voice input to an application and / or to a voice-assistance application running on the device.

[0056] At box 604, the electronic device may obtain application information for the application in response to the request. As an example, the application information may include an application identifier such as application ID 308, application text associated with an application control such as application control 310, and / or any other information indicating a particular application or application type. In one illustrative example, the application information may include an application type (e.g., a media player, browser, camera, or other type). In another illustrative example, the application information includes a file type of a file accessed by the application and associated with the image (e.g., an audio file type with an associated album artwork image, or a video file type with an associated cover or poster artwork image). In various specific implementations, the application information may be obtained from the image before or in response to receiving the search request.

[0057] At block 606, the electronic device may obtain one or more search results based on the text and the application information. In one or more specific implementations, obtaining the one or more search results may also include obtaining image context information of the image based on the application information, and obtaining the one or more search results based on the text and the image context information. For example, an image of the album art may not be identified as the album art from the image of the album art alone. However, the electronic device may determine that the image is an image of the album art based in part on the media player type of the application and / or the audio file type of the file from which the audio content is being played by the media player application. In one or more specific implementations, the image may also be used in conjunction with the text and the image context information. Figures 2 to 5 The search results obtained at block 606 are obtained based on context information derived from the image by performing one or more of the operations described in any of the figures in FIG.

[0058] In one or more specific implementations, obtaining one or more search results may include: providing text from the electronic device to a server; receiving a ranked search result set from the server at the electronic device; and re-ranking the ranked search result set based on the application information to generate one or more search results for output by the electronic device. In one or more other specific implementations, obtaining one or more search results may include: providing text and application information from the electronic device to a server (e.g., server 120), and receiving one or more search results from the server (e.g., performing or not performing re-ranking of one or more search results from the server locally at the electronic device).

[0059] At block 608, the electronic device may provide one or more search results for output by the electronic device. For example, providing one or more search results for output may include providing one or more search results for display on a display of the electronic device (e.g., display 302) (e.g., as described herein in conjunction with Figure 4 described).

[0060] Figure 7 A flowchart of an example process 700 for performing contextual text lookup for text in an image at a server according to one or more specific implementations is illustrated. For purposes of explanation, this document primarily refers to Figure 1 Process 700 is described with reference to server 120 of computing server 120. However, process 700 is not limited to computing server 120, and one or more blocks (or operations) of process 700 may be performed by one or more other components and / or other suitable devices. Further for the purpose of explanation, the blocks of process 700 are described herein as occurring sequentially or linearly. However, multiple blocks of process 700 may occur in parallel. In addition, the blocks of process 700 need not be performed in the order shown, and / or one or more blocks of process 700 need not be performed and / or may be replaced by other operations.

[0061] At block 702, a server (e.g., server 120) may receive text (e.g., image text 312) extracted by an electronic device from an image (e.g., image 300) and contextual metadata of the image from an electronic device (e.g., computing device 110). For example, receiving the text and contextual metadata may include receiving the text and contextual metadata via a communication such as Figure 1 The network of network 106 receives text and context metadata from the electronic device (e.g., after selecting text within an image displayed at the electronic device). As an example, the electronic device can extract text from the image by, for example, identifying the text in the image, displaying the image and the text in a selectable and / or searchable state, and receiving a request from the user to search for the text displayed in a selectable and / or searchable state.

[0062] At box 704, the server may generate one or more search results for the text based on the text and the contextual metadata. As an example, the contextual metadata may include one or more of the following: labels of objects in the image (e.g., foreground objects 314 and / or background objects 316), embedding of the region of interest in the image, additional text from the image (e.g., unselected text and / or unsearched text, such as image text 318), location associated with the image, or depth information associated with the object in the image. In one or more specific implementations, the contextual metadata may also or alternatively include application information of an application associated with the image at the electronic device.

[0063] In one or more specific implementations, generating one or more search results includes: generating a ranked search result set based on the text, and re-ranking the ranked search result set based on the contextual metadata to generate the one or more search results. In one or more other specific implementations, generating the one or more search results may include performing, by the server, a search for a combination of text and contextual metadata (e.g., by providing both the text and the contextual metadata as concurrent inputs to a search engine, and receiving the one or more search results as output of the search engine).

[0064] At block 706, the server may provide one or more search results to the electronic device. For example, providing one or more search results may include providing a search result such as Figure 1 The network 106 of the network sends one or more search results to the electronic device.

[0065] As described above, one aspect of the present technology is the collection and use of data available from specific and legitimate sources for contextual lookups for images. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0066] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data can be used for contextual lookups for images. Therefore, the use of such personal information data can facilitate transaction processing (e.g., online transaction processing). In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used according to the user's preferences to provide insights into their overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0067] The present disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the requirements of the industry or government that maintain user privacy. Such information about the use of personal data should be prominently and easily accessible to users, and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legitimate use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and privacy practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including specific considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0068] Notwithstanding the foregoing, the present disclosure also contemplates implementation scenarios in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of contextual searches for images, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that the user's personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0069] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be promoted when appropriate by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differential privacy.

[0070] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without requiring access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data.

[0071] Figure 8 An electronic system 800 is illustrated that can be used to implement one or more implementations of the subject technology. The electronic system 800 can be Figure 1 The computing device 110 and / or the server 120 and / or Figure 2 The electronic system 800 may include one or more of the systems 200 and / or may be a part of one or more of the systems 200. The electronic system 800 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 800 includes a bus 808, one or more processing units 812, a system memory 804 (and / or cache), a ROM 810, a permanent storage device 802, an input device interface 814, an output device interface 806, and one or more network interfaces 816, or a subset or variation thereof.

[0072] The bus 808 generally represents all system buses, peripheral busses, and chipset buses that communicatively connect the many internal devices of the electronic system 800. In one or more specific implementations, the bus 808 communicatively connects one or more processing units 812 with the ROM 810, the system memory 804, and the permanent storage device 802. The one or more processing units 812 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different specific implementations, the one or more processing units 812 can be a single processor or a multi-core processor.

[0073] ROM 810 stores static data and instructions required by one or more processing units 812 and other modules of the electronic system 800. On the other hand, permanent storage device 802 can be a read-write memory device. Permanent storage device 802 can be a non-volatile memory unit that stores instructions and data even when the electronic system 800 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) can be used as permanent storage device 802.

[0074] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) may be used as the permanent storage device 802. Like the permanent storage device 802, the system memory 804 may be a read-write memory device. However, unlike the permanent storage device 802, the system memory 804 may be a volatile read-write memory, such as a random access memory. The system memory 804 may store any of the instructions and data that one or more processing units 812 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 804, the permanent storage device 802, and / or the ROM 810. The one or more processing units 812 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0075] The bus 808 is also connected to an input device interface 814 and an output device interface 806. The input device interface 814 enables a user to transmit information and select commands to the electronic system 800. Input devices that can be used with the input device interface 814 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 806 may, for example, enable the display of images generated by the electronic system 800. Output devices that can be used with the output device interface 806 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0076] Finally, if Figure 8 As shown in FIG. 8 , bus 808 also couples electronic system 800 to one or more networks and / or to one or more network nodes, such as via one or more network interfaces 816. Figure 1 800. In this manner, electronic system 800 can be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or can be part of a network of networks, such as the Internet. Any or all components of electronic system 800 can be used with the subject disclosure.

[0077] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) programmed with one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0078] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0079] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0080] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0081] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0082] Those skilled in the art will recognize that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways), without departing from the scope of the subject technology.

[0083] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustration of the example method. Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the illustrated frames are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned embodiment should not be understood as requiring such division in all embodiments, and it should be understood that program components and system can be generally integrated together in a single software product or encapsulated in multiple software products.

[0084] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0085] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0086] The predicate words "configured to," "operable to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more specific implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0087] Phrases such as aspect, this aspect, on the other hand, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0088] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, in a manner similar to how the term "comprising" is interpreted when used as a transitional word in a claim.

[0089] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly stated in the claims. No claim element shall be interpreted under the provisions of 35 U.S.C. §112(f) unless the element is explicitly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “step for…”

[0090] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to the singular element is not intended to mean "only one", but to refer to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the subject disclosure.

Claims

1. A method, comprising: Receiving, by an electronic device displaying an image, a request to perform a search for text in the image; deriving, by the electronic device, contextual information from the image including the text; as well as In response to the request, obtaining one or more search results based on the text and the context information; as well as The one or more search results are provided for output by the electronic device. 2 . The method of claim 1 , wherein receiving the request comprises receiving, by the electronic device, a selection of the text in the image while displaying the image.

3. The method of claim 1 , wherein obtaining the one or more search results comprises: providing the text from the electronic device to a server; receiving, at the electronic device, a ranked set of search results from the server; as well as The ranked set of search results is re-ranked based on the context information to generate the one or more search results for output by the electronic device.

4. The method of claim 1, wherein deriving the context information comprises performing, by the electronic device, at least one of the following: determining labels of objects in the image; obtaining an embedding of a region of interest in the image; obtaining unselected text from the image; determining a location associated with the image; or Depth information associated with the image is obtained.

5. The method of claim 1, wherein the image is a planar image, and wherein the method further comprises: Before receiving the request: When the planar image is displayed, the electronic device detects the text in the planar image; as well as The display of the planar image is modified to display the text as selectable text.

6. The method according to claim 1, further comprising: obtaining the image from a memory of the electronic device, from a memory of a remote device, or from a camera of the electronic device; as well as The image is displayed using the electronic device.

7. The method of claim 1, wherein obtaining the one or more search results comprises: providing the text and the context information from the electronic device to a server; as well as The one or more search results are received from the server.

8. The method according to claim 1, further comprising: Obtaining, by the electronic device, application information of an application that displays the image; as well as The one or more search results are obtained based on the text, the context information, and the application information.

9. A method comprising: receiving, by an electronic device, a request to perform a search for text in an image displayed by an application running on the electronic device; obtaining, by the electronic device and in response to the request, application information of the application; obtaining one or more search results based on the text and the application information; as well as The one or more search results are provided for output by the electronic device.

10. The method of claim 9, wherein receiving the request comprises receiving, by the electronic device, a selection of the text in the image while the application is displaying the image. The method of claim 10 , wherein the application information comprises an application type.

12. The method of claim 10, wherein the application information includes a file type of a file accessed by the application and associated with the image.

13. The method of claim 10, wherein obtaining the one or more search results comprises: obtaining image context information from the image based on the application information; as well as The one or more search results are obtained based on the text and the image context information.

14. The method of claim 10, wherein obtaining the one or more search results comprises: providing the text from the electronic device to a server; receiving, at the electronic device, a ranked set of search results from the server; as well as The ranked set of search results is re-ranked based on the application information to generate the one or more search results for output by the electronic device.

15. The method of claim 10, wherein obtaining the one or more search results comprises: providing the text and the application information from the electronic device to a server; as well as The one or more search results are received from the server.

16. A method comprising: Receiving at the server from the electronic device: text extracted from the image by the electronic device, and contextual metadata of the image; generating, by the server, one or more search results for the text based on the text and the contextual metadata; as well as The one or more search results are provided to the electronic device.

17. A method according to claim 16, wherein the contextual metadata includes one or more of the following: labels of objects in the image, embedding of regions of interest in the image, additional text from the image, locations associated with the image, or depth information associated with objects in the image.

18. The method of claim 16, wherein generating the one or more search results comprises: generating a ranked set of search results based on the text; as well as The ranked set of search results is re-ranked based on the contextual metadata to generate the one or more search results.

19. The method of claim 16, wherein generating the one or more search results comprises performing, by the server, a search on a combination of the text and the contextual metadata.

20. The method of claim 16, wherein the context metadata comprises application information of an application associated with the image at the electronic device.

Images