Personalized entity library

Abstract

This invention relates to a personalized entity library. Systems and methods for using a personalized entity library are provided. For example, a computing device includes a personalized entity library having a fixed set of entities from an entity library stored at a server, a processor, and a memory storing instructions that cause the computing device to identify a fixed set of entities relevant to a user based on a context associated with the computing device, rank the fixed set by relevance, and update the personalized entity library using a selection set determined based on the ranking and the set applicable to the user using parameters. In another example, a method includes generating a fixed set of entities from the entity library, including a location-based set and a topic-based set, and providing a subset of the fixed set to a client, which requests the subset based on the client's location and items identified in content generated for display on the client.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201680061369.0, filed on October 18, 2016.

[0003] Related applications

[0004] This application claims priority to and is a continuation of U.S. non-provisional patent application No. 14 / 962,415, filed December 8, 2015, entitled “PERSONALIZED ENTITY REPOSITORY,” which in turn claims priority to U.S. provisional patent application No. 62 / 245,241, filed October 22, 2015, entitled “PERSONALIZED ENTITY REPOSITORY,” the entire contents of which are incorporated herein by reference. Background Technology

[0005] The use of mobile devices, such as smartphones, wearables, tablets, and laptops, is increasing. By understanding what users see and what actions they take, mobile devices can create a better user experience, for example, by providing personalized predictions and assistance. Part of understanding content and actions includes identifying and classifying entities identifiable within the content on the device screen. Entities can exist in entity libraries, such as knowledge bases or visual models, that store facts and information about entities. Large public entity libraries can contain millions of such entities. Most computing devices, especially mobile computing devices such as smartphones and tablets, have limited storage space and use entity libraries via connections to servers. Summary of the Invention

[0006] The implementation divides the entity library into several fixed sets or segments. These sets can be location-based, topic-based, action-based, or function-based. The system can determine the sets at the server and only provide those sets to client devices associated with that specific client. The sets downloaded to the client are a personalized entity library accessible without a connection to the server. The client device may include a set recognition engine. The set recognition engine may include a predictive model that predicts one or more sets given some text, images, or other features. The predictive model can be used to determine which sets are most beneficial to the user. The set recognition engine can also track the user's location to determine which location sets are most relevant.

[0007] The system can rank collections for users on client devices, for example, based on factors such as user location, search history, content viewed on the device, time, and signals from other devices. Thus, for example, if a user starts researching travel to Hawaii, the system can determine that collections containing entities related to Hawaii are particularly relevant to the user. If a user flies from New York to Los Angeles, the system can determine that the Los Angeles location collection is relevant. Client devices can use the ranking to determine which collections to retrieve and which to remove from storage. For example, if a previously downloaded collection is no longer relevant, the system can remove it to make room for another collection. The system can use collection usage parameters at any time to determine which collections to include in the personalized entity library. In some implementations, users can create and control collection usage parameters. Updating collections can include determining the increment of the collection to reduce the amount of data sent to the client device. In some implementations, these collections can be versioned, for example, so that schema changes do not break applications using the model.

[0008] According to certain aspects of this disclosure, a mobile device includes a display device, a personalized entity library stored in memory, the personalized entity library including multiple fixed entity sets from an entity library stored at a server, wherein each fixed set has a corresponding identifier and includes information about entities in the set, at least one processor, and memory storing instructions that, when executed by the at least one processor, cause the mobile device to perform operations. The operations include, based on a context associated with the mobile device, identifying fixed sets of entity libraries relevant to a user of the mobile device; ranking the fixed sets by relevance; using the ranking and set usage parameters suitable for the user, determining a selected set from the identified fixed sets; and using the selected set to update the personalized entity library.

[0009] According to certain aspects of this disclosure, a method includes: receiving a screen-captured image configured to display content on a display of a mobile device; determining text in the image by performing text recognition on the image; providing the text to an ensemble prediction model, wherein the ensemble prediction model is trained to predict one or more fixed entity sets; and storing at least one of the predicted fixed entity sets in a personalized entity library in a memory on the mobile device.

[0010] According to certain aspects of this disclosure, a method may include generating multiple fixed entity sets from an entity library. The fixed sets include location-based sets, each containing entities from the entity library with locations within a cell associated with the set, and topic-based sets, at least some of which include entities from the entity library that are related to each other via embedded similarity. The method may further include providing a subset of the fixed entity sets to a client device, the client device requesting the subset based on the client device's location and based on identified items identified in content generated for display on the client device.

[0011] In one general aspect, a computer program product embodied on a computer-readable storage device includes instructions that, when executed by at least one processor formed in a substrate, cause the computing device to perform any of the disclosed methods, operations, or processes. Another general aspect includes, substantially as shown and / or described in conjunction with at least one figure, and as set forth more fully in the claims, a system and / or method for generating a fixed set of entities in an entity library and providing some of the fixed set to a personal computing device as a personal entity library.

[0012] One or more implementations of the subject matter described herein can be implemented to achieve one or more of the following advantages. As an example, generating fixed entity sets allows the system to provide a scalable library of personal or custom entities. Custom entity libraries can aid on-device text and image analytics, supporting user accessibility features without requiring a network connection. Users can control the amount of resources dedicated to their personal entity libraries, and the system can automatically determine which sets best utilize the allocated resources. As another example, the system can predict which sets are most relevant to the user. Predictions can be based on multiple factors specific to the user and / or the individual computing device, such as location, time, signals from other computing devices associated with the user or known to the device, recent activity related to the computing device, etc. As yet another example, implementations can support versioning of sets, ensuring that changes to reconstruct information within a set do not disrupt applications using that set.

[0013] Details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating an exemplary system based on the disclosed subject matter.

[0015] Figure 2An exemplary display of a user interface for specifying parameters used in a collection of personalized entity libraries, based on the disclosed subject matter.

[0016] Figure 3 An exemplary display is shown, according to the disclosed embodiments, of content provided for determining which sets are most relevant to the user of the computing device.

[0017] Figure 4 An exemplary display of a user interface for suggesting additional collections to be included in a personalized entity library, according to the disclosed embodiments, is shown.

[0018] Figure 5 Another exemplary display of a user interface for suggesting additional collections included in a personalized entity library, according to the disclosed implementation, is shown.

[0019] Figure 6 A flowchart illustrating an exemplary process for generating and updating a fixed entity set according to the disclosed embodiments is shown.

[0020] Figure 7 A flowchart illustrating an exemplary process for building a personal entity library using a fixed set of entities, according to a disclosed embodiment.

[0021] Figure 8 A flowchart illustrating an exemplary process for identifying a set of fixed entities associated with a user of a client device, according to a disclosed embodiment.

[0022] Figure 9 A flowchart illustrating an exemplary process for selecting a set of fixed entities related to a user of a client device from a set of fixed entities based on set usage parameters, according to a disclosed embodiment.

[0023] Figure 10 An example of a computing device that can be used to implement the technology is shown.

[0024] Figure 11 An example of a distributed computing device that can be used to implement the technology is shown.

[0025] The same reference numerals in each figure represent the same elements. Detailed Implementation

[0026] Figure 1This is a block diagram of a personal entity library system according to an exemplary embodiment. System 100 can be used to construct a personalized entity library from a selection of pre-computed or fixed sets of entities. These sets are fixed in the sense that they are computed prior to any particular request for the sets, or in other words, they are not generated in response to a request for the sets. System 100 can determine which sets are most useful at a given point in time and can update the personalized entity library to ensure that the entities in the personalized library are relevant to the user. Updating the personalized entity library includes deleting sets that are now less relevant to the user compared to before and adding new sets that have become relevant. The personalized entity library can be used to determine entities in content generated for display on a computing device to personalize the user experience on the computing device. Personalized user experience may include predicted actions, themes, words, or phrases, etc. Figure 1 System 100 is shown as a client-server system, wherein some data processing or data storage occurs at server 110, while other data processing occurs at client device 150. However, other configurations and applications may be used, and the implementation is not limited to the exact configuration shown.

[0027] The personalized entity library system 100 may include a server 110, which may be one or more computing devices in various forms, such as a standard server, a group of such servers, or a rack server system. For example, the server 110 may be implemented in a distributed manner across multiple computing devices. Alternatively, the server 110 may be implemented as a personal computer (e.g., a laptop computer). The server 110 may be as follows: Figure 9 The computer device 1100 shown in the diagram, or as... Figure 11 An example of a computer device 1100 is illustrated. Server 110 may include one or more processors formed in a substrate, the processors being configured to execute one or more machine-executable instructions or software, firmware, or a combination thereof. The processor may be semiconductor-based—that is, the processor may include semiconductor material capable of performing digital logic.

[0028] Server 110 can store or access entity database 130. Entity database 130 can store information about entities. Entities can be people, places, projects, ideas, topics, words, phrases, abstract concepts, concrete elements, other suitable things, or any combination of these. In some implementations, entity database 130 can be a knowledge base that stores information about entities. In some implementations, the knowledge base stores information about entities in the form of relationships between entities. For example, entities in the knowledge base can be associated with each other through labeled edges representing relationships. The knowledge base can also store attributes of entities. Some knowledge bases are very large, sometimes containing millions of entities. A knowledge base with a large number of entities and even a finite number of relationships can have billions of connections.

[0029] Entity base 130 may also include image recognition entities that can be used to identify entities in an image. For example, entity base 130 may include entities representing known images of entities and / or characteristics of images, and said entities are used in an image recognition process (e.g., using image similarity techniques) to identify entities in an image. For example, characteristics may be features provided to an image recognition model (e.g., a machine learning algorithm). Entity base 130 may represent a single knowledge base, a combination of different knowledge bases, image recognition entities, and / or a combination of knowledge base and image information. In some embodiments, entity base 130 may be stored in an external storage device accessible to server 110. In some embodiments, entity base 130 may be distributed across multiple storage devices and / or multiple computing devices, such as multiple servers. Entities and relationships in the knowledge base may be searchable, for example, via an index. For example, the index may include text referencing entities. Thus, a reference to the knowledge base can be understood as including an index that facilitates finding entities using text equivalents.

[0030] Server 110 may also include one or more computer memories. Memory, such as main memory, may be configured to temporarily, permanently, semi-permanently, or in combination thereof, store one or more data. Memory may include any type of storage device that stores information in a format that can be read and / or executed by one or more processors. Memory may include volatile memory, non-volatile memory, or a combination thereof, and storage modules or engines that perform certain operations when accessed by one or more processors. In some embodiments, modules may be stored in external storage devices and loaded into the memory of server 110. In some embodiments, modules may include an entity set generation engine 120. Entity set generation engine 120 may divide entities in entity library 130 into fixed entity sets 132. Each set in fixed entity set 132 may be considered a fragment of entity library 130 that includes entities associated by some criteria. In some embodiments, fragments may contain metadata about the entity (e.g., name, description, standard image), relationships with other entities, and / or information used to detect entities in text, such as common names, aliases, abbreviations, nicknames, and other signals. Each fixed entity set in fixed entity set 132 may have an identifier that allows an application or client device to track, identify, and request a specific set.

[0031] In some implementations, the entity collection generation engine 120 can generate various types of collections, such as location-based collections, topic-based collections, or feature collections. Location-based collections can include entities associated with a specific geographic cell. A geographic cell is a region of the Earth's surface defined by boundaries (e.g., two parallels of latitude and two parallels of longitude). These cells are common in mapping applications such as Google Maps. A location-based fixed entity collection can include any entity known to be located within that cell. The entity library 130 can include attributes specifying the location of an entity or can include metadata identifying the location. For example, a restaurant entity in entity library 130 could have a location attribute specifying one or more specific geographic cells or latitude / longitude coordinates. Although geographic cells, i.e., cells used in mapping applications, are typically of similar size, a location-based entity collection does not need to correspond to a single geographic cell but can be based on a region with a specific number of entities within that region. Thus, for example, a cell for a location-based entity collection in New York City might be very small (e.g., 10 square miles) because it is densely populated, while a cell for a location-based entity collection in western Wyoming might be hundreds of square miles but have the same number of entities as the cell in New York City. Therefore, for example, the entity set generation engine 120 can merge adjacent geographic cells until the number of entities associated with the merged cells reaches a minimum and forms a location-based cell for fixing the entity set.

[0032] The entity set generation engine 120 can also generate fixed entity sets based on topics. In some embodiments, the entity set generation engine 120 can generate topic-based sets by clustering entities using conventional clustering techniques based on the relevance of the entities. For example, clustering can be based on entity characteristics, such as embeddings, on search records, or on text such as text from documents indexed by a search engine. In some embodiments, the entity set generation engine 120 can be given seed entities and can form clusters based on the seed entities. In some embodiments, one or more topic-based sets can be based on sets identified in an entity library. For example, entities in entity library 130 can include indications of one or more sets to which an entity belongs, or entity library 130 can include set definitions, or in other words, requirements to be included in a set, and entities can be evaluated relative to the requirements to determine membership in the set. The entity set generation engine 120 can use entity members in the set to generate one or more topic-based sets. In some embodiments, the entity set generation engine 120 can compute embeddings for fragments related to each topic. Embeddings can be used to compute similarity to, for example, query embeddings generated by a set identification engine, to determine which sets are most similar to the query embeddings. For example, embeddings can represent cluster centers.

[0033] The entity set generation engine 120 can also generate a set of entity functions. A function set may include entities considered most popular, such as those most frequently searched or accessed, for example, based on their appearance in search queries. A function set may also be based on capabilities, such as the ability to translate from one language to another. As another example, a function set may include entities that are likely to be encountered when used or requested by a particular application. Of course, these function sets are provided merely as examples, and a set of functional entities may include any set of entities common to a particular purpose, characteristic, or function. In some embodiments, the entity set generation engine 120 may generate a fixed set of entities combining the above-described features. For example, the entity set generation engine 120 may generate a fixed set of entities for a restaurant in San Jose, California, or another set for artwork identification in Paris, France. In some embodiments, the entity set generation engine 120 may include a user interface that allows a user to curate one or more fixed sets of entities.

[0034] The entity set generation engine 120 can also periodically update the fixed entity set 132. For example, when entities are added, deleted, or updated in the entity library 130, the entity set generation engine 120 can determine the changes to the fixed set. In some implementations, the entity set generation engine 120 can determine an increment for each set (e.g., a list of entities to be deleted from the fixed set and a list of entities to be added to the fixed set). The entity set generation engine 120 can provide the increment to the client device 150 as a push or pull. The increment reduces the amount of data transferred between the server 110 and the client device 150 while still allowing the client device 150 to have the latest entity set. This is particularly useful when a particular fixed entity set changes frequently, such as an entity set representing a movie. In some implementations, the entity set generation engine 120 can also version each fixed set, also called a fragment. For example, the entity set generation engine 120 can change the data format of a fragment, or it can change a fragment in a destructive mode. In a destructive mode, applications using these fragments may require corresponding updates to use the new fragments. Therefore, the entity collection generation engine 120 can version fixed collections, allowing applications that have not yet been updated to continue using earlier versions without failure. In some cases, the client device 150 may include two versions of the same fixed fragment (e.g., a fragment representing a specific location) until an application using the old pattern is updated.

[0035] In some implementations, the entity set generation engine 120 can also train and maintain an ensemble prediction model 122. The ensemble prediction model, such as ensemble prediction model 122, can be any type of machine learning algorithm, such as a Long Short-Term Memory (LSTM) neural network, a feedforward neural network, a Support Vector Machine (SVM) classifier, etc., which can predict a fixed set of entities given a set of features or signals. These features may include device location, text or images in content (e.g., documents, search history, etc.), applications typically used for various tasks, etc. As a machine learning algorithm, the ensemble prediction model has two modes: a training mode and an inference mode. For example, in training mode, ensemble prediction model 122 can use tagged search history 134 and tagged crawled documents 136 to predict one or more fixed entity sets from a fixed entity set 132 given a set of signals. Tags can be created automatically or by human experts. Signals can be obtained from content generated on a client device such as device 150, or from search history such as search history 134, or from user-specific records such as screen capture index 172. In training mode, the ensemble prediction model 122 can also accept location as a signal (e.g., current geographic cell location, coordinates, etc.) to predict an entity set for a given location. During training, the model's parameters are updated to better reflect the labels assigned to the text in the training document. In inference mode, the ensemble prediction model 122 predicts one or more fixed entity sets from a fixed entity set 132 in response to a set of features representing the signal. Each predicted set may have an associated confidence score or probability score representing the level of determinism by which the provided features predict a particular fixed entity set. In some implementations, the confidence score may be based on a similarity metric that varies depending on the type of ensemble. For example, the confidence score for a location-based ensemble may be based on the physical distance to a specified location, such as calculating the current location of the device. The confidence score for a topic-based ensemble may be based on the embedding distance to a query, such as an embedding generated based on a signal from a client device. Such a signal may include recently viewed text on the screen, the status or proximity of an external device, recently searched content, stated user interests, applications installed or running on the client device, timestamps, etc.

[0036] Once trained and periodically updated, the entity set generation engine 120 can provide a set prediction model 122 to the client device 150. The client device 150 can store this model as a set prediction model 164. In some implementations, the client device 150 can personalize the set prediction model 164 by performing further training. Training can use information from search history associated with the user (e.g., from search history stored at the client device 150) or from screen capture index 172. Therefore, the set prediction model 164 can be a copy of the set prediction model 122 or a personalized copy of the set prediction model 122.

[0037] Server 110 may include search records 134 and crawled documents 136. Search records 134 may include search logs, aggregated data collected from queries, or any other data based on the queries. In some embodiments, search records 134 may be generated by the search engine during the normal process of generating search results. In some embodiments, search records 134 may be stored on different computing devices accessible to server 110. In some embodiments, search records may be distributed across multiple computing devices. For example, crawled documents 136 may be documents obtained using, for example, known or later-developed web crawling technologies. In some embodiments, crawled documents 136 represent documents accessible via the Internet and may be in the form of an index of documents.

[0038] The personalized entity library system 100 may include a computing device 150. The computing device 150 can be any mobile computing device, such as a smartphone or other handheld computing device, tablet computer, wearable computing device, etc., operating in a closed mobile environment rather than the more common open web-based environment. The computing device 150 can also be other types of personal electronic computing devices, such as laptops or net-based computers, desktop computers, televisions with processors, etc. Figure 10 Examples of computer devices 1000 or 1050 are shown in the illustration. Computing device 150 can be a computing device used by a single user, or it can be a computing device shared by multiple users.

[0039] Computing device 150 may include one or more processors formed in a substrate, the processors being configured to execute one or more machine-executable instructions or software, firmware, or a combination thereof. The processor may be semiconductor-based—that is, the processor may include semiconductor material capable of performing digital logic. Thus, computing device 150 may include one or more computer memories configured to temporarily, permanently, semi-permanently, or in combination thereof store one or more data. Thus, computing device 150 may include application 155, representing machine-executable instructions in the form of software, firmware, or a combination thereof. Components identified in application 155 may be part of an operating system or may be applications developed to run using an operating system. In some embodiments, application 155 may be a mobile application. Traditionally, mobile applications operate in a closed environment, meaning that a user uses a separate application to perform activities typically performed in a web-based browser environment. For example, a user of computing device 150 may use the mobile application in application 155 provided by bookit.com instead of booking a hotel through bookit.com. Application 155 may also include web applications that can mirror mobile applications, for example, providing the same or similar content as a mobile application. In some implementations, application 155 may include functions performed by the operating system of computing device 150.

[0040] Application 155 may include screen content agent 160 and collection recognition engine 162. In some embodiments, one or more of these applications may be provided by the operating system (not shown) of computing device 150. In some embodiments, one or more of these applications may be downloaded and installed by a user.

[0041] Screen content agent 160 may include various functionalities. In some embodiments, screen content agent 160 may be configured to obtain text information represented on the screen of a computing device from an application programming interface (API). In some embodiments, screen content agent 160 may be built into an operating system that can determine the content of text fields displayed on the screen. Text information may be considered screen capture content, and each API call or each determination of the content of a text field may be considered screen capture. In some embodiments, screen content agent 160 may be configured to capture an image displayed on the screen by copying or reading the contents of the device's frame buffer. Therefore, the captured screen may be an image and may be referred to as a captured image. Screen content agent 160 may capture the screen at regular intervals. The interval may be small, such as every half second or every second. In some embodiments, screen content agent 160 may be configured to capture the screen each time a touch event occurs (e.g., each time a user touches the screen to scroll, zoom, click a link, etc.), in response to an explicit user request or command, or when the device switches from one mobile application to another. In some embodiments, when the screen remains unchanged, screen content agent 160 may increase the interval at which screen captures occur. In other words, when the screen is static, the screen content agent 160 can capture images less frequently.

[0042] Screen content agent 160 can provide captured content or screen images and metadata to a recognition engine, which may be part of screen content agent 160 and located on computing device 150. In some embodiments, the recognition engine may be located on a server such as server 110. When a screen capture image is provided to the recognition engine, the recognition engine can perform image and text recognition on the image to identify words, entities, identifiers, etc., in the content of the screen capture image. The recognition engine can be configured to perform various types of recognition using conventional or later-developed techniques, such as character recognition, image recognition, identifier recognition, etc. Therefore, the recognition engine can generate the recognized content, which may come from words as well as from images.

[0043] Screen content agent 160 can also determine and use metadata about the screen-captured image. Metadata may include timestamps, mobile device type, mobile device identifier, mobile application running when the content is captured (e.g., the application rendering the content displayed on the screen), etc. In some embodiments, metadata may also include which applications are active, the device's location, ambient light, device motion, etc. In some embodiments, metadata may include signals from other computing devices. For example, screen content agent 160 may provide or obtain information from external devices (such as home appliances, televisions, personal assistants, music devices, alarm systems, etc.) configured to communicate with client device 150. For example, a voice-activated electronic inventory management device may store a list of groceries to be purchased. This list may be transmitted directly from the electronic inventory management device or via user account information to client device 150. As another example, a user may have a tablet and a smartphone for sharing information, or a husband and wife may have a smartphone for sharing information. Therefore, in some embodiments, the information available to client device 150 may include information provided by other devices. Some or all of this information may be included in the metadata associated with the screen-captured image.

[0044] The system can use metadata and information obtained via screen capture images to assist the device's built-in intelligence, which analyzes information to help users perform tasks on mobile devices. For example, a user talking to a friend may include movie recommendations. The device's built-in intelligence can recognize the recommendation and offer actions such as viewing movie reviews or purchasing tickets. Screen content agent 160 can use an entity library to determine whether the identified content includes known entities. While screen content agent 160 can use a public entity library such as entity library 130, this requires a connection to server 110 and slows down the identification and action suggestion process. Therefore, client device 150 may have a personalized entity library 176 stored on client device 150. Personalized entity library 176 may be a collection of fixed entity sets obtained from server 110, such as from fixed entity set 132. Personalized entity library 176 may be generated and maintained by collection recognition engine 162.

[0045] In some implementations, screen content agent 160 may include an indexing engine configured to index captured content. The index may also associate screen-captured images with text, entities, images, identifiers, etc., identified within the images. Thus, for example, the indexing engine may generate index entries for the captured images and captured content (e.g., stored in screen capture index 172). In some implementations, the indexing engine may be on a server, such as server 110, and screen content agent 160 may provide the captured images and captured content to the server. The index may be a reverse index, where key values ​​(e.g., words, phrases, entities, images, identifiers, etc.) are associated with a list of images including the key values ​​(e.g., copies of screen capture images). The index may include metadata associated with each captured image in the list (e.g., where the key value occurs on the captured image, the ranking of the image's key value, etc.). In some implementations, the index may also include a list of captured images indexed by timestamps. The indexing engine may store the index in memory, such as screen capture index 172. Of course, in some implementations, the system may store the index in user accounts on a server, in addition to or in place of user accounts on computing device 150. A user of computing device 150 can control when screen content agent 160 is active. For example, a user can specify that screen content agent 160 is only active when other specified applications 155 are running (e.g., only when a social media mobile application is running). The user can also manually turn screen content agent 160 on and off, for example, via a settings app. In some implementations, the user can invoke screen content agent 160 using gestures or actions. Disabling screen content agent 160 can also prevent the prediction and maintenance of the personalized entity library described herein.

[0046] The computing device 150 may also include a set recognition engine 162. The set recognition engine 162 can be configured to determine which entity sets in the fixed entity set 132 should be included in the personalized entity library 176. The set recognition engine 162 can use information collected or generated by the screen content agent 160, along with the set prediction model 164 and set usage parameters, to determine which sets are likely relevant to the user and obtain those sets from the server 110. For example, the set recognition engine 162 can collect signals as input for setting the prediction model 164. Signals may include information and metadata such as device location, time, status of various external devices communicating with the client device 150, proximity to other devices, information from the content of a screen capture image or a series of screen capture images, information from the screen capture index 172 or the user's search history, information in the user profile, etc. As described above, external devices such as home appliances, televisions, personal assistants, music devices, alarm systems, etc., can be configured to communicate with the client device and provide status information to the client device, and this status information can be included in the signals generated by the recognition engine 162. The information used to generate the signal can be stored as metadata in the screen capture index 172 or generated / collected when the collection recognition engine 162 determines a fixed set of entities suitable for the personalized entity library 176.

[0047] As previously described, ensemble prediction model 164 may be a copy of ensemble prediction model 122, or it may be a personalized version of ensemble prediction model 122. When prediction model 164 is a personalized version, ensemble recognition engine 162 can provide training examples from data stored on client device 150 or from user profiles associated with the user of client device 150. For example, training examples can be generated using screen capture index 172 or search records of user 180. In some implementations, with user permission, client devices can provide updates to ensemble prediction model 122 on server 110, allowing the ensemble prediction model to learn from predictions made on many client devices.

[0048] In inference mode, set identification engine 162 can provide signals, for example, as a set of features, to set prediction model 164. In response, set prediction model 164 can then provide set identification engine 162 with identifiers of one or more predicted fixed entity sets. Set identification engine 162 can then rank the predicted fixed entity sets. Client device 150 may be a device with a small form factor, limiting the storage space on the device. Therefore, set identification engine 162 can rank the predicted fixed entity sets to determine which fixed sets will be included in personalized entity library 176. In some embodiments, set identification engine 162 can operate within set usage parameters selected by the user of the client device. Set usage parameters may include the maximum number of fixed sets to be stored on the client device, the maximum amount of storage used by personalized entity library 176, a minimum ranking, or a combination thereof. If the number of predicted sets cannot be accommodated on client device 150, set identification engine 162 can select as many sets with the highest ranking as appropriate to the set usage parameters.

[0049] In some implementations, the ranking can be determined by the set prediction model 164. In some implementations, the set identification engine 162 can adjust the ranking provided by the set prediction model 164. For example, if a user frequently visits or has installed a web application that requires a specific set, the set identification engine 162 can improve the ranking of that specific set. The set identification engine 162 can also use metadata to adjust the ranking. For example, if Alice and Ted are traveling and have provided consent to share data, Alice's client device can determine that Ted's client device has included a specific set in its personalized entity library. Therefore, when Alice's client device detects that Ted's device is very close, the set identification engine 162 on Alice's device can downgrade the ranking of that specific set on her device.

[0050] In some implementations, the set identification engine 162 may periodically or after certain events (such as installing a new application, activating an application that has not been used within a predetermined period of time, when available space changes, for example, when the device's disk space becomes low (e.g., below a specified percentage) or when some action releases a specified percentage of space, when the user changes location, etc.). The set identification engine 162 may also update the personalized entity library 176 in response to the ranking. For example, the set identification engine 162 may periodically update the personalized entity library 176, when an application using a specific set of fixed entities is installed, or when the ranking of a fixed entity set becomes higher than a set currently ranked in the personalized entity library 176. For example, if Alice starts researching a trip to Hawaii, then finally, the set prediction model 164 will predict fixed segments related to Hawaii based on what Alice has viewed (e.g., from screen capture index 172 or search history). Because the collection has never previously ranked high enough to be included in the personalized entity library 176, the collection identification engine 162 can automatically update the personalized entity library 176 or obtain permission from Alice before including the specific collection, provided that its ranking exceeds the current ranking of the collection in the personalized entity library 176 or when the change in ranking exceeds a predetermined threshold. The ranking of a specific segment may drop, for example, below another segment not in the personalized entity library 176 or decrease by a predetermined percentage, should Alice return from Hawaii, or if she does not go and does not access content about Hawaii for a period of time. This can trigger the execution of the collection identification engine 162, allowing another segment (i.e., another fixed entity collection) to take its place in the personalized entity library 176.

[0051] Updating the personalized entity library 176 can be accomplished in various ways. In some implementations, the entire library is deleted and replaced with a fixed collection selected based on ranking and collection usage parameters. In some implementations, the collection usage parameters can be set and controlled by the user. In some implementations, the collection identification engine 162 can determine whether a specific fixed collection that should be in the personalized entity library 176 already exists in the personalized entity library 176. If it does exist, the collection identification engine 162 may do nothing or determine whether the collection has changed, for example, been updated on server 110. If an update has occurred, the collection identification engine 162 can download the entire collection or download an increment to apply to the collection. The collection identification engine 162 may delete a collection from the personalized entity library 176 to make room for a collection with a higher ranking.

[0052] Computing device 150 can communicate with server 110 and other mobile devices via network 140. Network 140 can be, for example, the Internet, or network 160 can be a wired or wireless local area network (LAN), wide area network (WAN), etc., implemented using devices such as gateways, bridges, switches, etc. Network 140 can also represent a cellular communication network. Through network 140, server 110 can communicate with computing device 140 and transmit data to / from computing device 140, and computing device 140 can communicate with other mobile devices (not shown).

[0053] Personalized Entity Library System 100 represents an exemplary configuration, and implementations may include other configurations. For example, some implementations may combine one or more components of Screen Content Proxy 160, Set Recognition Engine 162, or Set Prediction Model 164 into a single module or engine. Similarly, some implementations may combine one or more of Entity Set Generation Engine 120 or Set Prediction Model 122 into a single module or application. As another example, one or more of the following databases, such as Screen Capture Index 172, Personalized Entity Library 176, or user profiles or entity libraries 130, fixed entity sets 132, search records 134, or crawled documents 136 on client device 150, may be combined into a single database or may be distributed across multiple computing devices or stored in another location.

[0054] Regarding the Personalized Entity Database system 100's collection and storage of user-specific data or potential use of personal information, it may offer users the opportunity to control the collection of user information (e.g., information about the user's social networks, social actions or activities, user preferences, or the user's current location), or the opportunity to control whether and / or how screen-captured images and content are stored. For example, the system may avoid capturing content from certain applications, such as banking apps, health apps, or other similar applications, or capturing such content in violation of the terms of service. Additionally, users may be offered the opportunity to prohibit the capture of screen content from specific applications or application categories. Furthermore, some data may be processed in one or more ways before being stored or used, thereby deleting personally identifiable information. For example, a user's identity may be processed to the point that the user's personally identifiable information cannot be determined, or location information (such as city, zip code, or state level) may be generalized to the user's geographic location, making it impossible to determine the user's specific location. Therefore, users can control how the Personalized Entity Database system collects and uses information about them.

[0055] Figure 2 An exemplary display of a user interface 200 for specifying sets of parameters for a personalized entity library, based on the disclosed subject matter, is shown. The display may be a mobile device or other personal computing device (such as...). Figure 1The display of the client device 150. In some implementations, the user interface may be provided by the client device's operating system, or it may be provided by a specific application, such as from... Figure 1 Application 155 is provided. User interface 200 includes features for content proxying, such as... Figure 1 The content proxy 160 has a control 205. Control 205 can adjust whether and how the content proxy performs, and what actions the content proxy performs. The user interface 200 may also include collection usage parameters for the personalized entity library. Collection usage parameters may include one or more of maximum storage 210, maximum collection size 215, or storage percentage 220. Maximum storage 210 can be set regarding the personalized entity library, such as... Figure 1 The personalized entity library 176 has limitations on the amount of memory used. Client devices can download and store a fixed set of entities (fragments), but not exceeding the maximum storage 210.

[0056] The collection usage parameters may also include a maximum collection 215. The maximum collection 215 limits the number of fixed entity collections that the client device will include in the personalized entity library. In other words, the client device can download and store only a specific number of collections. The storage percentage 220 can provide flexible limitations on the size of the personalized entity library. For example, when the client device has more memory, the storage percentage 220 parameter can allow for a larger personalized entity library, but when space on the client device is limited, the parameter can limit the collections stored in the personalized entity library. The user interface 200 can also allow the user to select combinations of collection usage parameters to turn the collection usage parameters on or off, for example, via checkboxes. Therefore, the user can control the size of the personalized entity library on their device, which in turn controls which fragments are included in the library. The user interface 200 may also include a control 225 that allows the user to specifically select one or more entity collections to include in the personalized entity model. When the user specifically uses control 225 to select a fragment from the entity library, the system can rank that fragment highly based on its relevance, ensuring that it is always included in the personalized entity model.

[0057] Figure 3 An exemplary display 300 is shown, according to the disclosed embodiment, providing content for determining which sets of contexts are most relevant to a user of a computing device. The display 300 may be a mobile device or other computing device (such as...). Figure 1 The client device (150) displays the screen. Figure 3In the example, the text "Alien Invaders" is selected. This selection can be made by the user or automatically by the client device (e.g., a content proxy or action suggestion application running on the client device). The system has already provided four suggested actions for selection 305, namely action 340, which can be used for a mobile application to provide local movie time, movie review action 325, and movie database action 320. The user interface can be based on recognizing selection 305 as an entity existing in a personalized entity library, with the selected actions 340, 325, and 320. Therefore, the system can provide display 300 even without a connection to a server and a server-based entity library.

[0058] Figure 4 An exemplary display of a user interface 400 for suggesting additional fragments to be included in a personalized entity library, according to the disclosed embodiment, is shown. The user interface 400 can be used on mobile devices or other computing devices (such as…). Figure 1 Generated on client device 150. Figure 4 In the example, the user has already entered text into search interface 405. Search interface 405 can be used in a browser or similar mobile application, but can come from any search bar in any application. The system can analyze the text provided to search interface 405 and, for example, determine that the text is highly relevant to a fixed set of topics-based entities using an ensemble prediction model. High relevance can be determined based on a high confidence score or probability from the ensemble prediction model. High relevance can be based not only on the content of user interface 400 but also on previously presented content. For example, a user might be looking at hotels in Hawaii in a booking app or reading about things to see in Hawaii. Even though the client device may not have an entity for Hawaii in its current personalized entity library, the ensemble prediction model can identify several words or images that predict that set. In some implementations, the ensemble recognition engine can continuously analyze the text in the relevance model. In some implementations, the ensemble recognition engine can be invoked during use of the search interface.

[0059] When the probability of a fixed entity set indicates high relevance, the system can automatically download and store the fixed entity set for Hawaii, also known as the Hawaii fragment. In some implementations, this can be a location-based set for all entities located in Hawaii, or a topic-based set, or a combination of these sets. In some implementations, the user interface may include a confirmation window 410. The confirmation window 410 allows the user to accept or reject the suggestion. If the user rejects the suggestion, the system does not download the Hawaii fragment. If the user accepts the suggestion, the system can download the Hawaii fragment and may remove one or more fragments currently in the personalized entity library to make room for the new fragment, as referenced. Figure 7 A more detailed explanation.

[0060] Figure 5 Another exemplary display of a user interface 500 for suggesting additional collections to be included in a personal entity library, according to the disclosed implementation, is shown. The user interface 500 can be displayed on mobile devices or other computing devices (such as…). Figure 1 Generated on client device 150. Figure 5 In the example, the user has just installed a restaurant reservation application 505. In response to the installation process, the set identification engine can determine that application 505 has explicitly requested a specific set of fixed entities, or, for example, using a set prediction engine, is likely referencing a specific set of fixed entities. Therefore, a confirmation window 510 can be presented to the user. Similar to confirmation window 410, confirmation window 510 can provide the user with the opportunity to choose to download a specific segment or decline the download. In some embodiments, application 505 may generate window 510. In some embodiments (not shown), the application installation process or the set identification engine may automatically download the specific set of fixed entities.

[0061] Figure 6 A flowchart is shown of an exemplary process 600 for generating and updating a fixed entity set according to the disclosed embodiment. Process 600 can be performed by, for example... Figure 1 The system 100 executes a personalized entity library system. Process 600 can be used to generate and maintain various fragments of the entity library, i.e., fixed sets of entities. The entity library can be any large entity library, for example... Figure 1 The system has an entity library 130. Process 600 can begin by generating a location-based entity set (605). The location-based entity set can be any entity in the entity library located in a specific cell. A cell can be defined by a boundary (e.g., a line connecting three or more geographic coordinates). The cell can correspond to one or more geographic cells (e.g., cells defined by a map application). The system can also generate a topic-based fixed entity set (610). The topic-based set can be formed by clustering, by similarity or relationship with a seed entity, by membership in the set, etc. The system can also generate a functional entity set (615). The functional set can be a combination of location-based and topic-based sets, can be based on the purpose of a specific application, can be based on popularity (i.e., the most frequently searched entities and / or those that appear in the most crawled documents), can be based on tasks or actions, etc. Steps 605 to 615 can be executed at the initial time and update the set. For example, at the initial time, the system can execute steps 605 to 615 to generate an initial entity set. Subsequently, the system can execute steps 605 to 615 to update the initial set, add new sets, etc.

[0062] The system can version fixed entity collections. For example, when a format change in a collection causes applications using those collections to malfunction, the system can retain the old version and generate a new version after the schema change. In this way, the system can handle schema changes in a way that does not cause errors on client devices. Therefore, the system can determine whether the newly generated collection includes a schema change (620). If so, a new version identifier (625) can be assigned to the newly generated collection (e.g., from steps 605 to 615). The version identifier will enable applications using those collections to correctly download and use the appropriate version. When an update involves a schema change, an incremental set is not required. If there is no schema change (620 is no), the system can generate an increment for each collection (630). Increments are used when a client device only downloads changes to a fixed entity collection, rather than deleting all fixed collections in a personalized entity model after storing the most recent collection among the highest-ranked fixed entity collections used for that device. An increment can be a list of entities to be deleted and a list of entities to be added. Therefore, an update to an entity (e.g., a name change, metadata or attribute change, etc.) can be to delete the entity after it has been read. Incremental processing allows the system to transfer less data between the server maintaining the collections and the client devices using the collections. However, step 630 is optional and can be performed on none of the collections or even on any single collection. For example, when updating the personalized entity library, the client device can delete all currently stored fragments and extract the desired fragments (e.g., predicted fragments that fall within the collection's usage parameters) and store them as personalized entity models. Each fixed entity collection can have a unique identifier. In some implementations, the unique identifier can be a hash of the fixed collection's contents. Process 600 then ends. Process 600 can be repeated periodically to ensure that the fixed entity collections are up-to-date.

[0063] Figure 7 A flowchart is shown of an exemplary process 700 for constructing a personal entity library using a pre-computed set of entities, according to a disclosed embodiment. Process 700 can be performed by, for example... Figure 1 The personalized entity library system of system 100 is executed. Process 700 can be executed on the client device to determine which fixed sets or fragments of entities are included in the personalized entity library stored on the client device. Process 700 is executed on the client device continuously, on a periodic basis, when network connectivity changes, when triggered by events such as installing a new application, receiving a call, adding a new contact, etc., or when a user requests synchronization. Process 700 can be executed by a set identification engine (such as set identification engine 162).

[0064] Process 700 can begin by identifying a fixed set of entities associated with the user of the client device (705). These sets can be fixed in the sense that they are determined before performing step 705, rather than as part of step 705. In other words, these sets are determined independently of any particular user or query on the entity library. The set identified as part of step 705 is a much smaller subset than the total number of fixed sets. Identifying the fixed sets can be based on, for example, the context associated with the client device. For example, the set prediction model can be provided with a set of features generated by various signals (i.e., contexts), which may include metadata and information collected from content generated on the client device, such as via content proxies, search history, user profiles, etc. In some implementations, the information used as signals may include information from devices near the client device, such as home appliances or other computing devices in an IoT home, other personal computing devices associated with the user or the user's resident, etc. While such signals originate from another device, proximity is a context associated with the client device. In some implementations, the information may include the location of the computing device, timestamps, content from the user's search history, content from a screen capture index, content associated with the user profile, applications installed on the client device, activities performed by the user (e.g., changing device settings, installing applications), etc. Signals may be provided to ensemble prediction models, such as ensemble prediction model 122 or ensemble prediction model 164, which may further provide one or more predicted sets of fixed entities. The set of fixed entities identified in step 705 may also be based on the device's location, applications installed on the device, tasks or activities typically performed on the device, etc.

[0065] The system can then rank the identified set of fixed entities by relevance (710). Relevance can be based on probability scores provided by a prediction model for the set. In some implementations, the system can adjust this probability based on other information, such as the device's location, time, other nearby devices, etc. For example, if two travel companions have linked their devices, the first companion's device can, for example, determine via device-to-device signaling that the second companion's device has a specific fragment of the stored entity library, and the first companion's device can downgrade the relevance of that fragment, for example, because one of the two devices already possesses a fragment of the entity library stored in a personalized entity library. As another example, a user's device can have a specific application that requires a specific fragment of the entity library, and the system can increase the relevance of that specific fragment. As another example, the system can, for example, via… Figure 2Control 225 is used to improve the relevance of user-specified segments. For a location-based fixed entity set, the system can set a relevance score for the set based on the distance between the client device and the cell represented by the set. For example, the relevance is very high when the client device is within the cell. The relevance of other cells can be based on the distance between the center of the cell and the current location of the client device, so that longer distances result in lower relevance.

[0066] The system can determine the selected set for the personalized entity library based on set usage parameters and ranking (715). In some implementations, the system can determine a relevance threshold based on set usage parameters and the ranking of a fixed set of entities identified as relevant to the user. Any set with a relevance score that meets the threshold can be included in the set selected for the personalized entity library. Figure 9 This is another example of selecting a set based on set usage parameters and ranking. The system can then determine whether the personalized entity library needs to be updated (720). For example, if the selected fixed entity set is the same as the set currently included in the personalized entity library, but the system has not recently checked for updates, the system can update the library (720 for Yes). As another example, if the system has already been instructed to update one of the fixed sets in the selected sets, the system can update the library (720 for Yes). If the fixed entity set is in the selected set but is not currently in the personalized entity library, the system can update the library (720 for Yes). If the system does not need to update the library (720 for No), process 700 ends.

[0067] If the system does update the library (720 if yes), the system can select the highest-ranking set among the selected sets (725). For ease of explanation, this highest-ranking set can be referred to as the first set. The system can determine whether the first set is already in the personalized entity library (730). If the first set is in the library (730 if yes), the system can update the first set on the client device if an update is needed (735). The system can determine whether an update is needed by dragging the first set from the server or by notifying the server that an update exists. The update can be incremental, such as a list of entities to be deleted and a list of entities to be added to a fixed group. If the first set is not in the personalized entity library (730 if no), the system can determine whether the personal entity library has space for the first set (740). For example, the personal entity library may have limits on the number of sets in the library or the memory used by the library, or both. If adding the first set would exceed the limit (740 if no), the system can determine whether there is currently a fixed set in the personalized entity library that can be deleted (745). If the set currently in the personalized entity library is not in the selected fixed entity set, i.e., in the entity set identified in step 715, the set can be deleted. If the set has a lower ranking than the first set, it can also be deleted. If there is a set to be deleted (745 if yes), the system can delete the set from the personalized entity library (750) and return to step 740. If no set can be deleted (745 if no), process 700 can end.

[0068] When space exists in the personalized entity library for the first set (740 if yes), the system can add the first set to the personalized entity library on the mobile device (755). The system can then determine if a next highest-ranking set exists in the selected sets (760). If a next highest-ranking set exists (760 if yes), the system can perform steps 725 to 760 on the next set. For ease of explanation, this next set will then become the first set. Thus, for example, the system can determine whether the next set (i.e., the current first set) is already in the personalized entity library (730), etc. When all sets in the selected sets have been processed via steps 730 to 760 (760 if no), the system has updated the personalized entity library and the process ends at 700.

[0069] Figure 8 A flowchart is shown of an exemplary process 800 for identifying a set of fixed entities associated with a user of a client device, according to a disclosed embodiment. Process 800 may be performed by, for example... Figure 1 The system executes a personalized entity library system of system 100. As... Figure 7As part of step 705, process 800 can be performed on the client device to determine which fixed entity sets or fragments are relevant to the user. Process 800 begins by determining the location of the client device (805). This location can be represented as coordinates, such as Global Positioning System (GPS) coordinates, or as an identifier for a geographic cell, or some other method of representing a location on Earth. The system can assign relevance of a location-based fixed entity set based on the distance from the client device to the cell represented by the location-based fixed set (810). This distance can be measured from the client device's location to the center of the cell. Relevance can also be inversely proportional to distance; therefore, shorter distances have higher relevance. A particular fixed entity set can receive the highest relevance score when the client device is located in a cell for that specific fixed entity set.

[0070] The system can also determine a topic-based fixed entity set (815) associated with the user. The topic-based set can be determined using a set prediction model (such as set prediction model 164 or set prediction model 122) that provides one or more identifiers for the feature-based fixed entity set. Features can be based on multiple information items, such as device location, content in search history, user profile or screen capture content, data from devices near the client device (e.g., data from devices exchanging data with the client device), timestamps, applications currently installed or running, etc. The system can set a relevance score for each topic-based fixed entity set as a similarity score to the embedding of the user on the client device (820). In some implementations, similarity can be a probability of association with a fixed set provided by the set prediction model. The system can also determine a functional fixed entity set (825). The functional set can be entities considered most popular, such as most frequently searched entities, most frequently accessed entities, entities frequently found in crawled documents, etc. The functional set can also include entities used for specific tasks, actions, or applications. For example, the functional set can include entities that enable the device to perform OCR in a specific language or entities used by an art appreciation application. The feature set can also include location-based sets and topic-based sets, such as a combination of movie theaters in Portland, Oregon, or national monuments in Washington, D.C. In some implementations, the system can use entity popularity as a relevance score for the fixed entity set of features. In some implementations, the fixed sets are ranked based on the applications the user has installed. Then process 800 ends, having identified the fixed entity sets relevant to the user on the computing device and assigned a relevance score to each set.

[0071] Figure 9A flowchart is shown of an exemplary process 900 for selecting a set of fixed entities related to a user of a client device from a set of fixed entities based on set usage parameters, according to a disclosed embodiment. Process 900 can be performed by, for example... Figure 1 The personalized entity library system of System 100 is executed. Process 900 can be executed on the client device as... Figure 7 Step 715 involves determining, based on set usage parameters and ranking, which user-related sets or fragments of fixed entities to include in the personal entity library. Process 900 begins by initializing set count and model size variables. The set count can be set to zero, and the library size can also be set to zero. The system then selects the fixed set with the highest ranking from the identified relevant fixed sets (910). The system can determine whether the set usage parameters include the quantity parameter (915). The quantity parameter is... Figure 2 The maximum set is 215, representing a limit on the number of fixed sets that can be in the personalized entity library. If there is a quantity parameter (915 if yes), the system can increment the set count by 1 (920). The system can then determine if the set count is greater than the quantity parameter (925). If yes, process 900 can end because the fixed set will exceed the limit represented by the set usage parameter. However, for the highest-ranked set, the set count is not greater than the quantity parameter (925 if no), so the system continues to step 930.

[0072] The system determines whether the set usage parameters include space parameters (930). Space parameters can be represented as a specified amount of memory (e.g., Figure 2 Maximum storage 210) or percentage of available memory (e.g., Figure 2The storage percentage (220) or a combination thereof. If the set usage parameters do include the space parameter (930 if yes), the system can add the size of the fixed set being checked to the library size (953). For example, this size can be known or estimated based on the average size of the fixed entity sets. The system can determine whether the library size is greater than the space parameter (940). If yes (940 if yes), process 900 ends because the fixed set would violate the restrictions determined by the set usage parameters. Otherwise (940 if no), the system advertises the set to the selected set (945). The selected set is the highest-ranking fixed set that is determined to be user-related and meets the restrictions determined by the set usage parameters. If another set exists among the identified sets (e.g., a fixed set identified as user-related), 950 if yes, the system can select the next highest-ranking set (955) and continue with steps 915 to 950 above. If no other set exists among the identified sets (950 if no), process 900 ends, and the set to be included in the library is identified based on the set usage parameters.

[0073] Figure 10 A general-purpose computer device 1000 is shown that can be used with the techniques described herein, and can be operated as follows: Figure 1 Examples of server 110 and / or client device 150. Computing device 1000 is intended to represent various example forms of computing devices, such as laptop computers, desktop computers, workstations, personal digital assistants, cellular phones, smartphones, tablet computers, servers, and other computing devices including wearable devices. The components shown herein, their connections and relationships, and their functions are intended only as examples and are not intended to limit the implementation of the invention described herein and / or claimed.

[0074] Computing device 1000 includes a processor 1002, a memory 1004, a storage device 1006, and an expansion port 1010 connected via interface 1008. In some embodiments, among other components, computing device 1000 may include a transceiver 1046, a communication interface 1044, and a GPS (Global Positioning System) receiver module 1048 connected via interface 1008. Device 1000 can communicate wirelessly via communication interface 1044, which may include digital signal processing circuitry if necessary. Each of components 1002, 1004, 1006, 1008, 1010, 1040, 1044, 1046, and 1048 may be mounted on a common motherboard or otherwise installed as appropriate.

[0075] Processor 1002 is capable of processing instructions for execution within computing device 1000, including instructions stored in memory 1004 or storage device 1006, to display graphical information for a GUI on an external input / output device such as display 1016. Display 1016 may be a monitor or a flat-panel touchscreen display. In some embodiments, multiple processors and / or multiple buses may be used together, along with multiple memories and memory types, where appropriate. Additionally, multiple computing devices 100 may be connected, each providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0076] Memory 1004 stores information within computing device 1000. In one embodiment, memory 1004 is one or more volatile memory cells. In another embodiment, memory 1004 is one or more non-volatile memory cells. Memory 1004 may also be another form of computer-readable medium, such as a magnetic disk or optical disk. In some embodiments, memory 1004 may include extended memory provided through an expansion interface.

[0077] Storage device 1006 provides mass storage for computing device 1000. In one embodiment, storage device 1006 may be or include computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, or magnetic tape devices, flash memory or other similar solid-state storage devices, or device arrays including devices in a storage domain network or other configuration. A computer program product may be tangibly embodied in such a computer-readable medium. The computer program product may also include instructions that, when executed, implement one or more methods as described above. Computer or machine-readable media are storage devices such as memory 1004, storage device 1006, or storage devices on processor 1002.

[0078] Interface 1008 may be a high-speed controller managing bandwidth-intensive operations of computing device 1000, a low-speed controller managing less bandwidth-intensive operations, or a combination of such controllers. An external interface 1040 may be provided to enable device 1000 to communicate with other devices in the vicinity. In some embodiments, controller 1008 may be coupled to storage device 1006 and expansion port 1014. The expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches and routers, for example, via a network adapter.

[0079] As shown in the figure, the computing device 1000 can be implemented in various different forms. For example, it can be implemented as a standard server 1030, or multiple such servers in a group. It can also be implemented as part of a rack-mount server system. Furthermore, it can be implemented as a computing device such as a laptop computer 1032, a personal computer 1034, or a tablet computer / smartphone 1036. The entire system can consist of multiple computing devices 1000 communicating with each other. Other configurations are possible.

[0080] Figure 11 A general-purpose computer device 1100 is shown that can be used with the techniques described herein, which may be Figure 1 Example of server 110. Computing device 1100 is intended to represent various exemplary forms of large-scale data processing devices, such as servers, blade servers, data centers, mainframes, and other large-scale computing devices. Computing device 1100 may be a distributed system with multiple processors and may include network-attached storage nodes interconnected via one or more communication networks. The components, their connections and relationships, and their functions shown herein are intended only as examples and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0081] The distributed computing system 1100 may include any number of computing devices 1180. The computing devices 1180 may include servers or rack servers, mainframes, etc., that communicate via local area networks or wide area networks, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc.

[0082] In some implementations, each computing device may include multiple racks. For example, computing device 1180a includes multiple racks 1158a-1158n. Each rack may include one or more processors, such as processors 1152a-1152n and 1162a-1162n. The processor may include a data processor, network-attached storage devices, and other computer-controlled devices. In some implementations, a processor may operate as a main processor and control scheduling and data distribution tasks. Processors may be interconnected via one or more rack switches 1158, and one or more racks may be connected via a switch 1178. Switch 1178 may handle communication between multiple connected computing devices 1100.

[0083] Each rack may include memory such as memory 1154 and memory 1164, and storage such as 1156 and 1166. Storage 1156 and 1166 may provide mass storage and may include volatile or non-volatile storage, such as network-attached disks, floppy disks, hard disks, optical disks, magnetic tapes, flash memory, or other similar solid-state storage devices, or arrays of devices including storage domain networks or other configured devices. Storage 1156 or 1166 may be shared among multiple processors, multiple racks, or multiple computing devices, and may include computer-readable media storing instructions executable by one or more processors. Memory 1154 and 1164 may include, for example, one or more volatile memory cells, one or more non-volatile memory cells, and / or other forms of computer-readable media, such as disks or optical disks, flash memory, cache, random access memory (RAM), read-only memory (ROM), and combinations thereof. Memory such as memory 1154 may also be shared among processors 1152a-1152n. Data structures such as indexes can be stored across storage 1156 and memory 1154. The computing device 1100 may include other components not shown, such as controllers, buses, input / output devices, communication modules, etc.

[0084] An entire system, such as server 110, can consist of multiple computing devices 1100 that communicate with each other. For example, device 1180a can communicate with devices 1180b, 1180c, and 1180d, and these devices can be collectively referred to as server 110. As another example, Figure 1 System 100 may include one or more computing devices 1100. Some computing devices may be geographically close to each other, while others may be geographically far apart. The layout of system 1100 is merely exemplary, and the system may employ other layouts or configurations.

[0085] According to certain aspects of this disclosure, a mobile device includes a display device, a personalized entity library stored in memory, the personalized entity library including multiple fixed entity sets from an entity library stored at a server, wherein each fixed set has a corresponding identifier and includes information about entities in the set, at least one processor, and memory storing instructions that, when executed by the at least one processor, cause the mobile device to perform an operation. This operation includes identifying fixed sets of entity libraries relevant to a user of the mobile device based on a context associated with the mobile device; ranking the fixed sets by relevance; determining a selected set from the identified fixed sets using the ranking and set usage parameters suitable for the user; and updating the personalized entity library using the selected set.

[0086] This and other aspects may include one or more of the following features. For example, updating the personalized entity library may occur in response to determining that a first fixed set among the identified fixed sets is not present in the personalized entity library. As another example, updating the personalized entity library may include removing sets from the personalized entity library that are not selected sets. As another example, set usage parameters may include the number of fixed sets and / or the amount of memory allocated to the personalized entity library, which is set by the user. As another example, the multiple fixed sets stored in the personalized entity library may include location sets, wherein entities in the location sets are located in the same cell. As another example, the multiple fixed sets stored in the personalized entity library may include topic sets, wherein entities in the topic sets are entities clustered together based on entity characteristics. In some implementations, the ranking of the first topic set is assigned by a predictive model based on the ranking of items identified in content generated for display on a display device. As another example, updating the personalized entity library may include adding fixed sets that are not currently present in the personalized entity library and using the increment of fixed sets that are present in the personalized entity library.

[0087] As another example, the operations may also include using a personalized entity library to identify entities in content generated for display on a display device. As another example, each of a plurality of collections has a version identifier that changes when an update to a particular collection disrupts the pattern of the personalized entity library. As another example, these operations may also include initiating the display of information identifying a first fixed collection from selected collections, and controls configured to enable a user to accept the installation of the first fixed collection, and downloading and adding the first fixed collection to the personalized entity library when the user accepts the installation. As another example, identifying a fixed collection of a user-related entity library based on context may include determining the location of the mobile device and using that location to determine a location-based fixed collection related to the user, using a collection prediction model to identify a topic-based collection related to the user, and determining a collection identified by at least one application installed on the mobile device. As another example, a fixed set of user-related entities based on context recognition may include determining content previously viewed on a mobile device and feeding the previously viewed content to an ensemble prediction model configured to predict at least one topic-based set from the previously viewed content and / or determining recent searches performed on a mobile device and feeding content from the recent searches to an ensemble prediction model configured to predict at least one topic-based set from the content. As yet another example, a fixed set of user-related entities based on context recognition may include determining activities performed by a user of a mobile device and predicting a fixed set of entities based on those activities.

[0088] According to certain aspects of this disclosure, a method includes: receiving a screen-captured image configured to display content on a display of a mobile device; determining text in the image by performing text recognition on the image; providing the text to an ensemble prediction model, wherein the ensemble prediction model is trained to predict one or more fixed entity sets; and storing at least one of the predicted fixed entity sets in a personalized entity library in a memory on the mobile device.

[0089] These and other aspects may include one or more of the following features. For example, the method may also use the personalized entity library to identify entities in the screen-captured image. The method may further include determining the location associated with the content identified in the screen-captured image; providing the location to the ensemble prediction model, which, based on the location, predicts at least one location-based fixed entity set; and storing the location-based fixed entity set in the personalized entity library. As another example, the prediction model may predict multiple predicted fixed entity sets, and the method further includes ranking each predicted fixed entity set; using ensemble usage parameters and the ranking to determine a selected fixed entity set from the predicted fixed entity sets; and storing the selected fixed entity set in the personalized entity library.

[0090] According to certain aspects of this disclosure, a method may include generating multiple fixed entity sets from an entity library. The fixed sets include location-based sets, each containing entities from the entity library with locations within a cell associated with the set, and topic-based sets, at least some of which include entities from the entity library that are related to each other via embedded similarity. The method may further include providing a subset of the fixed entity sets to a client device, the client device requesting the subset based on the client device's location and based on identified items identified in content generated for display on the client device.

[0091] These and other aspects may include one or more of the following features. For example, the method may also include identifying at least one entity added to the entity library; determining at least one fixed set to which the added entity belongs; and generating an increment for including the at least one fixed set of the added entity. As another example, the method may also include identifying at least one entity added to the entity library; determining at least one fixed set to which the added entity belongs; determining that the pattern of the fixed set has changed; and generating a new version of the at least one fixed set, the new version including the added entity and having a new version identifier. As another example, at least some of the topic-based sets may include entities similar to the seed entity and / or the fixed sets may also include feature sets, at least one of the feature sets including entities considered most popular.

[0092] Various implementations may include implementations using one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be dedicated or general purpose and is coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device.

[0093] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any non-transitory computer program product, apparatus, and / or device (e.g., disk, optical disk, memory (including read-access memory), programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor.

[0094] The systems and techniques described herein can be implemented in computing systems that include back-end components (e.g., as data servers), middleware components (e.g., application servers), front-end components (e.g., client computers having a graphical user interface or web browser that a user can interact with using implementations of the systems and techniques described herein), or any combination of these back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., communication networks). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0095] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship arises from computer programs running on the respective computers and having a client-server relationship with each other.

[0096] Various implementations have been described. However, various modifications can be made without departing from the scope of the invention as defined by the following claims. Furthermore, the logic flow illustrated in the drawings does not require a specific or sequential order to achieve the desired result. Additionally, other steps may be provided, or steps may be removed from the flow, and other components may be added to or removed from the system. Therefore, other implementations are also within the scope of the following claims.

Claims

1. A method implemented by one or more processors, the method comprising: Receive a screen capture image, which captures content displayed on the display of a mobile device; The text in the screen capture image is determined by performing text recognition on the screen capture image; The text is processed using a trained set prediction model to predict one or more fixed entity sets based on the text, wherein each of the one or more fixed entity sets has a corresponding identifier and includes information about the entities in that fixed entity set; This causes the mobile device to download at least one set of fixed entities from the predicted set of fixed entities; At least one set of fixed entities from the predicted set of fixed entities is stored in a personalized entity library in the memory of the mobile device; and After storing the at least one set of fixed entities: Using at least one stored set of fixed entities to identify entities in additional screen-captured images captured at the mobile device; and Render content based on the identified entities at the mobile device.

2. The method of claim 1, further comprising: Determine the location associated with the content identified in further screen capture images; The trained ensemble prediction model is used to process the location to predict at least one location-based fixed entity set based on the location. as well as The at least one location-based fixed entity set is stored in the personalized entity library.

3. The method of claim 2, further comprising: Use at least one stored set of location-based fixed entities to identify location-based entities based on data captured at the mobile device; as well as Further content based on the identified location-based entities is rendered on the mobile device.

4. The method of claim 1, wherein, The trained ensemble prediction model predicts multiple fixed sets of entities, and the method further includes: Rank each of the predicted fixed entities; Using set parameters and rankings, a selected set of fixed entities is determined from the predicted set of fixed entities, the selected set of fixed entities including the at least one set of fixed entities; and The selected set of fixed entities is stored in the personalized entity library.

5. The method of claim 4, wherein, The parameters used for the set include a fixed number of sets.

6. The method of claim 4, wherein, The parameters used in the collection include the amount of memory allocated to the personalized entity library.

7. A mobile device, comprising: Display devices; A personalized entity library stored in memory, the personalized entity library comprising multiple fixed entity sets from an entity library stored on a server, wherein each fixed entity set has a corresponding identifier and includes information about the entities in that set; At least one processor; and A memory storing instructions that, when executed by the at least one processor, cause the mobile device to: Capture a screen capture image, the screen capture image capturing the content displayed on the display device; This enables the use of a trained ensemble prediction model to process text in the screen-captured image in order to predict one or more fixed sets of entities based on the text; This causes the mobile device to download at least one set of fixed entities from the predicted set of fixed entities; At least one set of fixed entities from the predicted set of fixed entities is stored in a personalized entity library stored in the memory of the mobile device; and After storing the at least one set of fixed entities: Using at least one stored set of fixed entities to identify entities in additional screen-captured images captured at the mobile device; and Content based on the identified entities is rendered via the display device.

8. The mobile device of claim 7, wherein the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to: Determine the location associated with the content identified in further screen capture images; This enables the use of the trained ensemble prediction model to process locations determined via data captured at the mobile device, in order to predict at least one location-based set of fixed entities based on those locations; and The at least one location-based fixed entity set is stored in the personalized entity library stored in the memory of the mobile device.

9. The mobile device of claim 8, wherein the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to: Using at least one stored set of location-based fixed entities to identify location-based entities based on further data captured at the mobile device; and Further content based on the identified location-based entities is rendered via the display device.

10. The mobile device of claim 7, wherein, The trained ensemble prediction model predicts multiple predicted sets of fixed entities, ranks each of the predicted sets of fixed entities, and uses ensemble usage parameters and rankings to determine a selected set of fixed entities from the predicted sets of fixed entities, wherein the selected set of fixed entities includes the at least one set of fixed entities.

11. The mobile device of claim 10, wherein, The parameters used for the set include a fixed number of sets.

12. The mobile device of claim 10, wherein, The parameters used in the collection include the amount of memory allocated to the personalized entity library.

13. A method implemented by one or more processors, the method comprising: Receive a screen capture image, which captures content displayed on the display of a mobile device; The screen capture image is processed to determine the location associated with the content displayed in the screen capture image; The location is processed using a trained ensemble prediction model to predict one or more fixed entity sets based on the location, wherein each of the one or more fixed entity sets has a corresponding identifier and includes information about the entities in that fixed entity set; This causes the mobile device to download at least one set of fixed entities from the predicted set of fixed entities; At least one set of fixed entities from the predicted set of fixed entities is stored in a personalized entity library in the memory of the mobile device; and After storing the at least one set of fixed entities: Using at least one stored set of fixed entities to identify entities in additional screen-captured images captured at the mobile device; and Render content based on the identified entities at the mobile device.

14. The method of claim 13, wherein, The trained ensemble prediction model predicts multiple fixed sets of entities, and the method further includes: Rank each of the predicted fixed entities; Using set parameters and rankings, a selected set of fixed entities is determined from the predicted set of fixed entities, the selected set of fixed entities including the at least one set of fixed entities; and The selected set of fixed entities is stored in the personalized entity library.

15. The method of claim 14, wherein, The parameters used for the set include a fixed number of sets.

16. The method of claim 15, wherein, The parameters used in the collection include the amount of memory allocated to the personalized entity library.

17. The method of claim 14, wherein, The parameters used in the collection include the amount of memory allocated to the personalized entity library.

18. A method implemented by one or more processors, the method comprising: Determine that each of the multiple user interactions via a user device is associated with a specific location; A confidence score for a location-based set of entities is determined based on the multiple user interactions that are all related to the specific location. The confidence score of the location-based entity set satisfies the threshold: This enables the rendering of interactive prompts related to the location-based set of entities at the user device; In response to the user device accepting the interactive prompt: This causes the user equipment to download the location-based entity set, which is within a geographical boundary defined for the specific location; The location-based entity set is stored in a personalized entity library in the memory of the user device; as well as In response to the rejection of user interaction with the interactive prompt on the user device: The user equipment is prohibited from downloading the location-based entity set.

19. The method of claim 18, further comprising: After the user equipment downloads the location-based entity set: Identify additional user interactions via the user device that are related to the application currently running on the user device; Determine that the additional user interaction is associated with at least one entity from the location-based entity set; Based on determining that the additional user interaction is associated with at least one entity from the location-based entity set, one or more actions are selected for suggestion to the user of the user device; as well as This causes the user equipment to display one or more suggested actions based on the selected one or more actions.

20. The method of claim 19, wherein, The actions suggested in one or more of the above statements are all associated with different applications than the applications related to the additional user interaction.

21. The method of claim 19, wherein, Each of the one or more suggested actions is associated with at least one additional entity from the location-based entity set.

22. The method of claim 18, wherein, Determining that each of multiple user interactions via a user device is associated with a specific location includes: Determine the specific location of the user device during the plurality of user interactions.

23. The method of claim 22, wherein, The confidence score for the location-based entity set for the specific location is determined based on the multiple user interactions, all of which are related to the specific location. Based on the identified specific location, identify multiple sets of location-based fixed entities from multiple sets of location-based fixed entities associated with the identified specific location; An ensemble prediction model is used to identify the set of location-based fixed entities most relevant to the user from multiple identified sets of location-based fixed entities; and Determine the confidence score for the location-based set of entities for the specific location.

Images