APPLICATION STORE WITH GENERATIVE MODELS FOR APPLICATION DISCOVERY

Abstract

In one example, a computing system comprises a memory that stores instructions and a processing circuit that executes the instructions to: generate one or more user interface elements for an application store, wherein the application store hosts a variety of applications and each application is associated with corresponding application information; generate intermediate application information by the computing system by providing at least the application information as input to a first machine learning module, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing; generate filtered application information by the computing system by providing at least the intermediate application information as input to a second machine learning module;Receiving a request for one or more applications by the computing system; and in response to receiving the request, outputting the filtered application information by the computing system, along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

Description

APPLICATION DISCOVERY

[0001] This application claims priority over preliminary US patent application No. 63 / 674,428, filed on July 23, 2024, the entire contents of which are incorporated herein by reference. BACKGROUND

[0002] Application stores host applications, allowing users to download and install them on their devices. Some application stores also offer search services to help users find or discover applications among those hosted by the stores. SUMMARY

[0003] In general, various aspects of the techniques described in this revelation are directed at application stores that use generative models, such as large language models (LLMs), for application discovery. Application stores, sometimes called "app" stores, can contain a vast library of applications (e.g., tens of thousands, hundreds of thousands, or millions). Because of the sheer number of available applications, users may find it difficult to discover (e.g., locate and / or download) specific applications. In some cases, users are unsure how to search for a particular application (e.g., which search terms or keywords to use), and when they view the application listing, it is unclear to them what features a particular application offers and / or how one application differs from another.

[0004] Some systems allow users to search for applications, such as through keyword searches based on user-entered search terms. Even if the search results (e.g., listings of applications) include applications that match the user's search terms, these systems can still present an overwhelming number of titles. The systems may allow users to browse individual listings of applications within the search results to uncover information provided by developers about a particular application; however, this requires examining each application individually, and the information provided for each application may be insufficient for users to decide which application to download.

[0005] According to the techniques disclosed here, a computing system can provide an application store that uses generative models for application discovery. The application store can apply one or more generative models (e.g., LLM) to generate filtered application information for one or more applications, providing insights into the applications. For example, the filtered application information can identify the functionality provided by the applications, identify and summarize highlights (e.g., good ratings) and / or lowlights (e.g., bad ratings) of the applications, generate answers to questions about the applications for a display page or other user interface element, and provide brief summaries of developer or other information about the applications or various subsets thereof.

[0006] The application store can provide several types of filtered application information for an application. For example, the application store can identify features and summarize highlights for the application. The application store can present the filtered application information in a specific format (e.g., natural language summary, question and answer, or feature list). By providing the filtered application information, the application store enables users to discover applications that meet their needs, even if the application store contains numerous similar applications (e.g., applications within the same or a similar category). Although described herein, in some examples, particularly with regard to LLM, the application store may use various generative or other machine learning (ML) models.

[0007] In one example, various aspects of the techniques are directed toward a procedure that includes: generating one or more user interface elements for an application store by a computing system, wherein the application store hosts a multitude of applications and wherein each application from the multitude of applications is associated with corresponding application information; generating intermediate application information by the computing system by providing at least the application information as input for an initial machine learning module that runs on the computing system, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing;Generating filtered application information by providing at least the intermediate application information as input to a second machine learning module running on the computing system; receiving a request for one or more applications through the computing system; and, in response to receiving the request, outputting the filtered application information through the computing system, along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

[0008] In another example, various aspects of the techniques are directed at a computing system comprising: a memory that stores instructions and a processing circuit that executes the instructions to: generate one or more user interface elements for an application store by a computing system, wherein the application store hosts a multitude of applications and wherein each application from the multitude of applications is associated with corresponding application information; generate intermediate application information by providing at least the application information as input to an initial machine learning module that runs on the computing system, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing;Generating filtered application information by providing at least the intermediate application information as input for a second machine learning module running on the computing system; receiving a request for one or more applications; and, in response to receiving the request, outputting the filtered application information along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

[0009] In another example, various aspects of the techniques are directed at non-transient, computer-readable storage media, comprising instructions which, when executed by the processing circuit, cause the processing circuit to: generate one or more user interface elements for an application store by a computing system, wherein the application store hosts a plurality of applications and wherein each application from the plurality of applications is associated with corresponding application information; generate intermediate application information by providing at least the application information as input to a first machine learning module executed on the computing system, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing;Generating filtered application information by providing at least the intermediate application information as input for a second machine learning module running on the computing system; receiving a request for one or more applications; and, in response to receiving the request, outputting the filtered application information along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

[0010] The details of one or more examples are set out in the accompanying drawings and the description below. Further features, functions, and advantages will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a conceptual diagram illustrating an exemplary environment for an application store using generative models for application discovery according to one or more aspects of the present disclosure. Fig. Figure 2 is a block diagram illustrating an exemplary environment for an application store using generative models for application discovery according to one or more aspects of the present disclosure. Fig. Figure 3 is a block diagram illustrating an example of a machine learning (ML) module according to one or more aspects of the present disclosure. Fig. 4A is a block diagram illustrating a first example of a mode of discovery according to one or more aspects of the present revelation. Fig. Figure 4B is a block diagram illustrating an exemplary computing device according to one or more aspects of the present disclosure. Fig. Figure 5A is a block diagram illustrating a first example of an application store according to one or more aspects of the present disclosure. Fig. Figure 5B is a block diagram illustrating a first example of an application store according to one or more aspects of the present disclosure. Fig. Figure 6 is a flowchart illustrating an exemplary process for an application store using generative models for application discovery according to one or more aspects of the present disclosure. DETAILED DESCRIPTION

[0011] Fig. Figure 1 is a conceptual diagram illustrating an exemplary environment for an application store using generative application discovery models according to one or more aspects of the present disclosure. As shown in the example in Fig. As can be seen in Figure 1, the environment 100 can include one or more computing devices 120A to 120N (collectively referred to as "computing devices 120") that can communicate with computing device 110 via the network 102. In some examples, computing devices 120 and computing device 110 can be peer devices operating in a client / server configuration. For example, computing devices 120 can be clients used to access services such as application store services (e.g., application search services, download services) of an application store 140 provided by computing system 110.

[0012] Thus, the computing device 120 can enable users to interact with an application store 140 provided by the computing system 110. Computing devices 120 can be, for example, a smartphone, a tablet computer, a laptop computer, a desktop computer, a portable device, a gaming system, a media player, an e-book reader, a camera device, or a body-worn computing device (e.g., a computerized watch, computerized glasses, etc.). Fig. Figure 1 illustrates a specific example of a computing device 120, and many other examples of the computing device 120 can be used in other cases and may include a subset of the components included in the exemplary computing device 120, or may include additional components that are in Fig. 1 are not shown.

[0013] The computing device 120 can include an operating system that provides an execution environment for the store client 128, one or more local applications 125, or both. Local applications 125 can be applications 135 that have been downloaded from the computing system 110, such as via the application store 140, and that are installed on and / or run by the computing device 120. Examples of applications 135 include social networking applications, utilities, productivity applications, entertainment applications, creative applications, communication applications, shopping applications, games, and other software applications.

[0014] The store client 128 can interact with the application store 140 of the computing system 110 and perform other functions associated with the applications 135 or the local applications 125, including downloading applications 135, installing applications 135, updating local applications 125, and deleting local applications 125 or various subsets thereof. For example, in Fig. As shown in Figure 1, the store client 128 can present a representation of the application store 140, for example, by presenting one or more user interface elements that constitute the representation of the application store 140 or a part thereof (e.g., a page, a screen) to the computing device 120 via the user interface device 121. The application store 140 can contain information about applications 135 that are available to the computing device 120 via the application store 140, such as for downloading and / or installing them.

[0015] The Application Store 140 can present the Applications 135 in various ways, such as on a page, a screen, or another element of the Application Store 140 user interface. As in Fig. As shown in Figure 1, the application store 140 can present the applications 135 in an application listing containing one or more applications 135. In some examples, the application store can host a variety of applications 135A to 135N (collectively referred to as "applications 135").

[0016] Applications (135) can be hosted together with application information (142A to 142N) and filtered application information (144). Application information (142) can be information about applications (135) from one or more sources (e.g., application developers, users). Examples of application information include application short descriptions (e.g., 60 characters or less) and / or long descriptions (e.g., more than 60 characters), application names, application categorizations (e.g., productivity, messaging, social networking, games), application reviews, articles, social media posts, screenshots, icons, promotional content, application summaries, application ratings, and feature descriptions.

[0017] As further described below, instead of receiving filtered application information 144, the computing system 110 can generate filtered application information 144 based on application information 142. In general, the computing system 110 can generate and present filtered application information 144 to facilitate application discovery on computing devices 120. For example, the computing system 110, such as through the discovery module 134, can generate filtered application information 144, including summaries, feature enumerations (e.g., feature lists), or questions and answers in natural language format (e.g., English sentences / phrases). Although in the example from Fig. 1 together with the application listing of the application store 140, filtered application information 144 can initially be hidden and revealed in response to user input, such as a toggleable (e.g. collapsible) element of the user interface.

[0018] The computing system 110 can call the discovery module 134 to generate filtered application information 144 based on the application information 142. This filtered information provides insights into one or more of the applications 135, such as by summarizing features provided by the applications, summarizing highlights (e.g., good ratings) and / or lowlights (e.g., bad ratings) of the applications, presenting questions and answers about the applications 135, and providing brief summaries of the information provided by the application developer (e.g., descriptions of the application developer) or various subsets thereof. The discovery module 134 can generate filtered application information 144 so that differences between the applications 135, including similar applications 135 (e.g., applications within the same category, subcategory, or both), are easily identifiable.

[0019] For example, the discovery module 134 can generate filtered application information 144, including feature summaries or feature enumerations for applications 135. By providing filtered application information 144, the computing system 110 can provide a compressed (e.g., shortened, truncated, abbreviated) version of the application information 142 about one or more of the applications 135, offering insights into the applications. Accordingly, the filtered application information 144 can include insights into the applications 135 and requires less storage space compared to the application information 142. By publishing (e.g., making available) applications 135 along with filtered application information 144 via the application store 140, the computing system 110 increases the likelihood that the applications 135 will be downloaded and / or installed by users.Filtered application information 144 also helps the user to decide which application 135 is suitable for their use case.

[0020] The computing device 120 can receive user input (e.g., requests) for the store client 128, local applications 125, or both, and present the output of the store client 128, local applications 125, or both through one or more user interface devices 121 of the computing device 120. The user interface device 121 of the computing device 120 can be hardware that functions as an input and / or output device for the computing device 120. For example, the user interface device 121 can include a display component (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED) display), which can be a screen on which information from the user interface device 121 is displayed, and a presence-sensitive input device that can detect an object on and / or near the display component.The presence-sensitive input device can, for example, detect touch or other user input. The user interface 121 can provide tactile, audio, and video outputs. In some examples, the user interface device 121 includes one or more presence-sensitive displays, speakers, liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, haptic motors, linear actuators, or any other type of device for receiving input or generating output for a human or machine.

[0021] The Store Client 128 can receive user input in various forms (e.g., text input, touch gestures, and other input), such as via the User Interface Device 121. In some examples, the Store Client 128 can receive user input including queries (e.g., search terms), application information (e.g., reviews or ratings), or both. The Store Client 128 can send at least some of the user input to the Computing System 110, and the Computing System 110 can forward such user input (e.g., in anonymized or aggregated form) for purposes such as improving application recognition or training machine learning models.

[0022] The computing system 110 can be any suitable computing system, such as one or more desktop computers, laptop computers, mainframes, servers, cloud computing systems, virtual machines, etc., capable of sending and receiving information over the network 102. In some examples, the computing system 110 can be a cloud computing system that provides one or more services over the network 102. That is, in some examples, the computing system 110 can be a distributed computing system. One or more computing devices, such as the computing devices 120, can access the services provided by the cloud by communicating with the computing system 110. Fig. Figure 1 illustrates only one specific example of the Computing System 110, and many other examples of the Computing System 110 can be used in other cases and may include a subset of the components included in the exemplary Computing System 110, or may include additional components that are in Fig. 1 are not shown.

[0023] The computing system 110 can include a store module 138, which the computing system 110 can call to provide an application store 140. For example, the store module 138 can provide an application store by providing one or more application store services, such as for the store client 128 of the computing devices 120. The store module 138 can publish applications 135 developed by application developers, so that the computing devices 120 can download the applications 135 for installation. Once the application 135 is installed on the computing devices 120, it can be represented as a local application 125, as in the example from Fig. 1 can be seen. In some examples, the Store module 138 can process payment information and / or authorizations from the computing devices 120 to, for example, enable the purchase before downloading and installing paid applications 135 or paid features of applications 135.

[0024] The Store module 138 can publish application listings for applications 135, which, in addition to the filtered application information 144, can also include application information 142 about applications 135. Each of the applications 135 can have its own application information, which includes technical information about features, functionality, or other properties of the application 135, promotional content to advertise, promote, market, sell, or otherwise encourage users to download and / or install the application 135, or other information about the application 135.

[0025] The Store Module 138 can receive applications 135 and application information from various sources. For example, the Store Module 138 can receive applications 135 uploaded by application developers and application information 142 provided by application developers. For example, the Store Module 138 can receive application 135A and the application information 142 for application 135A from a first application developer and application 135N and the application information 142 for application 135N from an nth application developer. The Store Module 138 can receive a variety of application information 142 provided by the developer, including the application developer's long and / or short text (e.g., summary) promoting or otherwise describing the application 135 and / or its features, one or more screenshots, icons, or other images, and one or more categorizations (e.g.,Tags) that specify one or more categories (e.g., productivity, messaging, social networks, gaming) to which application 135 should be assigned.

[0026] The store module 138 can receive application information 142 from various sources, including application developers. For example, the store module 138 can receive application information 142, such as application feedback (e.g., user ratings) from one or more users of the applications 135. The store module 138 can receive application feedback for the applications 135 from the computing devices 120, such as via the store client 128 of the computing devices 120. The store module 138 can store the applications 135 and the application information 142 received by the store module 138 on one or more storage devices, such as one or more storage devices accessible to the computing system 110.

[0027] The discovery module 134 can generate filtered application information 144 based on the application information 142 for the purpose of application discovery. The discovery module 134 can use (e.g., process) application information 142, which can contain a variety of information about applications 135. The discovery module 134 can use application information 142 in various formats, including tabular (e.g., CSV), indexed (e.g., attribute-value storage of a database or other structured data format), graphical (e.g., images), natural language (e.g., English), or other formats suitable for storing information about applications 135 that the discovery module 134 can use (e.g., parse and process).

[0028] The discovery module 134 can obtain application information 142 from various sources. For example, the discovery module 134 can obtain application information 142 from the repository 133. As described above, the repository 133 can store application information 142 that the store module 138 obtains from application developers, users, or other sources. In some examples, the discovery module 134 can obtain application information from a source device connected over the network 102. For example, the discovery module 134 can obtain application information 142 from one or more content servers 104 (e.g., web servers) that store and / or host information about one or more applications 135. In some examples, the content servers 104 can be part of the environment 100, as in Fig. 1 shown. The discovery module 134 can retrieve context-related passages to retrieve application information 142 from content servers 104.

[0029] Application information 142 can include information about a variety of applications 135 from one or more sources. For example, applications 135 can have application information 142 that includes developer-provided information stored in the repository 133, user reviews stored in the repository 133, and articles, ratings, or other information about applications 135 stored by content servers 104, or various subsets thereof. As described above, units of application information 142 can be in various formats (e.g., tabular, indexed, graphical, natural language).

[0030] In some examples, the discovery module can identify application information from a larger dataset. For instance, the discovery module can use information from a series of previous application queries as application information. For example, the discovery module can filter specific queries (e.g., the first 1,000, 2,000, 3,000, ... 10,000 non-navigator queries) to identify a subset of those queries. The discovery module can then use the information resulting from this subset of queries as application information. In one example, the discovery module filters individual queries to generate a subset of queries that, when executed, each return 5 high-quality applications with valid application highlights (e.g., three or more positive reviews) in the top 10 impressions (e.g., the first 10 results).Discovery module 134 can receive user input for filtering queries. For example, discovery module 134 can receive user input indicating whether a query is too broad or whether a query contains sensitive information, both of which would exclude the query from being included in the subset of queries.

[0031] ++As part of scaling application discovery and ensuring the quality and relevance of the generated filtered application information, the Computing System 110 can implement a multi-stage pipeline via the Discovery Module 134 to refine the set of user queries used to generate or contextualize filtered application information. This process aims to identify a high-quality corpus of queries for efficient and effective information generation.++

[0032] Initially, a large number of queries can be identified, such as the first "X" most popular or relevant queries (e.g., the 1,000, 2,000, 3,000 or more non-navigation-related queries). This initial set can then undergo a series of progressive filtering stages.

[0033] The initial set of queries can first be subjected to automatic filtering rules. These rules, which can be stored in rule store 356, can exclude queries based on various qualification criteria, such as being too broad, containing sensitive information, or failing to meet preliminary quality metrics. An example of a quality metric might be that, when executed, the query returns at least a minimum number of high-quality applications with valid highlights (e.g., at least five applications with positive reviews) within a top "k" number of impressive positions (e.g., top 10 search results). At this stage, the initial query volume is significantly reduced by automatically identifying and removing less desirable queries.

[0034] Queries that pass the automatic pipeline filtering can subsequently undergo manual review. Human operators or designated reviewers can examine a subset of these queries for qualitative aspects that automated rules cannot readily capture, such as nuances, intent, or potential ambiguity. User input, as described in paragraph

[0038] , can be integrated here, with manual flagging of queries that are "too broad" or "contain sensitive information" leading to their exclusion. This stage provides an additional layer of curation and quality control.

[0035] At a further stage, human evaluators can assess the quality of the generated filtered application information in conjunction with the refined queries. This human evaluation provides critical feedback that may lead to further refinement of the query corpus or the rules used in earlier filtering stages. Filtered application information may only be published if it meets a human evaluation threshold, as described below. This stage serves as a final quality control check and provides valuable data for training and improving the machine learning modules.

[0036] This multi-stage filtering process, which gradually reduces the query corpus, can ensure that the generative models primarily operate on a high-quality and relevant set of queries, which can improve the use of computer resources and the human evaluation effort, while also increasing the overall quality and discoverability of applications.

[0037] The discovery module 134 can generate filtered application information 144 from the application information 142 and store filtered application information 144, such as in the repository 133. The store module 138 can publish an application listing with filtered application information 142 for one or more of the applications 135, along with details about the one or more applications. The store client 128 of the computing device 120 can request the application listing and present it to users. For example, the store module 138 can present filtered application information 144 for one or more of the applications 135, along with details about the one or more applications. As in Fig. Figure 1 shows, for example, application 135A together with application information 142, which provides information about application 135A (e.g., application icon, application title, application description), and filtered application information 144, which was generated for application 135A, which in this example can be a feature summary or a feature enumeration.

[0038] Discovery module 134 can generate filtered application information 144 for application 135A by identifying a subset of application information 142 about application 135A and using that subset to generate filtered application information 144 specific to application 135A. Discovery module 134 can identify the subset of application information 142 from the total corpus of application information 142 in various ways. For example, discovery module 134 can select developer-provided application information, user ratings, and / or user reviews for application 135A from repository 133 by selecting, or specifying, the application information in repository 133 that is assigned to application 135A.As another example, the discovery module 134 can apply one or more ML modules 136A to 136N (collectively “ML modules 136”) to identify the subset of application information 142, such as by classifying the application information 142 or parts thereof as containing or not containing the information about application 135A.

[0039] The Discovery Module 134 can generate various types of filtered application information 144. For example, the Discovery Module 134 can generate filtered application information 144 that includes a summary of the application information 142. For example, the Discovery Module 134 can generate filtered application information 144 that includes a summary of the application developer's application information, such as a long or short form of the application description 135A provided by the application developer. As another example, the Discovery Module 134 can generate filtered application information 144 that includes a summary of the application's highlights and / or lowlights 135A.For example, to generate filtered application information 144 that includes a summary of highlights, the discovery module 134 can generate filtered application information 144 that includes a summary of one or more positive user reviews. Similarly, to generate filtered application information 144 that includes a summary of lowlights, the discovery module 134 can generate filtered application information 144 that includes a summary of one or more negative user reviews. In some examples, the discovery module 134 can generate filtered application information 144 in a question-and-answer format, where the discovery module 134 generates filtered application information 144 including answers to one or more questions that can be posed via the application 135A.The discovery module 134 can, as further described below, summarize units of application information 142, such as based on content or mood, and generate filtered application information 144 (e.g., highlight / lowlight summaries) based on application information 142 in one or more of the clusters.

[0040] The Discovery Module 134 can generate filtered application information 144 in various formats. For example, the Discovery Module 134 can generate filtered application information 144, including summaries or questions and answers in natural language format (e.g., English sentences / phrases). In some examples, the Discovery Module 134 can generate filtered application information 144 in list format. For example, the Discovery Module 134 can generate filtered application information 144 that includes an enumeration of features (e.g., a feature list) of the application 135A based on application information 142 (e.g., application information provided by the developer). In some examples, the Discovery Module 134 can generate filtered application information 144 in tabular format.For example, the discovery module 134 can generate filtered application information 144 for multiple applications 135 in a table format, in which the respective features or properties form cells of a table that can be compared to identify, for example, differences between the applications 135.

[0041] In some cases, the discovery module 134 can generate filtered application information 144 to improve or replace application information 142. For example, application information 144, such as a short or long form of description provided by an application developer, may be of poor quality (e.g., not descriptive, too short, grammatically incorrect). In these cases, the discovery module 134 can generate filtered application information 144 that includes a higher-quality version of the application information 144 (e.g., a longer description in short form or in long form within the character limits of these descriptions). The discovery module 134 can use the low-quality application information 142 to be replaced, as well as other application information 142, as input to generate filtered application information 144 that replaces the application information 142.

[0042] In some examples, the Store module 138 can identify substandard application information 142 and automatically call the Discovery module 134 to generate filtered application information 144 to replace it. For example, the Store module 138 can determine that short descriptions, long descriptions, or both below a certain length threshold (e.g., 30 characters) are too short to ensure sufficient quality. The Discovery module 134 can receive user input that causes it to generate filtered application information 144 to replace the substandard application information 142. Once the substandard application information 142 has been replaced, the Store module 138 can determine that the application 135A is suitable for ad placement.Application 135A, for example, could be a high-quality application that, with improved application information 142, would be suitable for advertising placement. Once the low-quality application information 142 is replaced by filtered application information 144, the store module 138 can present application 135 in one or more advertising campaigns.

[0043] The discovery module 134 can include one or more machine learning (ML) modules 136A to 136N (collectively referred to as "ML modules 136") and manage their operation to generate filtered application information 144. The ML module 136 can be or include one or more of different types of machine learning models. Examples of such different types of machine learning models are provided below for illustration. Other models beyond those provided below may also be used.

[0044] One or more of the machine learning modules 136 described below can be used alone or in combination to provide (e.g., output) filtered application information 144 for one or more of the applications 135 in response to input data, including application information 142 or information about it. In some examples, one or more of the exemplary machine learning modules 136 can identify a subset of application information 142 that is used to generate filtered application information 144. For example, one or more machine learning modules 136 can be classification models that receive application information 142 as input and output an indication of which parts of the application information 142 contain (e.g., describe) or do not contain (e.g., do not describe) information about the application 135A.

[0045] In some implementations, the ML module 136 may be or include one or more classifier models, such as linear classification models, quadratic classification models, etc. The ML module 136 may also be or include one or more regression models, such as simple linear regression models, multiple linear regression models, logistic regression models, stepwise regression models, multivariate adaptive regression splines, locally estimated scatterplot smoothing models, etc. In some examples, the ML module 136 may be or include one or more generative networks, such as generative adversarial networks. Generative networks can be used to generate new data, such as artificial feedback texts.

[0046] In some examples, the ML Module 136 may be or include one or more artificial neural networks (also simply called neural networks). A neural network can consist of a group of connected nodes, which can also be called neurons or perceptrons. A neural network can be organized into one or more layers. Neural networks that include multiple layers can be called "deep" networks. A deep network may include an input layer, an output layer, and one or more hidden layers positioned between the input and output layers. The nodes of the neural network may be fully connected or not fully connected.

[0047] One or more neural networks can be used to create an embedding within a multidimensional embedding space for individual units of application information (e.g., individual application descriptions, feature descriptions, reviews, ratings, articles, paragraphs, sentences about the application). The embedding can be, for example, a representation of knowledge abstracted from input data, such as a unit of application information (e.g., a part of application information), into one or more learned dimensions. In some cases, embeddings can be a useful source for identifying related units. In some cases, embeddings can be extracted from the network's output, while in other cases, embeddings can be extracted from any hidden node or hidden layer of the network (e.g.,a layer located near the end, but not the last layer of the network).

[0048] For example, the ML module 136 can use an embedding space, such as a multidimensional embedding space, to determine whether units of application information 142 (e.g., user ratings) contain a similar concept. For instance, the ML module 136 can generate an embedding for each unit of application information 142 that specifies a position for each corresponding unit of application information 142 within the embedding space based on one or more concepts or sentiments contained in each corresponding unit of application information 142. The ML module 136 can generate the embeddings such that the distances in the embedding space correspond to the similarity of concepts between the units of application information 142.Thus, units of application information 142 with similar concepts or moods may be separated by smaller distances than units of application information 142 with dissimilar concepts or moods, which may be separated by relatively larger distances.

[0049] The ML module 136 can use these embeddings to cluster units of application information 142 with similar concepts in the embedding space. In some examples, the ML module 136 can include units of application information 142 within a threshold that contain similar concepts or sentiments, and identify units of application information 142 outside the threshold that contain different concepts or sentiments. The ML module 136 can use clusters of application information 142 to generate filtered application information, including rating summaries (e.g., highlights / lowlights). For example, the ML module 136 can select the n largest clusters (e.g., the 3 largest clusters) of application information 142 and generate filtered application information 144 using the respective application information 142 for each cluster.

[0050] In some examples, ML modules 136 can perform or be subjected to one or more reinforcement learning techniques, such as Markov decision processes, dynamic programming, Q functions or Q learning, value function approaches, deep Q networks, differentiable neural computers, asynchronous advantage actor critics, deterministic policy radii, etc. The computing system 110 can generate ML modules 136 using various training techniques, including supervised, unsupervised, semi-supervised, and reinforcement learning techniques, using one or more training datasets, including previous examples of filtered application information. For example, supervised or unsupervised reinforcement learning techniques can be used to generate the ML module 136, which, when applied by the discovery module 134, will provide filtered application information in a selected format (e.g.,The computer system 110 generates filtered application information (summaries, feature lists, questions and answers). During training, the computer system 110 can validate filtered application information generated by the ML module 136 using a validation dataset. This allows the filtered application information generated by the ML module 136 to be compared with previously validated filtered application information, such as summaries, feature lists, or questions and answers, that was validated by human validators. By validating the filtered application information generated by the ML module 136, the computer system 110 can ensure that the ML module 136 is of high quality or at least satisfactory quality before it is deployed for use with the application store 140. The computer system 110 can also validate the ML module 136 as part of a feedback loop that monitors the quality of the output (e.g.,filtered application information 144) of the ML module 136 is continuously improved.

[0051] Fig. Figure 2 is a block diagram illustrating an exemplary environment for an application store using generative models for application discovery according to one or more aspects of the present disclosure. As shown in the example in Fig. As can be seen in Figure 2, the environment 200 can include a computing device 220, which can communicate with the computing device 210 via the network 202. The computing system 210, the computing device 220, and the network 202 consist of Fig. 2 are below referred to as examples of the computer system 110, the computing devices 120 and the network 102, as in Fig. 1 shown, described.

[0052] The Computing System 210 can be any suitable computing system, such as one or more desktop computers, laptop computers, mainframes, servers, cloud computing systems, virtual machines, etc. Fig. Figure 2 illustrates only one specific example of a computing device 210, and many other examples of the computing device 210 can be used in other cases and may include a subset of the components included in the exemplary computing system 210, or may include additional components that are in Fig. 2 are not shown.

[0053] As illustrated by the example from Fig. As shown in Figure 2, the computing system 210 can include one or more processors 212, one or more input devices 214, one or more output devices 216, one or more communication units 218, and one or more storage devices 230. Communication channels 219 can connect any of the components 212, 214, 216, 218, and 230 for cross-component communication (physical, communicative, and / or operational). In some examples, the communication channels 219 can include a system bus, a network connection, a data structure for communication between processes, or any other method for data communication. The storage device 230 of the computing system 210 can include a store module 238 and a discovery module 234. The computing system 210 can call the store module 238 to access an application store, such as the application store 140 from Figure 210. Fig. 1, to provide, and can call the discovery module 234 to generate filtered application information 244 from the application information 242 for the purpose of application discovery. The application information 242 and the filtered application information 244 from Fig. 2 can be examples of the application information 142 or the filtered application information 144, as in Fig. 1 illustrates.

[0054] One or more input devices 214 of the computing system 210 can receive inputs. Examples of inputs are tactile, acoustic, and optical. The input devices 214 of the computing system 210 include, in one example, a presence-sensitive display, a touch-sensitive screen, a mouse, a keyboard, a voice-controlled system, a video camera, a microphone, or any other type of device for detecting inputs from a human or a machine.

[0055] One or more output devices 216 of the computing system 210 can generate an output. Examples of outputs include tactile, acoustic, and visual outputs. The output devices 216 of the computing system 210 include, in one example, a presence-sensitive display, a sound card, a video graphics adapter card, a loudspeaker, a monitor, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other type of device for generating an output to a person or a machine.

[0056] One or more communication units 218 of the computing system 210 can communicate with external devices by transmitting and / or receiving network signals on one or more networks via one or more wired and / or wireless networks. Examples of one or more communication units 218 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, or any other type of device capable of transmitting and / or receiving information. Other examples of one or more communication units 218 may include shortwave radio devices, cellular data radio devices, wireless network radio devices, and universal serial bus controllers (USB controllers).

[0057] Network 202 can be any public or private communications network, such as a cellular network, Wi-Fi®, and / or any other type of network for transmitting data between computer systems, servers, and computing devices. Network 202 can include one or more network hubs, network switches, network routers, or other network equipment that are interconnected to provide information exchange between Computer System 210 and Computing Device 220. Computing Device 220 and Computer System 210 can transmit and receive data over Network 202 using any suitable communication techniques. For example, Computer System 210 and Computing Device 220 can communicate Applications 235A through 235N (collectively referred to as "Applications 235"), Application Information 242, and / or Filtered Application Information 244 over Network 202 (e.g.,(send and receive). The computing device 220 and the computing system 210 can each be operatively coupled to the network 202, using appropriate network connections such as Ethernet, Wi-Fi, BLUETOOTH® or other types of wired and / or wireless network connections.

[0058] The Store module 238 can provide a protocol (e.g., an application programming interface (API)) for communicating requests and responses with the Store client 228. The protocol can include one or more fields for storing filtered application information 244. For example, the protocol can include one or more fields for storing summaries, questions and answers, highlights or lowlights, feature enumerations or feature summaries, or various subsets thereof. In some examples, a field can be a named field. For example, a field for storing filtered application information 244, including question and answer information, can be named "AppQuestionAndAnswerInformation".

[0059] One or more processors 212 can implement functionality and / or execute instructions within the computing system 210. For example, one or more processors 212 of the computing system 210 can receive and execute instructions stored by the one or more storage devices 230 that perform the functionality of the operating system 232, the store module 238, and the discovery module 234. The instructions executed by one or more processors 212 can cause the computing system 210 to store information in one or more storage devices 230 during program execution. Examples of one or more processors 212 include application processors, display controllers, sensor hubs, and any other hardware configured to function as a processing unit.One or more processors 212 can execute instructions from the operating system 232, the store module 238, and the discovery module 234 to perform actions or functions. That is, the operating system 232, the store module 238, and the discovery module 234 can be operated by one or more processors 212 to perform various actions or functions of the computing system 210.

[0060] The Store module 238 and the Discovery module 234 from Fig. 2 are each given as examples for the Store module 138 and the Discovery module 134, as in Fig. 1 illustrates and describes. The Store module 238 can be an application store, such as application store 140 from Fig. 1. Provide by making one or more application store services available to the computing device 220. The store module 238 can publish applications 235, such as in one or more application listings, along with filtered application information 244 for the purpose of application discovery. The store module 238 can publish applications 235 so that the computing device 220 can download applications 235 for installation. Once installed, the applications 235 can be represented as local applications 225. The store module 238 can perform the operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware that resides in and / or runs on the computing system 210.

[0061] One or more storage devices 230 within the computer system 210 can store information for processing during the operation of the computer system 210. That is, the computer system 210 can store data that the operating system 232, the store module 238, and the discovery module 234 can access during execution in the computer system 210, including application information and filtered application information. The computer system 210 can store application information, filtered application information, or both, for example, on one or more storage devices 230 that the discovery module 234, the store module 238, or both can access during execution in the computer system 210.

[0062] In some examples, the storage device 230 is temporary storage; that is, long-term storage is not its primary purpose. One or more storage devices 230 on the computing device 210 may be configured as volatile memory for short-term storage of information and therefore do not retain stored contents when the device is powered off. Examples of volatile memory include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memory known in engineering.

[0063] In some examples, one or more storage devices 230 also include one or more computer-readable storage media. One or more storage devices 230 may be configured to store larger amounts of data compared to volatile memory. One or more storage devices 230 may also be configured for long-term storage of information as non-volatile memory, where information is retained after power cycles. Examples of non-volatile memory include magnetic hard disks, optical media, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). One or more storage devices 230 may store program instructions and / or information (e.g.,Store data) that is assigned to the operating system 232 and the discovery module 234.

[0064] In some examples, the storage device 230 can include a repository 233 that can implement a database, a file format, or another structured data format suitable for storing and retrieving information, such as application information 242, filtered application information 244, or both. The repository 233 consists of Fig. 2 can be an example of a repository 133 as in Fig. 1. Illustrated.

[0065] The store module 238 can store application information 242 in the repository 233 of the storage device 230. The repository 233 can store units of application information 242 (e.g., parts of application information) such that each unit of application information 242 is assigned to at least one of the applications 235. In this way, the repository 233 can retrieve application information 242 that is relevant to a particular application of the applications 235. For example, the store module 238 can receive a unit of application information 242, including a title and description provided by the application developer of application 235A, and / or a user rating of application 235A. The repository 233 can store such units of application information 242 along with an indication of application 235A in order to assign these units of application information to application 235A.Repository 233 can retrieve the application information relevant to application 235A (e.g., assigned to application 235A) by querying the application information using the specification of application 235A.

[0066] Similarly, the discovery module 234 can store filtered application information 144, generated by the discovery module 234, in the repository 233. The repository 233 can store units of filtered application information 244 (e.g., parts of filtered application information) such that each unit of filtered application information 244 is assigned to at least one of the applications 235. In this way, the repository 233 can retrieve filtered application information 244 that is relevant to a specific application of the applications 235. For example, the store module 238 can receive a unit of filtered application information 244 that includes a summary or feature enumeration generated by the discovery module 234.Repository 233 can store such units of filtered application information 244 together with a reference to application 235A in order to assign these units of filtered application information 244 to application 235A. Repository 233 can retrieve filtered application information 244 relevant to application 235A (e.g., assigned to application 235A) by querying filtered application information 244 using the reference to application 235A.

[0067] The discovery module 234 can call one or more ML modules 236A to 236N (collectively, "ML modules 236") to generate at least filtered application information 244. One or more ML modules 236 can be used alone or in combination to provide (e.g., output) filtered application information 244 for one or more of the applications 235 in response to input data, including application information 242 or information about it. As further described below, the ML modules 236 can include one or more ML models, including generative models (e.g., LLM). In some examples, one or more ML modules 236 can identify a subset of application information 244 to be used to generate filtered application information 244.For example, one or more ML modules 236 can be classification models that receive application information 242 as input and output information about which units of the application information 242 contain or do not contain information about the application 235A (e.g., information describing the application 235A). In some examples, the storage device 230 can store one or more ML modules 236, such as together with or as part of the discovery module 234.

[0068] The computing devices 220 can be, for example, a smartphone, a tablet computer, a laptop computer, a desktop computer, a portable device, a gaming system, a media player, an e-book reader, a camera device, or a body-worn computing device (e.g., a computerized watch, computerized glasses, etc.). Fig. Figure 2 illustrates a specific example of the computing device 220, and many other examples of the computing device 220 can be used in other cases and may include a subset of the components included in the exemplary computing device 220, or may include additional components that are in Fig. 2 are not shown.

[0069] The computing device 220 includes one or more user interface devices 221, one or more processors 222, one or more storage devices 226, and one or more communication units 224. One or more storage devices 226 of the computing device 220 may contain an operating system that provides an execution environment for the store client 228, one or more local applications 225, or both. As described above, local applications 225 may be applications 235 that have been downloaded from the computing system 210 and are installed on and / or run by the computing device 220.

[0070] The user interface device 221 of the computing device 220 can be hardware that functions as an input and / or output device for the computing device 220. For example, the user interface device 221 can include a display component (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED) display), which can be a screen on which information from the user interface device 221 is displayed, and a presence-sensitive input device that can detect an object on and / or near the display component. The presence-sensitive input device can, for example, detect touch or other user input. The user interface device 221 can provide tactile, audio, and video output.The user interface device 221 includes, in some examples, a presence-sensitive display, a loudspeaker, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, haptic motors, linearly actuated devices, or any other type of device for receiving inputs and generating outputs to a human or a machine.

[0071] One or more communication units 224 of the computing device 220 can communicate with external devices by sending and / or receiving communication signals, such as via one or more wireless networks or wireless connections. Examples of one or more communication units 224 include a network interface card (such as an Ethernet or Wi-Fi card), an optical transceiver, a radio frequency transceiver, a global positioning system (GPS) receiver, or any other type of device capable of sending and / or receiving information. Other examples of communication units 224 may include shortwave radios, cellular data radios, wireless network radios, and universal serial bus (USB) controllers, or any other type of device capable of sending and / or receiving information via a wired or wireless connection.

[0072] One or more processors 222 can implement functionality and / or execute instructions within the computing device 220. For example, one or more processors 222 on the computing device 220 can receive and execute instructions stored by one or more storage devices 226, which perform the functionality of the store client 228 and the local applications 225. The instructions executed by one or more processors 222 can cause the computing device 220 to store information in one or more storage devices 226 during program execution. Examples of one or more processors 222 include application processors, display controllers, sensor hubs, and any other hardware configured to act as a processing unit.

[0073] One or more storage devices 226 within the computing device 220 can store information for processing during the operation of the computing device 220. That is, the computing device 220 can store data that the store client 228 accesses while running on the computing device 220, including local applications 225, application information, and other data or various subsets thereof. In some examples, the storage device 226 is temporary storage; that is, long-term storage is not the primary purpose of the storage device 226. One or more storage devices 226 on the computing device 220 can be configured as volatile storage for short-term information storage, and therefore, stored content is not retained when the device is powered off.Examples of volatile memory include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memory known in engineering.

[0074] In some examples, one or more storage devices 226 also include one or more computer-readable storage media. One or more storage devices 226 may be configured to store larger amounts of data compared to volatile memory. One or more storage devices 226 may also be configured for long-term storage of information as non-volatile memory, where information is retained after power cycles. Examples of non-volatile memory include magnetic hard disks, optical media, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). One or more storage devices 226 may store programming instructions and / or information (e.g.,Store data associated with the Store client 228 and local applications 225.

[0075] In some examples, one or more storage devices 226 can store an operating system, a store client 228, and one or more local applications 225. The compute device 220 and / or the operating system can provide an execution environment for the store client 228, which can send and receive information from the compute system 210 to access the application store provided by the compute system 210. In some examples, the store client 228 can communicate with the store module 238 of the compute system 210 to request and receive listings from the application store, including one or more applications, filtered application information 244, and application information 242. The store client 228 can generate data for output to a graphical user interface, including one or more user interface elements that display the application listings, such as through the user interface device 221.

[0076] The store client 228 can perform the operations described herein by using software, hardware, firmware, or a mixture of hardware, software, and firmware located in and / or running on the computing device 220 to interact with the application store of the computing system 210 and with other functions associated with applications 235 or local applications 225, including installing applications 235 and / or updating or deleting local applications 225. The computing device 220 can run local applications 225 with multiple processors or multiple devices, as virtual machines running on underlying hardware, as one or more services of an operating system or computing platform, and / or as one or more executable programs on an application layer of a computing platform of the computing device 220.

[0077] Fig. Figure 3 is a block diagram illustrating an example of an ML module according to one or more aspects of the present disclosure. Fig. 3 is referred to below in connection with Fig. 1 described. The ML module 336 from Fig. 3 is below used as an example for the ML module 136. Fig. 1 described. The ML module 336 includes the language model module 354, the training module 352, and the rule memory 356. The ML module 336 can generate filtered application information 344, which can be an example of filtered application information 144, as described in Fig. Figure 1 shows that the Language Model Module 354 can contain one or more large language models (LLMs).

[0078] In general, the Language Model Module 354 can accurately perform natural language processing (NLP) tasks, such as generating text and other content. However, with regard to certain content (e.g., specific types of information), the Language Model Module 354 may have an increased probability of generating incorrect or inaccurate information. To address the problem of generating incorrect information, the Language Model Module 354 can be configured to exclude the generation of content relating to a set of excluded information types. This set of excluded information types might include, for example, advertising material, telephone numbers, addresses, web addresses, and so on. Thus, the Language Model Module 354 can process application information 142 under certain conditions or "rules" that can be stored in the rule memory 356.The rules can also be text inputs, such as "What are the main features of the application?", "Avoid advertising in the summary", and "Keep the summary short". In this way, the rule store 356 can store a large number of rules (e.g., text inputs) that further define how filtered application information 244 is to be generated by the language model module 354.

[0079] Therefore, the language model module 354 can be applied to structured text (e.g., application information) according to one or more predefined rules stored in the rule store 356. These rules may include, for example, one or more prohibited terms, topics, or lengths for filtered application information 344 generated by the ML module 336. The rules may also include other qualification attributes or instructions that the ML module 336 uses to generate filtered application information 344. Because the language model module 354 can interpret the rules along with the structured text, the ML module 336 can provide a more accurate and user-friendly summary of one or more parts of a comment section in a user application.

[0080] The ML module 336 can include a training module 352, which trains the language model module 354 (e.g., pre-training, fine-tuning, etc.). The training module 352 can pre-train the language model module 354 using a large and diverse text corpus. This dataset can cover a wide range of topics and domains to ensure that the language model module 354 learns various linguistic patterns and contextual relationships. The training module 352 can train the language model module 354 to optimize an objective function. The objective function can be or include a loss function, such as a cross-entropy loss function, which compares the output data generated by the model from the training data with labels associated with the training data (e.g., ground truth labels) (e.g., determines a difference between them).The objective function of language model module 354 can, for example, be to correctly predict the next word in a sequence of words or to fill in missing words as accurately as possible.

[0081] In some examples, training module 352 can continuously or periodically train language model module 354. In some examples, training module 352 can fine-tune language model module 354 by using feedback during the training process. For example, the store module 138 can... Fig. 1. User input is received via the application store 140. This user input can be feedback (e.g., thumbs up, thumbs down, application download / installation or lack thereof, etc.) relating to one or more of the applications 135 presented with filtered application information 344. In some examples, the feedback can indicate whether the filtered application information 344 is accurate or inaccurate, correct or incorrect, of high or low quality, results in an application download / installation, does not result in an application download / installation, etc. The store client 128 can receive the feedback. Additionally, the store client 128 can send the feedback to the store module 138 via the network 102. The training module 352 of the ML module 336 can receive the feedback, and the training module 352 uses the feedback for training purposes.For example, the training module 352 can convert the feedback into labeled data for supervised training. Additionally or alternatively, the training module 352 can fine-tune the language model module 354 by monitoring the relationship between the performance of the language model module 354 and the feedback, and repeating the fine-tuning process as needed (e.g., to receive more positive and less negative user feedback). In this way, the techniques of this revelation can establish a feedback loop that continuously improves the quality of the output (e.g., filtered application information 344) of the ML module 336.

[0082] Although described with reference to the language model module 354, which may include one or more LLMs, the ML module 336 may include other modules and / or models, including various classification models (e.g., Naive Bayes, k-nearest neighbors, support vector machines (SVMs), neural networks) or generative models (e.g., diffusion, generative adversarial networks (GANs)).

[0083] As described above, the Discovery Module 134 can identify application information 142 from a larger dataset. In some examples, the Discovery Module 134 can identify application information 142 by identifying information in the larger dataset based on representativeness, usefulness, and uniqueness. The Discovery Module 134 can call the Machine Learning Module 336 with rules to identify such application information 142, such as by excluding other application information 142 in the output of the Machine Learning Module 336. Examples of such rules include: "Add representative information to the summary," "Add useful information to the summary," and "Add unique information to the summary."

[0084] Fig. 4A is a block diagram illustrating a first example of a mode of discovery according to one or more aspects of the present revelation. Fig. 4A is referred to below in connection with Fig. 1 described. Discovery Module 434 is described below as an example of Discovery Module 134. Fig. 1 described. The discovery module 434 includes one or more ML modules 436A to 436N (collectively, "ML modules 436"), which can be examples of ML modules 136. The discovery module 434 can generate filtered application information 444 using application information 442 as input. The application information 442 and filtered application information 444 can be examples of application information 142 and filtered application information 144, respectively. As described above, the discovery module 434 can receive application information 442 from various sources (e.g., from the repository 133 or the content server 104).

[0085] The discovery module 434 can call one or more machine learning modules 436 to generate filtered application information 444. The discovery module 434 can call a specific machine learning module from among the machine learning modules 436 to generate filtered application information 444. For example, using inputs that include application information 442, the discovery module 434 can call machine learning module 436A to generate filtered application information 444. One or more of the machine learning modules 436 can generate specific types of filtered application information 444. For example, the discovery module 434 can call machine learning module 436A to generate a summary (e.g., application description summary, highlights / lowlights summary), a feature enumeration, and questions and answers, or various subsets thereof, using the application information 442 as input for machine learning module 436A.In some examples, the discovery module 434 can call one or more ML modules 436 to identify (e.g., classify) application information 142 from a larger dataset.

[0086] The discovery module 434 can generate various types of filtered application information 444 in different parts (e.g., pages) of the application store 140. For example, the discovery module 434 can generate query-level information, including filtered application information 444, for an application listing (e.g., a page of search results or featured applications) that includes a large number of different applications 135. The discovery module 434 can also generate application-level information, including filtered application information 444, for a specific application 135. For example, the discovery module 434 can generate application-level information for a part (e.g., a page) of the application store 140 that is intended for the specific application 135A.

[0087] In some examples, the discovery module 434 can call ML modules 436 to generate filtered application information 444 that corresponds to application-level and query-level information. For example, a first ML module 436A can generate application-level information in the form of filtered application information 444, including summaries or certain types of summaries (e.g., description summaries or review summaries), and a second ML module 436N can generate query-level information in the form of filtered application information 144, including feature enumerations.

[0088] The discovery module 434 can manage ML modules 436, such as by selecting one or more ML modules 436 to generate filtered application information 144. For example, the discovery module 434 can select and call the first ML module 436A to generate a summary of an application (e.g., a description summary), and select and call the second ML module 436N to generate a feature enumeration for the application. The discovery module 434 can select the first ML module 436A when generating filtered application information 444 for application-level information, and select the second ML module 436B to generate filtered application information 444 for query-level information. The discovery module 434 can provide inputs, such as application information 442, to one or more ML modules 436 and receive outputs in the form of filtered application information 444 from the ML modules 436.

[0089] Fig. 4B is a block diagram illustrating an exemplary computing device according to one or more aspects of the present disclosure. Similar to Fig. 4A will Fig. 4B below in connection with Fig. 1 described. As from the example in Fig. As can be seen in Figure 4B, the discovery module 434 can include a pipeline 462 of ML modules 436. For example, the discovery module 434 can comprise a multitude of interconnected ML modules 436, where at least one of the ML modules 436 uses the output of another ML module 436 as its input. As in the example of Fig. As shown in Figure 4B, a first ML module 436A can generate output that is used by a second ML module 436B, and the second ML module 436B can generate output that is used by a third ML module 436C, and so on and so forth up to an nth ML module 436N. Even if in Fig. Figure 4B illustrates four ML modules 436; however, fewer or additional ML modules 436 can also form a pipeline of ML modules 436. In some examples, the discovery module 434 can call one or more ML modules 436 in parallel (e.g., simultaneously). The discovery module 434 can call the ML modules 436 in different sequences.

[0090] One or more ML modules 436 of the pipeline 462 can use intermediate information (e.g., information generated by another ML module 436) as input. For example, a first ML module 436A can extract information from the application information 442 (e.g., summarize application information 442 about an application) to generate intermediate application information, and a second ML module 436B can use the intermediate application information to generate filtered application information 444. By summarizing the application information 442, the first ML module 436A can provide a condensed version of the application information 442 when it generates the intermediate application information.

[0091] ML modules 436 can apply various rules to extract information from a larger corpus of information. For example, the first ML module 436A can apply one or more rules, such as one or more rules from rule memory 356. Fig. 3, which prohibit the inclusion of advertising content in the intermediate application information. For illustration, the first ML module 436A can generate intermediate application information that contains a compressed version of the application information (e.g., a summary) without any advertising content. The second ML module 436B can generate filtered application information 444 that does not contain advertising content by generating filtered application information 444 from the intermediate application information.

[0092] The pipeline 462, with the first ML module 436A and the second ML module 436, can, as described above, provide advantages over using a single ML module 436 to generate filtered application information 144 from application information 142. For example, the first ML module 436A generates the information about the intermediate application, which can be stored for later use, such as in repository 133. Repository 133 can associate individual units of information about intermediate applications with the respective applications 135. For example, repository 133 can store a unit of information about an intermediate application along with a reference to the application 135 that describes the unit of information about the intermediate application. The discovery module 434 can generate filtered application information 444 using intermediate application information instead of raw application information 142.For example, to generate filtered application information 444 for a large number of applications 135, the second ML module 436B can retrieve and use the intermediate application information for each individual application 135, instead of using the raw application information 442 of each individual application 135.

[0093] A single ML module 436 may be limited to processing contexts of a certain length (e.g., 4,000 or 8,000 tokens), for example, based on the computing capacity of the computing system 110, which can be quickly saturated by processing raw application information 142. In contrast, the intermediate application information of the pipeline 462 provides a compressed version of the application information 142, thereby saving computing resources for processing the context and enabling the second ML module 436B to generate filtered application information 444 for multiple applications 135, including comparing multiple applications without excessive use of computing resources, using the intermediate application information for multiple applications 135.

[0094] In another example, pipeline 462 can include a first ML module 436A that generates intermediate application information, as described above. In this example, pipeline 462 can include a second ML module 436B. The second ML module 436B can use the intermediate application information generated by ML module 436A to generate filtered application information 444 that represents query-level information. For example, the second ML module 436B can use the intermediate application information generated by ML module 436A to generate filtered application information 444, including feature enumerations for one or more applications 135, for use as part of the search results.

[0095] In some cases, query-level information may depend on application-level information, while application-level information may not depend on query-level information. This attribute of query-level and application-level information can cause application store 140 to present inconsistent filtered application information 444. For example, application store 140 may present query-level information for application 135A, such as on a search results page. If the query-level and application-level information do not match, application store 140 may present filtered application information 444 on the page dedicated to application 135A (e.g., application-level information) that does not match the filtered application information 444 for application 135A on the search results page (e.g.,Information at query level).

[0096] In some examples, pipeline 462 may include a third ML module 436C to ensure consistency between outputs (e.g., filtered application information) generated by other ML modules 436. Continuing the previous example, the third ML module 436C may apply one or more rules, such as the rules from rule store 356. Fig. 3. To ensure that the application-level information is consistent with the query-level information. In some examples, the third ML module 436C can take the query-level information (along with the application information 442) as input and generate application-level information in such a way that the application-level information and the query-level information are consistent. An example rule that ensures consistency of application-level information might be: "Summarize the application in accordance with the query-level information."

[0097] The discovery module 434 can use zero-shot prompts (e.g., rules without examples or demonstrations) as rules in ML modules 436. Continuing the example above, for instance, the first ML module 436A, the second ML module 436B, and the third ML module 436C can each apply rules that include zero-shot prompts. To illustrate, the first ML module 436A can apply a rule such as "Summarize the application information but exclude advertising content," the second ML module 436B can apply a rule such as "List the application features using the intermediate application information," and the third ML module 436C can apply a rule such as "Summarize the application according to the query-level information." As can be seen, a rule can refer to the application information 442, intermediate application information, or other information.The discovery module 434 can provide the referenced information to the relevant ML module. For example, the discovery module 434 can provide application information 424 as input to the first ML module 436A so that the first ML module 436A can apply the exemplary rule mentioned above (namely, "Summarize the application information, but exclude advertising content.") and provide intermediate application information as input to the second ML module 436B so that the second ML module 436B can correctly apply the exemplary rule mentioned above (namely, "List the application's features based on the intermediate application information").

[0098] The Discovery module 434 can generate multiple units of filtered application information 444 for a given application out of applications 135. For example, the Discovery module 434 can create multiple summaries of application descriptions, such as for use as query-level information. The Store module 138 can select a default unit of filtered application information 444 (e.g., a default summary) to display for a given application out of applications 135. The Store module 138 can select the filtered application information 444 resulting from a particular query (e.g., the most popular query) as the default filtered application information 444.

[0099] The discovery module 434 can output evaluation information 464 in some examples, such as via one or more output devices 216. Fig. 2. The rating information 464 can be used for the human evaluation of filtered application information 444 generated by the discovery module 434. For example, the discovery module 434 can generate rating information 464 that includes citations and / or explanations of information in filtered application information 444. The rating information 464 can include a description of the coverage (e.g., breadth) of the filtered application information 444, the distinctiveness of the filtered application information 444, or both. A human evaluator can use the rating information 464 to validate the filtered application information 444. In some examples, the application store 140 may not publish filtered application information 444 until it has been validated by a human evaluator.The Discovery module 434, the Store module 138, or both can receive input from a human evaluator indicating whether certain filtered application information 444 has been validated or not.

[0100] In some examples, the discovery module 434 can use the evaluation information 464 to generate filtered application information 444. For example, the discovery module 434 can use explanations, coverage, and distinctive information from the evaluation information 464 to trigger the "chain of thought" effect for LLM from ML modules 436, resulting in a better basis for generating filtered application information 444 and higher quality filtered application information 444.

[0101] As described above, the discovery module 434 can generate filtered application information 444 and store filtered application information 444, for example in a repository 133, so that it can be retrieved and presented by the storage module 138 in the application store 140. The discovery module 434 can selectively present filtered application information 444 for a subset of applications 135. For example, the discovery module 434 can generate filtered application information 444, for example based on application information 442 that corresponds to the search results of certain queries, such as popular queries received by the store module 138 (e.g., top 1,000, 2,000, 3,000, ... 10,000 non-navigative queries).As described above, the discovery module 134 can filter the individual queries to generate a subset of queries that, when executed, each return 5 high-quality applications 135 with valid application highlights (e.g., three or more positive reviews) in the top 10 impressions (e.g., the first 10 results). The discovery module 434 can generate filtered application information 444 using the respective queries, filtered or unfiltered.

[0102] To expand the number of applications with filtered application information, the discovery module can extend the corpus of certain queries by adding more queries. For example, instead of the first 1,000 search queries, the discovery module can generate filtered application information using the first 2,000 or 3,000 search queries. By expanding the query corpus, the discovery module can generate filtered application information, such as highlight summaries, for additional applications, including new applications.

[0103] Fig. Figure 5A is a block diagram illustrating a first example of an application store according to one or more aspects of the present disclosure. Fig. 5A is below in connection with Fig. 1 described. The application store 540, the applications 535, the application information 542 and the filtered application information 544 from Fig. 5. Examples can be found for the application store 140, the applications 135, the application information 142, and the filtered application information 144. Fig. 1. The application store 540 can be provided by the store module 138.

[0104] The application store 540 from Fig. 5A can provide an example of query-level information. As can be seen, the application store 540 presents a user interface element that includes an application listing for a variety of applications 535. The query-level information can include filtered application information 544 for each application 535. For example, the filtered application information 544 can be from Fig. 5A Include feature lists and / or summaries that highlight features or other characteristics of applications 535.

[0105] In some examples, filtered application information 544 can be presented for only a limited number of applications 535 in an application listing. For example, filtered application information 544 can be presented for the first (e.g., top) n results of a query submitted to the application store 540. As described above, the ML module 136 of the discovery module 134 can generate filtered application information 544, including summaries of specific user ratings (e.g., ratings with the most positive or negative feedback, ratings with the most views) as highlights or lowlights of the application 535.

[0106] As can be seen, the application store 540 can present application information 542, such as application titles and descriptions provided by application developers. In the example from Fig. 5A The application information 542N can, for example, be a description of the corresponding application 535. As described above, the application information 542N can, in some cases, be of poor quality. Thus, the application store 540 can replace this application information 542 by presenting filtered application information 544 instead. For example, the application store 540 can present filtered application information 544, including a description of the application 535, instead of application information 542N. In such a case, the filtered application information 544 can occupy the space within the user interface element of the application store 540 that is otherwise reserved for the application information 542N. The application store 540 can include an indication of the source of the description. For example, only the application information 542 can be presented with an indication (e.g.,a quotation) which identifies the application developer or the publisher of application 535 as the source of the description.

[0107] Fig. Figure 5B is a block diagram illustrating a first example of an application store according to one or more aspects of the present disclosure. Similar to Fig. 5A is Fig. 5B below in connection with Fig. 1 described. The application store 540 in Fig. 5B can provide an example of application-level information. As can be seen, the application store 540 presents a user interface for the application 535A along with filtered application information 544 for the application 535A. The filtered application information 544 in Fig. Section 4B can include, for example, feature lists, summaries, questions and answers, or various subsets thereof. The summaries can include summaries of information provided by the developer or user reviews.

[0108] In some examples, application store 540 can determine whether an application belongs to local applications 125 (e.g., installed) or applications 135 (e.g., not installed). Application store 540 can present different information for local applications 125 (e.g., installed applications) compared to applications 135 that are hosted in application store 140 and are not installed. For example, application store 140 can present filtered application information 544, including feature enumerations, highlights, or both, for applications 135, and filtered application information 544, including questions and answers, for local applications 125. Application store 140 can hide certain types of filtered application information 144 for applications 135.For example, filtered application information 544, including questions and answers for applications 135, can be hidden until toggled, such as by activating a user interface element (e.g., a button), and shown by default for local applications 125. The application store 140 can provide different questions and answers for applications 135 than for local applications 125. The application store 140 can display questions and answers in a collapsible user interface, with the answers hidden until the corresponding questions are selected (e.g., tapped or clicked). In some examples, the collapsible user interface can be automatically expanded to hide one or more answers.The discovery module 134 can generate filtered application information 544 in response to user input to display questions and answers or other filtered application information 544.

[0109] Although in connection with Fig. 5A and Fig. 5B, each described as part of query-level information and application-level information, can in some examples filter application information 544, the query-level information from Fig. 5A form, filtered application information 544 be, the information at the application level form, how the information at the application level from Fig. 5B, and vice versa. Although the description refers to certain types of filtered application information 544, different types of filtered application information 544 can be presented as described above. For example, in addition to the questions and answers, summaries (such as application highlights), feature lists, or both can be hidden until toggled or presented by a collapsible or other user interface element that shows or hides these types of filtered application information 544 based on user input to toggle the presentation of such filtered application information 544.

[0110] Fig. Figure 6 is a flowchart illustrating an exemplary process for an application store using generative models for application discovery according to one or more aspects of the present disclosure. Fig. 6 will be discussed below in the context of Fig. 1 described.

[0111] The computing system 110 can generate one or more user interface elements for an application store 140 that hosts a variety of applications 135 (602). Each of the applications 135 can be associated with corresponding application information 142. Examples of application information 142 include short application descriptions (e.g., 60 characters or less) and / or long application descriptions (e.g., more than 60 characters), application names, application categorizations (e.g., productivity, messaging, social networking, games), application reviews, articles, social media posts, screenshots, icons, promotional content, application summaries, application ratings, and feature descriptions.

[0112] The computer system 110 can generate intermediate application information by providing at least application information 142 as input to the first ML module 136A running on the computer system 110 (604). The intermediate application information can be a condensed version of the application information 142 and may not include a specific subject of the application information 142. For example, the computer system 110 can generate a condensed version of the application information 142 using the first ML module 136A by generating a summary of the application information 142. The computer system 110 can generate the intermediate application information that lacks the specific subject by applying one or more rules, using the first ML module 136A, to exclude the specific subject from the application information when generating summaries that constitute the application information.In some examples, the item in question may also contain advertising content (e.g., marketing or advertising content).

[0113] Computing system 110 can generate filtered application information 144 by providing at least the intermediate application information as input to the second ML module 136B, which runs on computing system 110 (606). This filtered application information 144 can be considered as first filtered application information, which can correspond to query-level information (e.g., for a search results page). Computing system 110 can also generate second filtered application information by providing the intermediate application information as input to the third ML module 136C, which runs on computing system 110. In this example, the second filtered application information can be considered application-level information (e.g., for a page dedicated to an application of applications 135).To generate the second filtered application information, the computing system 110 can apply one or more rules to at least the first filtered application information 144 using the third ML module 136C to ensure that the second filtered application information matches the first filtered application information 144. The filtered application information 144 can include one or more application summaries, questions and answers, or feature lists.

[0114] Computing system 110 can receive a request for one or more applications from applications 135 (608). For example, computing system 110 can receive a request in the form of a query from computing devices 120. In response to receiving the request, computing system 110 can output filtered application information 144 along with information about the one or more applications 135 for presentation within one or more user interface elements of the application store 140 (610). For example, computing system 110 can output an application listing that includes information about one or more applications 135 along with filtered application information 144 for one or more applications 135.

[0115] In some examples, the computing system 110 can output different filtered application information 144, depending on whether the application 135A is a local application 125 or not (e.g., installed on the computing device 120). To output filtered application information 144, the computing system 110 can output a first subset of the filtered application information 144 (e.g., the first set of questions and answers) if one or more applications 135 are not one or more local applications 125, and output a second subset of the filtered application information 144 (e.g., the second set of questions and answers) if one or more applications 135 are one or more local applications 125. The one or more user interface elements of the application store 140 can be toggled to show and hide the filtered application information.

[0116] If some application information is of poor quality, the computing system 110 can replace at least some of the application information 142 with filtered application information 144. For example, the computing system 110 can replace a portion of the application information 142 (e.g., a poor short or long description provided by an application developer) if the computing system 110 determines that it is of poor quality (e.g., too short). In these examples, the computing system 110 can generate filtered application information 144 as a replacement for such application information 142. In some examples, the computing system 110 can generate filtered application information 144 that is longer than the replaced application information 142 and shorter than a text size threshold (e.g., 60 characters, 500 characters).In this way, the computing system can generate 110 descriptions in short or long form within the text size limit of the application store 140 for such information.

[0117] In some examples, the computing system 110 can divide the application information 142 into a multitude of clusters within a multidimensional embedding space, based on the sentiment contained in the application information. For each cluster of the multitude of clusters, the computing system 110 can generate at least some of the filtered application information based on the application information within the cluster.

[0118] In some examples, one or more processors 212 execute the discovery module 234, the store module 238, or both, to perform the actions described above in relation to the flowchart. Fig. 6. to provide the described functionality.

[0119] Aspects of this revelation include the following examples.

[0120] Example 1: A method involves a computing system generating one or more user interface elements for an application store, wherein the application store hosts a plurality of applications and wherein each application from the plurality of applications is associated with corresponding application information; generating intermediate application information by the computing system by providing at least the application information as input to a first machine learning module running on the computing system, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing; generating filtered application information by the computing system by providing at least the intermediate application information as input to a second machine learning module running on the computing system;Receiving a request for one or more applications by the computing system; and in response to receiving the request, outputting the filtered application information by the computing system, along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

[0121] Example 2: Method according to Example 1, wherein the filtered application information is first filtered application information and the method further includes generating second filtered application information by the computing system and using a third machine learning module based on the intermediate application information.

[0122] Example 3: Procedure according to Example 2, wherein generating the second filtered application information includes applying one or more rules by the third machine learning model to the first filtered application information to ensure that the second filtered application information matches the first filtered application information.

[0123] Example 4: Method according to Example 1, wherein generating the intermediate application information includes applying one or more rules by the first machine learning module to exclude the specific item from the application information.

[0124] Example 5: Procedure according to Example 4, where the specific object includes advertising content.

[0125] Example 6: Procedure according to Example 1, wherein the filtered application information includes one or more application summaries, questions and answers, or feature lists.

[0126] Example 7: Procedure according to Example 1, further comprising replacing at least some of the application information with the filtered application information by the computer system.

[0127] Example 8: Method according to Example 1, wherein one or more user interface elements of the application store are toggleable to show and hide the filtered application information.

[0128] Example 9: Procedure according to Example 1, wherein outputting the filtered application information includes outputting a first subset of the filtered application if the one or more applications are not one or more local applications, and outputting a second subset of the filtered application information if the one or more applications are one or more local applications.

[0129] Example 10: Method according to Example 1, further comprising clustering the application information by the computing system and within a multidimensional embedding space into a multitude of clusters based on sentiment; and generating at least some of the filtered application information based on the application information within the cluster by the computing system and for each cluster of the multitude of clusters.

[0130] Example 11: A computing system includes a memory that stores instructions and a processing circuit that executes the instructions to: generate one or more user interface elements for an application store, wherein the application store hosts a plurality of applications and wherein each application from the plurality of applications is associated with corresponding application information; generate intermediate application information by providing at least the application information as input to a first machine learning module that runs on the computing system, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing;Generating filtered application information by providing at least the intermediate application information as input for a second machine learning module running on the computing system; receiving a request for one or more applications; and, in response to receiving the request, outputting the filtered application information along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

[0131] Example 12: Computing system according to Example 11, wherein the filtered application information is first filtered application information and the processing circuit executes one or more instructions to generate second filtered application information based on the intermediate application information using a third machine learning module.

[0132] Example 13: Computing system according to Example 12, wherein, to generate the second filtered application information, the processing circuit executes one or more instructions to apply one or more rules to the first filtered application information using the third machine learning module to ensure that the second filtered application information matches the first filtered application information.

[0133] Example 14: Computing system according to Example 11, wherein, to generate the intermediate application information, the processing circuit executes one or more instructions to apply one or more rules using the first machine learning module to exclude the specified item from the application information.

[0134] Example 15: Computing system according to Example 14, where the specific item includes advertising content.

[0135] Example 16: Computing system according to Example 11, wherein the filtered application information includes one or more of an application summary, questions and answers, or feature lists.

[0136] Example 17: Computing system according to Example 11, wherein the processing circuit executes one or more instructions to replace at least some of the application information with the filtered application information.

[0137] Example 18: Computing system according to Example 11, wherein one or more user interface elements of the application store can be toggled to show and hide the filtered application information.

[0138] Example 19: Computing system according to Example 11, wherein, to output the filtered application information, the processing circuit executes one or more instructions to output a first subset of the filtered application if the one or more applications are not one or more local applications, and to output a second subset of the filtered application information if the one or more applications are one or more local applications.

[0139] Example 20: Non-transient computer-readable storage media include instructions which, when executed by the processing circuit, cause the processing circuit to: generate one or more user interface elements for an application store by a computing system, wherein the application store hosts a plurality of applications and wherein each application from the plurality of applications is associated with corresponding application information; generate intermediate application information by providing at least the application information as input to a first machine learning module executed on the computing system, wherein the intermediate application information is a compressed version of the application information and certain content of the application information is missing;Generating filtered application information by providing at least the intermediate application information as input for a second machine learning module running on the computing system; receiving a request for one or more applications; and, in response to receiving the request, outputting the filtered application information along with an indication of the one or more applications for presentation within the one or more user interface elements of the application store.

[0140] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted on a computer-readable medium as one or more instructions or as code and executed by a hardware-based processing unit. Computer-readable media may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates the transmission of a computer program from one location to another, for example, according to a communication protocol. In this way, computer-readable media can generally correspond to (1) a tangible, computer-readable storage medium that is non-transitory, or (2) a communication medium, such as a signal or carrier wave.Data storage media can be any available media that one or more computers or one or more processors can access to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0141] As an example, and not a limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage media that can be used to store the desired program code in the form of instructions or data structures and that a computer can access. Furthermore, any such connection is correctly referred to as a computer-readable medium.For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted-pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted-pair cable, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it is understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other volatile media, but instead refer to non-volatile, tangible storage media.In the sense used here, "disk" and "disc" include Compact Disc (CD), Laser Disc, Optical Disc, Digital Versatile Disc (DVD), floppy disk, and Blu-ray Disc, whereby "discs" typically reproduce data magnetically, while "discs" reproduce data optically using lasers. Combinations of the above are also included in the scope of computer-readable media.

[0142] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor," as used here, can refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described in this document. Furthermore, some aspects of the functionality described in this document can be provided in dedicated hardware and / or software modules. Additionally, the techniques could be implemented entirely in one or more circuits or logic elements.

[0143] The techniques of this disclosure can be implemented in a wide variety of devices or arrangements, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). This disclosure describes various components, modules, or units to highlight functional aspects of devices configured to perform the disclosed techniques, but does not necessarily require implementation by different hardware units. Rather, as described above, different units can be combined in a single hardware unit or provided by a collection of intraoperative hardware units, including one or more processors, as described above in conjunction with suitable software and / or firmware.

[0144] It should be noted that, depending on the example, certain actions or events of one of the procedures described here may be performed in a different sequence, added, combined, or omitted entirely (i.e., not all described actions or events are necessary for the practical application of the procedure). Furthermore, in certain examples, actions or events may be performed concurrently, for example, through multithreaded processing, interrupt handling, or multiple processors, rather than sequentially.

[0145] In some examples, a computer-readable storage medium includes a non-transitory medium. The term "non-transitory" means that the storage medium is not contained within a carrier wave or transmitted signal. In certain examples, a non-transitory storage medium can store data that may change over time (e.g., in RAM or cache).

[0146] Several examples have been described. These and other examples are within the scope of the following patent claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 674,428

[0001]

Images