Interacting with skill repository

Abstract

Systems and methods for interacting with a skill repository are provided herein. In some examples, a generative large model (GLM) is used to process a task to coordinate skills for performing the task. The coordinated skills include a plurality of skills related to the task. At least one of the coordinated skills is determined to be unavailable to the GLM, and an indication corresponding to the at least one skill is sent to a remote skill repository. The indication may be associated with a description of at least one skill based on which a similarity for retrieving the skill from a remote skill repository may be determined. Remote skills are received from a remote skill repository corresponding to the sent indication, and the task is executed using a generative LLM. The generative LLM uses remote skills to perform tasks.

Description

Background Art

[0001] A computing device can be relied upon to perform any of a variety of different tasks. Some of the different tasks can be performed using skills. Additionally, some of the different tasks can be performed using an automated system. However, if the skills required to perform a task are not available, it may result in damages such as a reduced user experience, increased user disappointment, and / or wasted computing resources.

[0002] It is in view of these and other general considerations that the aspects described in the present disclosure have been made. Moreover, although relatively specific problems have been discussed, it should be understood that the aspects disclosed herein should not be limited to solving the specific problems identified in the background art. Summary of the Invention

[0003] Aspects of the present disclosure relate to methods, systems, and media for interacting with a skill repository, such as using a generative large language model (GLLM) or a generative large model (GLM). Exemplary systems and methods for interacting with a skill repository are provided herein. In some examples, a generative large model (GLM) is used to process a task to coordinate skills for performing the task. The coordinated skills include a plurality of skills related to the task. At least one of the coordinated skills is determined to be unavailable to the GLM, and an indication corresponding to the at least one skill is sent to a remote skill repository. The indication can be associated with a description of the at least one skill, based on which a similarity for retrieving a skill from the remote skill repository can be determined. A remote skill is received from the remote skill repository corresponding to the sent indication, and a generative LLM is used to perform the task. The generative LLM uses the remote skill to perform the task.

[0004] The present summary is provided to introduce in a simplified form a selection of concepts that are further described below in the detailed description. The present summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description that follows, and in part will be obvious from the description, or can be learned by practice of the present disclosure. Brief Description of the Drawings

[0005] Non-limiting and non-exhaustive examples are described with reference to the following drawings.

[0006] Figure 1 An overview of an example system in accordance with some aspects described herein is shown.

[0007] Figure 2 Examples of content in accordance with some aspects described herein are shown, such as private content and public content.

[0008] Figure 3 Shows an example vector space in accordance with some aspects described herein.

[0009] Figure 4 Shows an example process for interacting with a skill repository in accordance with some aspects described herein.

[0010] Figure 5 Shows an example method for interacting with a skill repository in accordance with some aspects described herein.

[0011] Figure 6 Shows an example method for interacting with a skill repository in accordance with some aspects described herein.

[0012] Figure 7A and Figure 7B Shows an overview of an example generative machine learning model that can be used in accordance with the aspects described herein.

[0013] Figure 8 Shows a block diagram of example physical components of a computing device in which aspects of the present disclosure may be practiced.

[0014] Figure 9 Shows a simplified block diagram of a computing device in which aspects of the present disclosure may be practiced.

[0015] Figure 10 Is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. Detailed Description

[0016] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific aspects or examples. Without departing from the present disclosure, these aspects may be combined, other aspects may be utilized, and structural changes may be made. The aspects may be practiced as a method, system, or device. Thus, the aspects may take the form of a hardware implementation, a fully software implementation, or an implementation combining software and hardware aspects. Accordingly, the following detailed description should not be considered limiting, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0017] As described above, a computing device may be relied upon to perform any of a variety of different tasks. Some of the different tasks may be performed using skills. Additionally, some of the different tasks may be performed using automated systems (such as models trained using artificial intelligence (AI) or machine learning (ML)). However, if the skills required to perform a task are not available, it may result in impairments such as a reduced user experience, increased user disappointment, and / or wasted computing resources.

[0018] The present disclosure provides a skill repository that can be used by agents (e.g., hardware and / or software components) designed to expand the capabilities of the agent and built on top of semantic kernel (SK) capabilities. The skill repository provided herein can be a skill marketplace that allows agents (such as models trained using AI or ML) to discover skills for solving problems (e.g., tasks provided by the system and / or user). The repository can be primarily accessed by AI agents and can act as a global and / or centralized repository for skill lookup.

[0019] When processing a task, problem, or request, a coordination of skills that perform actions to solve the task, problem, or request can be generated. This coordination of skills can include skill chains, scripts, code, action graphs, etc. When the SK (e.g., SK local to the device) cannot solve parts of the skill coordination, such as a skill chain (e.g., one or more skills are not resolved or located on the device), the unresolved skills can be sent as an application programming interface (API) request to the skill repository. The skill repository returns one or more skills that can solve the problem, such as by comparing or ranking based on vectors or semantic embeddings of parameters corresponding to the one or more skills. The lookup and execution of skills can be transparent to the user. The repository can return not only a single skill (e.g., perfect hit, highest rank), but also find possible candidates, such as based on a confidence score associated with the result, and refine (e.g., ask, introspect) to improve the accuracy of skill determination. The skills discussed herein can refer to other models, parameters, source code locations, and methods for solving problems.

[0020] In some examples, the SK receives a query. The SK can break the query into components (e.g., skills, parameters) for execution or coordination (e.g., chaining). The mechanisms provided herein can determine that local skills are not applicable to one or more aspects (e.g., chain steps) of the skill coordination, and thus a mechanism for interacting with a remote skill repository is needed. The interaction with the remote skill repository can occur via an API, causing the skill repository to receive a request (e.g., identify a cat in an image) based on semantic parameters of the skill (e.g., parameters).

[0021] After interacting through the API, the skill repository can return one or more matching skills based on geometric distance, similarity, sorting, or some other computational comparison. The SK can select the best skill from one or more matching skills (e.g., based on introspection, user feedback, or some other evaluation). The SK can also obtain resources for the returned skills, such as models, parameters, source locations, how to call the skills, how to host the skills, how to parse the returned information, or other resources that a person of ordinary skill in the art can recognize. The SK completes previously unknown skills in skill coordination (e.g., skill chains), so that operations can be performed to solve the task or problem being processed. In some examples, skills can be imported and / or dynamically loaded into the SK from an external repository for frequent use. In some examples, the skill storage can be locally, regionally, and / or globally accessible.

[0022] Figure 1 FIG. 4 shows an example of a system 100 in accordance with some aspects of the disclosed subject matter. The system 100 can be a system for interacting with a skill repository. Additionally or alternatively, the system 100 can be a system for interacting with a skill repository using a generative large model (GLM) (such as a generative large language model (LLM)). The system 100 includes one or more computing devices 102, one or more servers 104, a content data source 106, an input data source 107, and a communication network or network 108.

[0023] The computing device 102 can receive content data 110 from the content data source 106, which can be, for example, a microphone, a camera, a global positioning system (GPS), etc. that sends content data, a computer-executed program that generates content data, and / or a memory that stores data corresponding to the content data. The content data 110 can include visual content data, audio content data (e.g., speech or ambient noise), gaze content data, calendar entries, emails, document data (e.g., virtual documents), weather data, news data, blog data, encyclopedia data, and / or other types of private and / or public content data that a person of ordinary skill in the art can identify. In some examples, the content data can include text, source code, commands, skill, or program evaluations.

[0024] The computing device 102 can also receive input data 111 from the input data source 107, which can be, for example, a camera, a microphone, a computer-executed program that generates input data, and / or a memory that stores data corresponding to the input data. The content data 111 can be, for example, user input (such as a voice query, a text query, etc.), an image, an action performed by the user and / or the device, a computer command, a program evaluation, or some other input data that a person of ordinary skill in the art can identify.

[0025] Additionally or alternatively, network 108 may receive content data 110 from content data source 106. Additionally or alternatively, network 108 may receive input data 111 from input data source 107.

[0026] Computing device 102 may include a communication system 112, a skill coordination engine or component 114, a skill repository retrieval engine or component 116, and / or a self-introspection engine or component 118. In some examples, computing device 102 may execute at least a portion of skill coordination component 114 to coordinate skills. Coordination of skills may include generating a skill chain, script, code, action graph, etc. for performing a task. For example, coordination of skills (e.g., a skill chain) may include multiple skills and / or one or more prompts corresponding to the skills.

[0027] Furthermore, in some examples, computing device 102 may execute at least a portion of skill repository retrieval component 116 to retrieve skills from one or more skill repositories. For example, one or more skill repositories may be a private skill repository and / or a public skill repository. In some examples, each skill repository may be accessed by a different tenant (e.g., an organization, an individual, etc.). In some examples, one or more skill repositories may be local to a computing device (such as computing device 102). Additionally and / or alternatively, in some examples, one or more skill repositories may be located remotely from computing device 102, such as stored on a server (e.g., server 104).

[0028] Furthermore, in some instances, computing device 102 may execute at least a portion of self-introspection component 118 to query a function proxy (e.g., a GLM) to determine if it knows how to perform a skill (e.g., from a skill chain). For example, self-introspection may include determining with confidence whether the mechanisms disclosed herein know how to perform a skill, know how to perform the skill correctly, etc. Those of ordinary skill in the art will recognize additional and / or alternative types of self-introspection that may be used to evaluate the selection and / or execution of skills.

[0029] Server 104 may include a communication system 120, a skill coordination engine or component 122, a skill repository retrieval engine or component 124, and / or a self-introspection engine or component 126. In some examples, server 104 may execute at least a portion of skill coordination component 122 to coordinate skills. Coordination of skills may include generating a skill chain, script, code, action graph, etc. for performing a task. For example, coordination of skills (e.g., a skill chain) may include multiple skills and / or one or more prompts corresponding to the skills.

[0030] In addition, in some examples, the server 104 may execute at least a portion of the skill repository retrieval component 124 to retrieve skills from one or more skill repositories. For example, the one or more skill repositories may be a private skill repository and / or a public skill repository. In some examples, the one or more skill repositories may each be accessed by a different tenant (e.g., an organization, an individual, etc.). In some examples, the one or more skill repositories may be local to the client device (such as the computing device 102). Additionally and / or alternatively, in some examples, the one or more skill repositories may be remote from the client device, such as being stored on a server (e.g., the server 104).

[0031] In addition, in some instances, the server 104 may execute at least a portion of the introspection component 126 to query a function proxy (e.g., the GLM described herein, another model, and / or a software or hardware component) as to whether it knows how to perform a skill (e.g., from a skill chain). For example, introspection may include determining with confidence whether the mechanisms disclosed herein know how to perform a skill, know how to perform the skill correctly, etc. Those of ordinary skill in the art will recognize additional and / or alternative types of introspection that may be used to evaluate the selection and / or execution of a skill.

[0032] Additionally or alternatively, in some instances, the computing device 102 may send data received from the content data source 106 and / or the input data source 107 to the server 104 via the communication network 108, and the server 104 may execute at least a portion of the skill coordination component 114, the skill repository retrieval component 116, and / or the introspection component 118. In some examples, the skill coordination component 114 may perform one or more portions of the method / process 400 and / or 500 described below in connection with Figure 4 and Figure 5 respectively. In addition, in some examples, the skill storage retrieval component may perform one or more portions of the method / process 400 and / or 500 described below in connection with Figure 4 and Figure 5 respectively. Further, in some instances, the introspection component may perform one or more portions of the method / process 400 and / or 500 described below in connection with Figure 4 and 5 respectively.

[0033] In some examples, computing device 102 and / or server 104 can be any suitable computing device or combination of devices, such as a desktop computer, a mobile computing device (e.g., a laptop computer, a smartphone, a tablet computer, a wearable computer, etc.), a server computer, a virtual machine executed by a physical computing device, a web server, etc. Additionally, in some examples, there can be multiple computing devices 102 and / or multiple servers 104. Those of ordinary skill in the art should recognize that content data 110 and / or input data 111 can be received at one or more of the multiple computing devices 102 and / or one or more of the multiple servers 104, such that the mechanisms described herein can interact with the skill store based on the aggregation of content data 110 and / or input data 111 received across the computing devices 102 and / or servers 104.

[0034] In some examples, content data source 106 can be any suitable content data source (e.g., a microphone, a camera, a GPS, a sensor, etc.). In more specific examples, content data source 106 can include a memory storing content data (e.g., the local memory of computing device 102, the local memory of server 104, cloud storage, a portable memory connected to computing device 102, a portable memory connected to server 104, etc.). In another more specific example, content data source 106 can include an application configured to generate content data. In some examples, content data source 106 can be local to computing device 102. Additionally or alternatively, content data source 106 can be remote from computing device 102 and can transmit content data 110 to computing device 102 (and / or server 104) via a communication network (e.g., communication network 108).

[0035] In some examples, input data source 107 can be any suitable input data source (e.g., a microphone, a camera, a sensor, etc.). In more specific examples, input data source 107 can include a memory storing input data (e.g., the local memory of computing device 102, the local memory of server 104, cloud storage, a portable memory connected to computing device 102, a portable memory connected to server 104, a privately accessible memory, a publicly accessible memory, etc.). In another more specific example, input data source 107 can include an application configured to generate input data. In some examples, input data source 107 can be local to computing device 102. Additionally or alternatively, input data source 107 can be remote from computing device 102 and can transmit input data 111 to computing device 102 (and / or server 104) via a communication network (e.g., communication network 108).

[0036] In some examples, communication network 108 can be any suitable communication network or combination of communication networks. For example, communication network 108 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, 4G network, 5G network, etc. that complies with any suitable standard), a wired network, etc. In some examples, communication network 108 can be a local area network (LAN), a wide area network (WAN), a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Figure 1 The illustrated communication links (arrows) can each be any suitable communication link or combination of communication links, such as a wired link, an optical fiber link, a Wi-Fi link, a Bluetooth link, a cellular link, etc.

[0037] Figure 2 Examples are shown of aspects described herein, such as private content 200 and public content 250. As discussed with respect to system 100, the described examples can receive content data (e.g., content data 110) from a content data source (e.g., content data source 106). The received content data can include private content 200 and / or public content 250. Additionally or alternatively, the content data can include source code, commands, program evaluations, or skills. Figure 2 The content shown in can provide context to the model and / or skill repository regarding which skills are intended to be retrieved and / or executed to complete the intended task. That is, the content can help precisely define aspects of the intended task and / or the corresponding skills associated therewith.

[0038] Generally, when users interact with a computing device (e.g., computing device 102), they can interact with applications stored locally on the computing device and / or that can be executed locally on the computing device. Information accessed or executed locally by the user on their device can include private content 200.

[0039] As an example, private content includes audio content 202, visual content 204, gaze content 206, calendar entries 208, emails 210, and documents 212. Those of ordinary skill in the art can recognize additional and / or alternative types of private content.

[0040] The audio content 202 may include data corresponding to the generated speech data. For example, the audio content 202 may be generated by the computing device 102 to correspond to audio received from a user (e.g., in a case where the user is speaking into a microphone of a computing device that may be separate from the computing device 102). Additionally or alternatively, the audio content 202 may correspond to a type of audio data that may be generated by the computing device, such as synthetic speech, animal sounds, beeps, buzzes, or another type of generated audio data.

[0041] The visual content 204 may include data corresponding to graphical content that may be displayed and / or generated by the computing device. For example, the visual content 204 may be content generated via an application running on the computing device 102 (e.g., a web browser, a presentation application, a teleconference application, a business management application, etc.). The visual content 204 may include data captured from the screen display of the computing device 102. For example, any visual indication displayed on the computing device 102 may be included in the visual content 204.

[0042] The gaze content 206 may include data corresponding to the location where the user is looking. For example, a specific action to be performed by the computing device may be associated with a combination of a specific location where the user is looking and / or the location where the user is looking for a predefined duration.

[0043] The calendar entry 208 may include calendar data specific to one or more users. For example, the calendar data may include meetings, appointments, reservations, or other types of calendar entries. Additionally or alternatively, the calendar data may include time, location, participants, and / or notes regarding a specific calendar entry. Those of ordinary skill in the art may recognize additional and / or alternative data associated with calendar entries.

[0044] The email 210 may include email data for one or more emails. For example, the email 210 may include email data corresponding to a collection or multiple emails. The email data may include the sender and recipient, subject, message, image, timestamp, and / or other types of information that may be associated with an email. Those of ordinary skill in the art may recognize additional and / or alternative data associated with calendar entries.

[0045] The virtual document 212 may include the types of documents found in a virtual environment. For example, the virtual document 212 may be a text editing document, a presentation, an image, a spreadsheet, a series of animated images, a notification, or any other type of virtual document that those of ordinary skill in the art may recognize.

[0046] Each of the various types of private content 200 can be a subset of the private content 200, which can be received by the mechanisms described herein as content data. Additionally, while specific examples of private content types have been discussed above, one of ordinary skill in the art will recognize additional and / or alternative types of private content.

[0047] Public content 250 includes weather 252, news 254, encyclopedia 256, blogs 258, etc. Weather 252 can include information about the weather around the user and / or at locations determined to be of interest to the user. For example, for a given time, the location where the user is located (e.g., based on location content) and / or the weather information (e.g., precipitation, temperature, humidity, etc.) at locations determined to be of interest to the user can be received or otherwise obtained.

[0048] News 254 can include information about recently published news stories determined to be of interest to the user. For example, for a given time, relatively recent news stories covering significant events may have been published. Additional or alternative types of news stories can include holidays, birthdays, local events, national events, natural disasters, celebrity news, scientific discoveries, sports news, or any other type of news that one of ordinary skill in the art will recognize.

[0049] Encyclopedia 256 can include publicly available encyclopedia information. For example, encyclopedia 256 can include information from an online database of encyclopedia information. Additionally or alternatively, encyclopedia 256 can include pages from an online encyclopedia website. One of ordinary skill in the art will recognize additional or alternative types of encyclopedia information.

[0050] Blogs 256 can include information from blogs. For example, blogs can include publicly available posts from users of blog websites and / or social media platforms. Blogs can be posted by, for example, well-known individuals such as chefs, politicians, actors, etc. Alternatively, blogs can be posted by other users whose content posted online can be publicly accessed through the mechanisms disclosed herein.

[0051] Generally, with respect to Figure 2 The different content types discussed provide various types of content that a computing device can receive or otherwise access, which can help provide context information for determining the skills described herein. Additionally, while specific subsets of content have been described above with respect to one of private content 200 and public content 250, it should be recognized that in some examples, the subset of content can instead be described with respect to the other of private content 200 or public content 250. Additionally, note that one of ordinary skill in the art will recognize additional and / or alternative types of private content 200 and / or public content 250.

[0052] Figure 3 Illustrates an example vector space 300 in accordance with some aspects described herein. The vector space 300 includes a plurality of feature vectors, such as a first feature vector 302, a second feature vector 304, a third feature vector 306, a fourth feature vector 308, and a fifth feature vector 310. Each of the plurality of feature vectors 302, 304, 306, and 308 corresponds to a respective embedding 303, 305, 307, 309 generated based on a plurality of skills and / or a plurality of subsets of content data (e.g., subset 110 of content data, private content 200, and / or public content 250). The embeddings 303, 305, 307, and 309 may be semantic embeddings. The fifth feature vector is generated based on an input embedding 311. The input embedding may be generated based on an input (e.g., input data 111). For example, the input may be a user input corresponding to a task to be performed.

[0053] The feature vectors 302, 304, 306, 308, 310 each have a distance that is measurable relative to one another. For example, the distance between the feature vectors 302, 304, 306, and 308 and the fifth feature vector 310 corresponding to the input embedding 311 may be measured using cosine similarity. Alternatively, the distance between the feature vectors 302, 304, 306, 308 and the fifth feature vector 310 may be measured using another distance measurement technique (e.g., an n-dimensional distance function) that would be recognized by one of ordinary skill in the art.

[0054] The similarity of each of the feature vectors 302, 304, 306, 308 to the feature vector 310 corresponding to the input embedding 311 may be determined, for example, based on the measured distance between the feature vectors 302, 304, 306, 308 and the feature vector 310. The similarity between the feature vectors 302, 304, 306, 308 and the feature vector 310 may be used to group or cluster the feature vectors 302, 304, 306, and 308 into one or more sets of feature vectors (such as set 312), thereby generating a set of embeddings.

[0055] In some examples, the set 312 may include a predetermined number of feature vectors such that the group of feature vectors is given a predetermined size. Additionally or alternatively, in some examples, the distance between each of the feature vectors 302, 304, 306, 308 and the feature vector 310 corresponding to the input embedding 311 is compared to a predetermined threshold.

[0056] Embeddings 303 and 305 corresponding to feature vectors 302 and 304, respectively, may fall within the same content group and / or the same skill category. For example, embedding 303 may be related to the skill of sending an email using a first email protocol, and embedding 305 may be related to the skill of sending an email using a second email protocol. Those of ordinary skill in the art will recognize additional and / or alternative examples of content groups and / or skill categories into which the embeddings may be classified.

[0057] Set 312 may be stored in a data structure, such as an ANN tree, a k-d tree, an octree, another n-dimensional tree, or another data structure capable of storing a vector space representation that is recognizable to those of ordinary skill in the art. Additionally, the memory corresponding to the data structure in which set 312 is stored may be arranged or stored within the data structure in a manner that groups together the embeddings and / or vectors in set 312. In some examples, the feature vectors and their corresponding embeddings generated according to the mechanisms described herein may be stored indefinitely. Additionally or alternatively, in some examples, as new feature vectors and / or embeddings are generated and stored, the new feature vectors and / or embeddings may overwrite older feature vectors and / or embeddings stored in the memory (e.g., based on metadata of the embedding indicating a version) in order to increase the memory capacity. Additionally or alternatively, in some examples, the feature vectors and / or embeddings may be deleted from the memory at a specified time interval and / or based on the amount of available memory (e.g., in one or more skill repositories described herein) in order to increase the memory capacity.

[0058] Generally, the ability to store embeddings corresponding to received content data and / or stored skills allows users to associate and locate data in a novel way with the benefit of computational efficiency. Thus, the mechanisms described herein are efficient in terms of reducing memory usage as well as reducing the processing resources used to search for stored content and / or skills. Those of ordinary skill in the art will recognize additional and / or alternative advantages.

[0059] Figure 4 An example process 400 for interacting with a skill repository 410 in accordance with some aspects described herein is shown. In some examples, a user request 402 is received. At a coordinator 404, the user request 402 may be processed to coordinate skills for performing a task. For example, the coordinated skills may include skill chains, scripts, code, operation graphs, and the like.

[0060] The coordinator 404 can interact with the self-skill model 406, the third-party skill model 408, and / or the skill repository 408, such as through a generative large language model (GLM). For example, the coordinator 404 can perform skill discovery, such as based on a prompt or hint, to locate one or more skills. The prompt or hint can be in the form of natural language and / or an intermediate language from which at least a portion of the user intent of the request 402 can be derived.

[0061] The user request 402, the coordinator 404, and / or the self-skill model 406 can be stored, accessed, and / or executed on a local domain. The self-skill model 406 can include multiple skills specific to the user providing the request 402 and / or accessible only to users permitted to access the local domain.

[0062] The third-party skill model 408 and / or the skill repository 410 can be stored, accessed, and / or executed on an internet domain. The third-party skill repository can include multiple skills specific to an organization (e.g., a third-party organization managing skills for a specific task) and / or accessible to users permitted to access services provided by a third party responsible for the third-party skill repository 408 in the internet domain. The skill repository 410 can be an artificial intelligence (AI) and / or machine learning (ML) vector skill repository. In some examples, the skill repository 410 is a graph, or a database, or another data structure that one of ordinary skill in the art will recognize as being capable of storing skills and interacting with the mechanisms described herein.

[0063] The skill repository 410 can be implemented using tenant isolation. The skill repository 410 can have a sharing rule that allows certain individuals to access certain skills and / or prohibits certain individuals from accessing certain skills. In some examples, the skill repository 410 is multiple skill repositories that together form the skill store 410. In some examples, skills can be uploaded to, updated within, and / or removed from the skill repository 410. In some examples, skills uploaded to the skill repository 410 require verification and / or malware checking, such as through a signature mechanism.

[0064] In some examples, the skills within the skill repository 410 or performed by the self-skill model 406 and / or the third-party skill model 408 can have a description, such as a short description and / or a long description, to help identify the skill. The description can include natural language and / or an intermediate language. In some examples, the description can be an embedding for semantic similarity matching.

[0065] In some examples, the local Semantic Kernel (SK) 412 determines whether a skill returned from one of the self-skill model 406, third-party skill model 408, and / or skill repository 410 is appropriate. In some examples, SK 412 asks the user for feedback on the returned skill and / or returns a confidence score regarding the probability that the returned skill will perform a task to complete at least a portion of the user request 402. The SK can include a memory and local skills executable based on the content stored in the memory.

[0066] In some examples, if no appropriate skill can be found, a skill can be generated, such as based on the description of the skill that could not be found. In some examples, the SK invokes skills coordinated by the coordinator 404 and / or invokes skill models (such as the self-skill model 406 and / or third-party skill model 408) to complete the skill.

[0067] In some examples, the results of skills performed by multiple different SKs 412 can be combined, and a single result can be returned. In some examples, SK 412 can have hard-coded logic for chunking the input and / or output, such as the input for performing certain skills and / or the output provided as a result of performing certain skills.

[0068] In some examples, the SK can be associated with a repository directory 414. The repository directory can include one or more default skill repositories and / or custom preferences for retrieving and / or executing skills.

[0069] Figure 5 An example method 500 for interacting with a skill repository in accordance with some aspects described herein is shown. In an example, aspects of method 500 are performed by a device, such as the computing device 102 and / or server 104 discussed above with respect to Figure 1 discussed above.

[0070] Method 500 begins at operation 502, where a task is processed to coordinate the skills for performing the task (e.g., generating a skill chain). A model such as a Generative Large Model (GLM) can be used to process the task. In some examples, the GLM can be a Generative Large Language Model (LLM). In some examples, the model can be another type of machine learning model as would be recognized by one of ordinary skill in the art.

[0071] The task can be received as an input (e.g., input data 111) or generated based on an input. The coordinated skills (e.g., skill chain) can include multiple skills related to the task. Additionally or alternatively, the coordinated skills (e.g., skill chain) can include multiple indications corresponding to the skills related to the task, such as descriptions, prompts, parameters, variables, etc.

[0072] To coordinate skills (e.g., generate a skill chain), a chain coordinator can generate coordination cues, which can include indications of one or more skills from a skill library or repository or list and at least a portion of the input, such that a generative LLM model coordinates the skills for processing the user input (e.g., generate a skill chain). Accordingly, the chain coordinator can map one or more intents of the input to one or more models and / or program skills of the skill library.

[0073] The coordinated skills (e.g., skill chain) can include one or more sequential skills, a set of hierarchical skills, a set of parallel skills, and / or skills that depend on or otherwise process the output from two or more previous skills, etc. In an example, the evaluation order of the coordinated skills can be determined based on the available skills in the skill repository. Additionally or alternatively, the coordinated skills include one or more program skills, code, machine instructions, cues, prompts, etc. In some examples, when coordinating skills (such as generating a skill chain), context can be provided to the model (e.g., can be included as part of the generated cue), such as can be determined by a recall engine from a semantic memory engine.

[0074] At operation 504, it is determined whether there is at least one skill in the coordinated skills (e.g., skill chain) that is not available to the generative LLM. For example, determining that at least one skill in the coordinated skills is not available to the generative LLM can include determining that at least one skill in the skill chain does not correspond to one or more local skills stored on the local device.

[0075] In some examples, one or more skills in the coordinated skills are stored on the local device. The one or more skills stored on the local device can be stored in a common skill repository and / or a private skill repository. The private skill repository can have access permissions limited to a specific group of users and / or organizations. For example, the private skill repository can have permissions to exclude certain users and / or systems from accessing the skills in the private skill repository. In some examples, the common skill repository can have no restrictions on access. Alternatively, the common skill repository can have more lenient access restrictions than the private skill repository (e.g., allowing a relatively greater amount of access than the private skill repository).

[0076] In some examples, there can be multiple skill repositories located locally and / or remotely to the client device. The multiple skill repositories can have tenant isolation, which restricts which users and / or organizations can access each skill repository. Those of ordinary skill in the art can recognize additional and / or alternative security protocols for controlling access to and / or maintaining the skill repositories described herein.

[0077] If it is determined that all skills in the coordinated skills are available to the generative LLM (e.g., there is no at least one skill in the coordinated skills that is not available to the generative LLM), then the process branches "no" to operation 506, where a default action is performed. For example, the coordinated skills (e.g., the generated skill chain) can have an associated pre-configured action. In other examples, method 500 can include determining whether the coordinated skills have an associated default action such that, in some cases, no action may be performed as a result of the coordinated skills. Method 500 can terminate at operation 506. Alternatively, method 500 can return to operation 502 to provide an iterative loop for processing tasks for the coordinated skills and determining whether at least one skill in the coordinated skills is not available to the generative LLM.

[0078] However, if it is determined that there is at least one skill in the coordinated skills that is not available to the generative LLM, then the process instead branches "yes" to operation 508, where an indication corresponding to the at least one skill is sent to a remote skill repository. The remote skill repository can be stored on one or more remote devices, such as the server 104 described above with respect to Figure 1 Description.

[0079] In some examples, sending an indication corresponding to the at least one skill to the remote skill repository includes generating a description of the at least one skill. Additionally, in some examples, the indication corresponding to the at least one skill is sent to the remote skill repository via an application programming interface (API).

[0080] At operation 510, remote skills corresponding to the sent indication are received, such as from the remote skill repository. In some examples, receiving the remote skills includes receiving the remote skills based on the description of the at least one skill. In some examples, the description is a short description. Additionally, in some examples, sending an indication corresponding to the at least one skill to the remote skill repository further includes generating a long description of the at least one skill (e.g., by the generative LLM). Receiving the remote skills can include receiving the remote skills based on the generated long description and the generated short description of the at least one skill.

[0081] Generally, providing a short description can provide a quick way to locate a skill within the skill repository. However, by providing a long description (e.g., a more detailed description than the short description), the skill can be located within the skill repository relatively more precisely. Thus, one of ordinary skill in the art should recognize that by continuing to increase the length and / or detail of the description corresponding to a skill, the mechanisms described herein can be capable of increasing the precision with which a desired skill can be located to perform the processed task of operation 502.

[0082] In some examples, the generated description (e.g., long description and / or short description) is natural language. In some examples, the generated description is an intermediate language, such as a programming language or a representation used as an intermediate step in the compilation or interpretation process of a high-level language. In some examples, the generated description is an embedding, such as an embedding generated by a model (e.g., based on vision processing and / or natural language processing).

[0083] In some examples, receiving a remote skill includes performing introspection on the remote skill based on the description to generate a confidence threshold corresponding to the remote skill and the task. For example, if a user directly or indirectly (e.g., as part of a larger task) requests to send an email using a specific application, but the skill repository does not know how to send an email using the specific application, the skill repository can instead evaluate how to send an email using an application that has a similar protocol to the specific application. Thus, one or more remote skills can be evaluated and assigned a confidence score corresponding to how well the one or more remote skills meet the user's intent. Content information that provides semantic context to the user's intent (see Figure 2 ) can be used to clarify the user's intent. Those of ordinary skill in the art will recognize additional and / or alternative examples.

[0084] In some examples, the skills in the skill repository (such as the received remote skills) can be associated with models, parameters, source locations, how to call the skills, how to host the skills, and how to parse the returned information and / or other information that those of ordinary skill in the art will recognize.

[0085] At operation 512, a generative LLM is used, for example, to perform a task. The generative LLM can use the received remote skills to perform the task. For example, the received remote skills can be used in parallel and / or sequentially with other skills (e.g., received from a local and / or remote skill repository) to perform the task. In some examples, one or more of the multiple skills (e.g., from a coordination skill) include machine-executable instructions. Performing the task can include adapting one or more devices to execute the machine-executable instructions. For example, the one or more devices can include a single device (e.g., computing device 102 and / or server 104) and / or multiple devices each performing at least a portion of the machine-executable instructions.

[0086] In some examples, before performing the task at operation 512, method 500 includes providing the user with a notification corresponding to the remote skill. For example, the notification can include an audio and / or visual report that describes the remote skill. In response to the notification, feedback that confirms the remote skill can be received (e.g., from the user). In such examples, the received remote skill can be executed only when verification feedback is received, before performing the task.

[0087] Method 500 may terminate at operation 512. Alternatively, method 500 may return to operation 502 to provide an iterative loop of processing tasks to coordinate skills for performing tasks, interacting with a remote skills repository, and performing tasks using a generative LLM with skills received (at least) from the remote skills repository.

[0088] Figure 6 An example method 600 for interacting with a skills repository in accordance with some aspects described herein is shown. In an example, aspects of method 600 are performed by a device, such as computing device 102 and / or server 104 discussed above with respect to Figure 1 discussion.

[0089] Method 600 begins at operation 602, where parameters of a skills repository are received from a local device. The parameters may be associated with one or more descriptions, such as a long description and / or a short description. One or more descriptions (e.g., short description, long description) may each be natural language data or an embedding. For example, a description may be a natural language description written by a user and / or a natural language description generated by a machine learning model (e.g., a generative LLM). In some examples, the description is an intermediate language description, such as a programming language or a representation used as an intermediate step in the compilation or interpretation of a high-level language. Additionally or alternatively, the description may be an embedding generated based on attributes or associations of the received parameters.

[0090] At operation 604, it is determined whether the skills repository has a subset of skills associated with at least one of the one or more descriptions. For example, it may be determined whether the skills repository has a subset of skills associated with the short description. Generally, using a short description to search the skills repository may be computationally efficient compared to a long description, such as because the short description provides fewer criteria for processing the search. However, a relatively longer description may help improve the accuracy of the search, such as by providing more details about one or more skills to be located within the skills repository.

[0091] If it is determined that the skills repository does not have a subset of skills associated with the short description, the process branches "no" to operation 406, where a default action is performed. For example, the received parameters may have a preconfigured action associated with them. In other examples, method 400 may include determining whether the received parameters have a default action associated with them such that, in some cases, as a result of the received parameters, skills are not located in the skills repository. Method 600 may terminate at operation 606. Alternatively, method 600 may return to operation 602 to provide an iterative loop of receiving parameters of the skills repository and determining whether the skills repository has a subset of skills associated with the description associated with the parameters.

[0092] However, if it is determined that there is a subset of skills associated with the short description, the process branches "Yes" to operation 608, where, based on the short description, the corresponding similarity between the parameter and each skill (or one or more skills) stored in the skill repository is determined. In some examples, the similarity can be a semantic similarity. For example, the similarity can be based on the abstract meaning behind the elements being compared.

[0093] In some examples, the skill repository includes skill descriptions for one or more of the skills stored therein. The skill descriptions can be generated by a model, such as by a generative LLM. Additionally, the corresponding similarity can be determined at least in part based on the description of the skill. For example, the skill description can be compared with the short description associated with the parameter. The skill description can include a natural language description. In some examples, the skill description includes an intermediate language, such as a programming language or a representation used as an intermediate step in the compilation or interpretation of a high-level language. Additionally or alternatively, the skill description can be an embedding.

[0094] At operation 610, one or more similarities determined at operation 608 can be compared with a predetermined threshold. Additionally or alternatively, one or more similarities determined at operation 608 can be sorted. In some examples, at operation 610, the parameter can be matched with at least one skill in the skill repository based on the semantic similarity determined from operation 608. In this regard, operation 610 can include performing a semantic match to determine which skills should be retrieved from the skill repository.

[0095] At operation 612, a subset of skills is retrieved based on the comparison or sorting, such that a subset of skills determined to be relevant to the short description is retrieved from the skill repository. In some examples, the subset of skills is retrieved based on a semantic match. In some examples, the skill repository can be one of a plurality of skill repositories isolated by tenant, such that there is a separate skill repository for each tenant (e.g., individual, entity, organization). In some examples, there is a tenant-restricted skill repository, such that for example, certain skills within the skill repository are only accessible to certain tenants.

[0096] In some examples, the skill repository is one of a plurality of skill repositories, and the plurality of skill repositories includes a private skill repository and a public skill repository. Thus, the skill repository from which the subset of skills is received in operation 612 can be either the public skill repository or the private skill repository. The private skill repository can have access restricted to a particular group of users and / or organizations. For example, the private skill repository can have permissions that exclude certain users and / or systems from accessing the skills in the private skill repository. In some examples, the public skill repository can have no restrictions on access. Alternatively, the public skill repository can have more permissive access restrictions than the private skill repository (e.g., allowing a relatively greater amount of access than the private skill repository).

[0097] In some examples, the skill repository includes one or more of a vector representation, a hierarchical representation, or a skill graph of the skills stored therein. When the skill repository includes a vector representation of the skills stored therein, the vector representation can be stored in at least one of an approximate nearest neighbor (ANN) tree, a k-d tree, an n-dimensional (e.g., multi-dimensional) tree, an octree, or another data structure that one of ordinary skill in the art can identify based on the teachings described herein. One of ordinary skill in the art can recognize additional and / or alternative types of storage mechanisms capable of storing vector space representations.

[0098] In operation 614, skills are received from the subset of skills based on the long description. For example, corresponding similarities between the parameter and each skill in the subset of skills can be determined based on the long description. Additionally, the predetermined threshold can be a first predetermined threshold, and operation 614 can further include comparing one or more of the similarities between the parameter and each skill in the subset of skills with a second predetermined threshold. Additionally or alternatively, one or more of the similarities can be ranked. Operation 614 can further include retrieving skills based on the comparison and / or ranking, such that skills determined to be relevant to the long description are retrieved from the subset of skills.

[0099] In operation 616, the skills are returned. For example, the skills can be returned as output to a user, a generative LLM, a system executing method 600, and / or a remote system of the system executing method 600. Additionally, in some examples, method 600 can further include adapting a computing device to perform an action based on the returned skills.

[0100] In some examples, new skills can be added to the skill repository. A signature mechanism can be used to add new skills to the skill repository. For example, a new skill can be received (e.g., at computing device 102 and / or server 104) for storage in the skill repository. The new skill can be received from a user and / or from the system. The new skill can be verified according to a security protocol. For example, the security protocol can detect what actions the new skill performs, whether the skill includes malware, the resources used to execute the new skill, what permissions are required to execute the new skill, and / or other security verifications that a person of ordinary skill in the art can recognize. After verifying the new skill, the skill repository can be updated to include the new skill.

[0101] Figure 7A and Figure 7B illustrates an overview of an example generative machine learning model that can be used in accordance with the aspects described herein. First, referring to Figure 7A , the conceptual diagram 700 depicts an overview of a pre-trained generative model package 704 in accordance with the aspects described herein, which processes an input 702 to generate a model output for interacting with a skill memory 706. Examples of the pre-trained generative model package 704 include but are not limited to the Megatron-Turing Natural Language Generation Model (MT-NLG), Generative Pre-trained Transformer 3 (GPT-3), Generative Pre-trained Transformer 4 (GPT-4), BigScience BLOOM (Large Open Science Open Access Multilingual Language Model), DALL-E, DALL-E2, StableDiffusion, or Jukebox.

[0102] In an example, the generative model package 704 is pre-trained according to various inputs (e.g., various human languages, various programming languages, and / or various content types), and thus does not need to be fine-tuned or trained for a specific scenario. Instead, the generative model package 704 can be pre-trained more generally such that the input 702 includes a prompt that is generated, selected, or otherwise designed to cause the generative model package 704 to produce a specific generative model output 706. It should be understood that the input 702 and the generative model output 706 can each include any one of various content types, including but not limited to text output, image output, audio output, video output, program output, and / or binary output, etc. In an example, the input 702 and the generative model output 706 can have different content types, such as may occur when the generative model package 704 includes a generative multimodal machine learning model.

[0103] Thus, the generative model package 704 can be used in any of a variety of scenarios, and further, different generative model packages can be used in place of the generative model package 704 without substantially modifying other related aspects (e.g., similar to those described herein with respect to Figures 1 to 6Those aspects described). Thus, the generative model package 704 serves as a tool for performing machine learning processing, where certain inputs 702 to the generative model package 704 are programmatically generated or otherwise determined, such that the generative model package 704 produces a model output 706 that can subsequently be used for further processing.

[0104] The generative model package 704 can be provided or otherwise used according to any of a variety of paradigms. For example, the generative model package 704 can be used locally on a computing device (e.g., Figure 1 the computing device 102 therein), or can be accessed remotely from a machine learning service. In other examples, aspects of the generative model package 704 are distributed across multiple computing devices. In some instances, the generative model package 704 can be accessed via an application programming interface (API), such as an API provided by an operating system of a computing device and / or by a machine learning service and other examples.

[0105] Now referring to the illustrated aspects of the generative model package 704, the generative model package 704 includes input tokenization 708, input embedding 710, model layers 712, output layer 714, and output decoding 716. In an example, the input tokenization 708 processes the input 702 to generate the input embedding 710, which includes a sequence of symbolic representations corresponding to the input 702. Thus, the input embedding 710 is processed by the model layers 712, the output layer 714, and the output decoding 716 to produce the model output 706. In Figure 7B An example architecture corresponding to the generative model package 704 is depicted, which is discussed in further detail below. Even so, it should be understood that the architectures shown and described herein should not be considered restrictive, and in other examples, any of a variety of other architectures can be used.

[0106] Figure 7B is a conceptual diagram depicting an example architecture 750 of a pre-trained generative machine learning model that can be used according to aspects described herein. As described above, any of a variety of alternative architectures and corresponding ML models can be used in other examples without departing from the aspects described herein.

[0107] As shown, the architecture 750 processes the input 702 to produce a generative model output 706, aspects of which are discussed above with respect to Figure 7A discussed. The architecture 750 is depicted as a transformer model including an encoder 752 and a decoder 754. The encoder 752 processes an input embedding 758 (aspects of which can be similar to Figure 7AThe input embedding 710), which includes a sequence of symbolic representations corresponding to the input 756. In an example, the input 756 includes input content 702 corresponding to a content type, aspects of which may be similar to the input data 111, private content 200, and / or public content 250.

[0108] In addition, the positional encoding 760 can introduce information about the relative and / or absolute positions of the tokens of the input embedding 758. Similarly, the output embedding 774 includes a sequence of symbolic representations corresponding to the output 772, and the positional encoding 776 can similarly introduce information about the relative and / or absolute positions of the tokens of the output embedding 774.

[0109] As shown, the encoder 752 includes example layers 770. It should be understood that any number of such layers can be used, and the depicted architecture is simplified for illustrative purposes. The example layer 770 includes two sub-layers: the multi-head attention layer 762 and the feed-forward layer 766. In an example, residual connections are included around each layer 762, 766, followed by normalization layers 764 and 768, respectively.

[0110] The decoder 754 includes example layers 790. Similar to the encoder 752, in other instances any number of such layers can be used, and the depicted architecture of the decoder 754 is simplified for illustrative purposes. As shown, the example layer 790 includes three sub-layers: the masked multi-head attention layer 778, the multi-head attention layer 782, and the feed-forward layer 786. Aspects of the multi-head attention layer 782 and the feed-forward layer 786 can be similar to those discussed above with respect to the multi-head attention layer 762 and the feed-forward layer 766, respectively. Additionally, the masked multi-head attention layer 778 performs multi-head attention on the output of the encoder 752 (e.g., the output 772). In an example, the masked multi-head attention layer 778 prevents positions from attending to subsequent positions. This masking combined with offset embeddings (e.g., offset by one position, as shown in the multi-head attention layer 782) can ensure that the prediction at a given position depends on the known outputs of one or more positions less than the given position. As shown, residual connections are also included around the layers 778, 782, and 786, followed by normalization layers 780, 784, and 788, respectively.

[0111] The multi-head attention layers 762, 778, and 782 can each linearly project queries, keys, and values to corresponding dimensions using a set of linear projections. Each linear projection can be processed using an attention function (e.g., dot product or additive attention) to produce an n-dimensional output value for each linear projection. The resulting values can be concatenated and projected again such that the values are then processed as Figure 7B shown (e.g., by the corresponding normalization layers 764, 780, or 784).

[0112] The feedforward layers 766 and 786 can each be fully connected feedforward networks, which are applied to each position. In an example, the feedforward layers 766 and 786 each include a plurality of linear transformations, with rectified linear unit activations therebetween. In an example, each linear transformation is the same at different positions, and different parameters can be used compared to other linear transformations of the feedforward network.

[0113] Additionally, aspects of the linear transformation 792 can be similar to the linear transformations discussed above with respect to the multi-head attention layers 762, 778, and 782, and the feedforward layers 766 and 786. The Softmax (normalized exponential function) 794 can also convert the output of the linear transformation 792 into predicted next token probabilities, as shown by the output probabilities 796. It should be understood that the illustrated architecture is provided as an example, and in other examples, any of a variety of other model architectures can be used in accordance with the disclosed aspects.

[0114] Thus, the output probabilities 796 can thus form the result output 706 in accordance with the aspects described herein, such that the output of the generative ML model (e.g., which can include a structured output) is used as an input for determining skills in accordance with the aspects described herein (e.g., similar to the skills retrieved by the skill store retrieval component 116). In other examples, the result output 706 is provided as the generated output for interacting with the skill store.

[0115] Figures 8 to 10 And the related description provides a discussion of various operating environments in which aspects of the present disclosure can be practiced. However, the devices and systems shown and discussed Figures 8 to 10 are for purposes of example and illustration, and do not limit the numerous computing device configurations that can be used to practice the aspects of the present disclosure described herein.

[0116] Figure 8 is a block diagram showing the physical components (e.g., hardware) of a computing device 800 in which aspects of the present disclosure can be practiced. The computing device components described below can be applicable to the computing devices described above, including Figure 1 the computing device 102 in. In a basic configuration, the computing device 800 can include at least one processing unit 802 and a system memory 804. Depending on the configuration and type of the computing device, the system memory 804 can include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read only memory), flash memory, or any combination of these memories.

[0117] The system memory 804 may include an operating system 805 and one or more program modules 806 suitable for running software applications 820, such as one or more components supported by the systems described herein. By way of example, the system memory 804 may store a skill coordination engine or component 824, a skill store retrieval engine or component 826, and / or a self-introspection engine or component 828. The operating system 805 may be suitable for controlling the operation of the computing device 800, for example.

[0118] In addition, aspects of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application programs and are not limited to any particular application or system. This basic configuration is illustrated by those components within the dashed line 808 in Figure 8 The computing device 800 may have additional features or functionality. For example, the computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or magnetic tapes. Such additional storage is illustrated by the removable storage device 809 and the non-removable storage device 810 in Figure 8

[0119] As described above, multiple program modules and data files may be stored in the system memory 804. When executed on the processing unit 802, the program modules 806 (e.g., the applications 820) may perform processes including but not limited to the aspects described herein. Other program modules that may be used in accordance with aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, etc.

[0120] In addition, aspects of the present disclosure may be practiced in a circuit that includes discrete electronic elements, a package or integrated electronic chip that contains logic gates, a circuit that utilizes a microprocessor, or a single chip that contains electronic elements or a microprocessor. For example, aspects of the present disclosure may be practiced via a system-on-a-chip (SOC), where Figure 8 each or many of the components shown in

[0121] The computing device 800 may also have one or more input devices 812, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. Output devices 814, such as a display, speaker, printer, etc., may also be included. The above devices are examples, and other devices may be used. The computing device 800 may include one or more communication connections 816 that allow communication with other computing devices 850. Examples of suitable communication connections 816 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuits; universal serial bus (USB), parallel, and / or serial ports.

[0122] As used herein, the term "computer-readable medium" may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 804, removable storage device 809, and non-removable storage device 810 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and that can be accessed by the computing device 800. Any such computer storage media may be part of the computing device 800. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0123] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal having one or more characteristics set or changed to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0124] Figure 9 is a block diagram of an architecture showing one aspect of a computing device. That is, the computing device may include a system (e.g., architecture) 902 for implementing some aspects. In some examples, the system 902 is implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 902 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0125] One or more applications 966 may be loaded into the memory 962 and run on or associated with the operating system 964. Examples of applications include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 902 also includes a non-volatile storage area 968 within the memory 962. The non-volatile storage area 968 may be used to store persistent information that should not be lost if the system 902 loses power. The applications 966 may use and store information in the non-volatile storage area 968, such as emails or other messages used by an email application. A synchronization application (not shown) also resides on the system 902 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 968 synchronized with the corresponding information stored at the host computer. It should be understood that other applications may be loaded into the memory 962 and run on the mobile computing device 900 described herein (e.g., an embedded object memory insertion engine, an embedded object memory retrieval engine, etc.).

[0126] The system 902 has a power supply 970, which may be implemented as one or more batteries. The power supply 970 may also include an external power supply, such as an AC adapter or a power supply dock that supplements or recharges the battery.

[0127] The system 902 may also include a radio interface layer 972 that performs the functions of sending and receiving radio frequency communications. The radio interface layer 972 facilitates a wireless connection between the system 902 and the "outside world" via a communication carrier or service provider. Transmissions to and from the radio interface layer 972 are under the control of the operating system 964. In other words, communications received by the radio interface layer 972 may be propagated to the applications 966 via the operating system 964, and vice versa.

[0128] The visual indicator 920 can be used to provide visual notifications, and / or the audio interface 974 can be used to generate audible notifications via the audio transducer 925. In the example shown, the visual indicator 920 is a light-emitting diode (LED), and the audio transducer 925 is a speaker. These devices can be directly coupled to the power supply 970 so that when activated, they remain on for the duration indicated by the notification mechanism even if the processor 960 and / or the dedicated processor 961 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes an action to indicate the powered-on state of the device. The audio interface 974 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 925, the audio interface 974 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to aspects of the present disclosure, the microphone can also be used as an audio sensor to facilitate control of notifications, as will be described below. The system 902 can also include a video interface 976 that enables the on-board camera 930 to record still images, video streams, and the like.

[0129] The computing device implementing the system 902 can have additional features or functionality. For example, the computing device can also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or magnetic tapes. Such additional storage is Figure 9 shown by the non-volatile storage area 968.

[0130] The data / information generated or captured by the computing device and stored by the system 902 can be stored locally on the computing device as described above, or the data can be stored on any number of storage media that the device can access via the radio interface layer 972 or via a wired connection between the computing device and an associated computing device (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed by the computing device via the radio interface layer 972 or via a distributed computing network. Similarly, such data / information can be easily transmitted between computing devices for storage and use according to well-known data / information transmission and storage means, including email and collaborative data / information sharing systems.

[0131] Figure 10 An aspect of the architecture of a system for processing data received at a computing system from a remote source (e.g., a personal computer 1004, a tablet computing device 1006, or a mobile computing device 1008) is shown, as described above. The content displayed at the server device 1002 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 1024, a web portal 1025, a mailbox service 1026, an instant messaging repository 1028, or a social networking site 1030.

[0132] The application 1020 (e.g., similar to application 820) can be adopted by a client communicating with the server device 1002. Additionally or alternatively, the skill coordination engine 1021, the skill repository retrieval engine 1022, and / or the introspection engine 1023 can be adopted by the server device 1002. The server device 1002 can provide data to and from client computing devices such as a personal computer 1004, a tablet computing device 1006, and / or a mobile computing device 1008 (e.g., a smart phone) via the network 1015. As an example, the above computer system can be embodied in a personal computer 1004, a tablet computing device 1006, and / or a mobile computing device 1008 (e.g., a smart phone). In addition to receiving graphical data that can be used for preprocessing at the graphics origination system or postprocessing at the receiving computing system, any of these examples of computing devices can obtain content from the storage 1016.

[0133] For example, aspects of the present disclosure have been described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to aspects of the present disclosure. The functions / actions indicated in the blocks may not occur in the order shown in any flowchart. For example, depending on the functions / actions involved, two consecutive blocks shown may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0134] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the claimed aspects of the present disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Whether shown and described combinatorially or individually, various features (structures and methods) are intended to be selectively included or omitted to yield aspects with a particular set of features. Having provided the description and illustration of this application, those skilled in the art can envision variations, modifications, and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application, which do not depart from the broader scope of the claimed disclosure.

Claims

1. A method for interacting with a skill repository, the method comprising: Using a generative large language model (GLM) to process a task to coordinate skills for performing the task, wherein the coordinated skills include a plurality of skills related to the task; Determining that at least one of the coordinated skills is not available to the GLM; Sending an indication corresponding to the at least one skill to a remote skill repository; Receiving a remote skill corresponding to the sent indication from the remote skill repository; And Performing the task.

2. The method according to claim 1, wherein determining that at least one of the coordinated skills is not available to the GLM comprises: Determining that at least one of the coordinated skills does not correspond to one or more local skills stored on a local device.

3. The method according to claim 1, wherein said sending an indication corresponding to said at least one skill to the remote skill repository comprises: Generating a description of the at least one skill; And Wherein receiving the remote skill includes: receiving the remote skill based on the description of the at least one skill.

4. The method according to claim 3, wherein said receiving the remote skill comprises: Performing introspection on the remote skill based on the description to generate a confidence threshold corresponding to the remote skill and the task.

5. The method according to claim 3, wherein the description is a short description, and wherein the sending of an indication corresponding to the at least one skill to the remote skill repository further comprises: Generating a long description of the at least one skill, and wherein receiving the remote skill includes: receiving the remote skill based on the generated long description and the generated short description of the at least one skill.

6. The method according to claim 5, wherein the generated long description and the generated short description include natural language or intermediate language.

7. The method according to claim 1, wherein the task is processed using the GLM, and wherein the GLM uses the remote skill to perform the task.

8. The method according to claim 1, wherein one or more of the coordinated skills are stored on a local device, and wherein the one or more skills stored on the local device are stored in one of a common skill repository or a private skill repository.

9. The method according to claim 1, wherein one or more of the plurality of skills include machine-executable instructions, and wherein performing the task includes: Adapting one or more devices to execute the machine-executable instructions.

10. The method according to claim 1, further comprising: Before performing the task, providing a notification to a user corresponding to the remote skill, and receiving feedback from the user to verify the remote skill in response to the notification.

11. A method for interacting with a skill repository, the method comprising: Receiving parameters for a skill repository from a local device, wherein the parameters are associated with a short description and a long description; Receiving a subset of skills from the skill repository based on the short description, wherein retrieving the subset of skills includes: Based on the short description, determining a corresponding similarity between the parameters and each skill stored in the skill repository; Comparing one or more of the similarities with a predetermined threshold, or sorting one or more of the similarities; and Retrieving the subset of skills based on the comparison or the sorting, thereby retrieving from the skill repository a subset of skills determined to be related to the short description; Receiving skills from the subset of skills based on the long description; Returning the skills.

12. The method according to claim 11, wherein each of the short description and the long description comprises one of natural language data or an embedding.

13. The method according to claim 11, wherein the skill repository comprises one or more of a vector representation, a hierarchical representation, or a graph of the skills stored therein.

14. The method according to claim 13, wherein the skill repository comprises the vector representation of the skills stored therein, and wherein the vector representation is stored in at least one of an approximate nearest neighbor (ANN) tree, a k-d tree, or a multi-dimensional tree.

15. A system for interacting with a skill repository, the system comprising: a processor; and a memory storing instructions which, when executed by the processor, cause the system to perform a set of operations, the set of operations comprising: receiving, from a local device, parameters for the skill repository, wherein the parameters are associated with a description; receiving, from the skill repository, a subset of skills based on the description, wherein retrieving the subset of skills comprises: determining, based on the description, a respective semantic similarity between the parameters and each of the skills stored within the skill repository; matching the parameters with at least one of the skills within the skill repository based on the determined semantic similarity; and retrieving the subset of skills based on the matching, thereby retrieving from the skill repository a subset of skills determined to be relevant to the description based on semantic similarity; returning the subset of skills.

Images