On-device language model for syntax-guided responses

By managing adapters and distillation models on personal computing devices, the security and fast access issues of language models in third-party applications are solved, and the constraint decoding process is simplified, enabling efficient and structured output.

CN123114489APending Publication Date: 2026-11-17APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610719720.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-11-19
Filing Date
2026-05-12
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, language models on personal computing devices are difficult to securely and quickly provide robust access and efficient language model services to third-party applications, and the process of constraining the output of language models is complex.

Method used

By managing adapters and distillation models on personal computing devices, third-party applications can access the language model, and a compiler can be used to generate grammars to simplify the constraint decoding process and ensure the structured nature of the output.

Benefits of technology

It enables secure and rapid provision of language model services to third-party applications on personal computing devices, simplifies the constraint decoding process, and improves the structure and efficiency of the output.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114489A_ABST
    Figure CN123114489A_ABST
Patent Text Reader

Abstract

This disclosure relates to on-device language models for syntax-guided responses. The technology relates to executing language models on personal computing devices, thereby providing enhanced privacy and security compared to cloud-based solutions. This technology addresses the challenge of allowing robust access to on-device language models by third-party applications while maintaining secure operation. Key innovations include an adapter management system that bridges the gap between application and system space, enabling seamless integration of fine-tuned models. Additionally, the technology proposes a mechanism for developers to create distilled models that are smaller and more efficient, ensuring optimal performance on resource-constrained devices. The technology also facilitates simplified integration via a software development kit (SDK), enabling third-party applications to utilize system APIs. The technology further simplifies the process of generating structured output from language models, allowing third-party applications to easily leverage on-device language models to interact with tools via APIs and provide responses consistent with the programming language type.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of Chinese invention patent application filed on May 12, 2026, with national application number 202610646642.1 and invention title "On-device language model for syntax-guided response". Background Technology

[0002] To date, the benefits of running language models on personal computing devices (i.e., on-device language models) have been largely limited to operations performed by the operating system of the personal computing device or to applications developed by the operating system's developers (first-party applications).

[0003] One use of language models is to interpret unstructured data. While language models excel at understanding unstructured data, they are not good at producing strictly structured responses. Attached Figure Description

[0004] Details of one or more aspects of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. However, the drawings illustrate only some typical aspects of this disclosure and should not be considered as limiting the scope of this disclosure. Other features, aspects, and advantages will become apparent from the description, drawings, and claims.

[0005] Figure 1 Example system implementations based on some aspects of this technology are illustrated.

[0006] Figure 2 Example routines are illustrated according to some implementations of this technology for using a compiler to create grammars for constraining the output of a language model (constrained decoding).

[0007] Figure 3 Example routines for creating and using prefix trees to assist in a constrained decoding process are illustrated according to some implementations of this technology.

[0008] Figure 4A The grammar defines the string "hello world".

[0009] Figure 4B An example is shown after the prefix tree has been sampled for the word "h".

[0010] Figure 5 The timeline illustrates various processes that occur during the inference operation of a language model when constrained decoding is used, according to some embodiments of the present technology.

[0011] Figure 6 Examples of the efficiency provided by a constrained decoding system according to some implementations of this technology are illustrated.

[0012] Figure 7Example routines for processing grammars with another grammar nested within them, according to some implementations of this technology, are illustrated.

[0013] Figure 8A and Figure 8B Example sequence diagrams illustrating some implementations of the present technology are shown, illustrating instances where an application transmits instructions to a language model for generating grammatically constrained output.

[0014] Figure 9 Example routines for calling one or more tools by a language model are illustrated according to some implementations of this technology.

[0015] Figure 10 Example routines for creating fine-tuning models and distillation models according to some implementations of this technology and for managing these assets by app stores and client devices are illustrated.

[0016] Figure 11 Examples of adapters and distillation models residing in the application space and copied to the system space for inference processes according to some embodiments of the present technology are illustrated.

[0017] Figure 12 Example routines for using adapters and distillation models in the inference process are illustrated according to some implementations of this technology.

[0018] Figure 13A , Figure 13B and Figure 13C Example transformer architectures are illustrated according to some implementations of this technology.

[0019] Figure 14 This is a system diagram illustrating some implementations of the device according to the present technology.

[0020] Figure 15 Examples of systems for implementing this technology are illustrated. Detailed Implementation

[0021] Various embodiments of this disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other components and configurations can be used without departing from the spirit and scope of this disclosure.

[0022] Additional features and advantages of this disclosure will be set forth in the following description and will be apparent in part from that description, or may be learned by practicing the principles disclosed herein. The features and advantages of this disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of this disclosure will become more fully apparent from the following description and the appended claims, or may be learned by practicing the principles set forth herein.

[0023] Language models continue to demonstrate tremendous potential in many different applications. However, many language models operate in cloud environments, which are inherently less secure than those that run entirely or even primarily on personal computing devices.

[0024] Running language models on personal computing devices is possible, and while personal computing devices are more resource-constrained than public clouds, language models still perform remarkably well. For example, personal computing devices have less powerful GPUs or neural engines, less memory, and typically rely on battery power. These trade-offs are worthwhile in many instances because running language models on personal computing devices provides greater privacy and security for user account data.

[0025] To date, the benefits of running language models on personal computing devices (i.e., on-device language models) have been largely limited to operations performed by the operating system of the personal computing device or to applications developed by the operating system's developers (first-party applications). This technology describes several mechanisms that make language models on personal computing devices accessible to third-party applications.

[0026] A key challenge is providing developers with robust access to on-device language models while ensuring these models can be manipulated securely. More specifically, some operating systems run on-device language models in a secure environment in system space, while applications tend to run in a sandboxed environment in application space. This becomes challenging when application developers want to run fine-tuned versions of the language model, which typically requires providing an adapter weight layer along with the language model's weights to memory. When the language model is in system space, the adapter cannot be in application space. This technique addresses this by managing the adapter and copying it to memory in system space when it is used.

[0027] Another challenge is that running a full version of a language model on a personal computing device can be slow due to its limited resources. Furthermore, slower performance may not deliver the user experience developers want to provide. Therefore, this technology provides developers with the ability to create their own distilled models, which are smaller, compressed, and more efficient models that typically produce output comparable to larger on-device language models.

[0028] A potential advantage of granting third-party developers access to the language model on the device is that the operating systems of these personal computing devices are typically associated with software development kits (SDKs). These SDKs facilitate the integration of third-party applications with operating system services and services provided by other third-party applications. These SDKs greatly simplify the development of third-party applications, making them more functional. Typically, these SDKs define the APIs that applications can call. Now, when third-party applications gain access to the language model on the device, they will expect the language model on the device to utilize these APIs for its own purposes.

[0029] APIs are typically structured grammars, and constraining language models to provide output in a structured manner can be challenging. This process usually requires defining a grammar to constrain the language model's output. This technique aims to simplify this process for third-party developers by providing only the data's type, and it interacts with the compiler to generate the grammar. This approach eliminates the need for JSON schema definition and parsing, ensuring developers can easily receive well-structured output.

[0030] As described above, some aspects of this technology specifically enhance the privacy of users of personal computing devices. However, this technology may include the collection and use of data from various sources. This disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate specific individuals. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, user account identifiers, home addresses, data or records relating to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information. This disclosure recognizes that the use of such personal information data in the present invention's technology can be used to benefit users.

[0031] This disclosure anticipates that entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with robust privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to privacy policies and measures that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of these legitimate purposes. Furthermore, such collection / sharing should be conducted only after receiving informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard the right to access such personal information data and ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Furthermore, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and privacy measures. Moreover, policies and measures should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including considerations of specific jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0032] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to their personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. This technology can be configured to allow users to opt-in or opt-out at any time during or after service registration. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0033] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0034] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data.

[0035] Regardless of the foregoing, this disclosure also contemplates implementations that allow users to selectively restrict and / or block the use and / or access to data. That is, this disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to data. For example, with respect to some services, the inventive technology should be configured to allow users to opt-in or opt-out to participate in data collection at any time during or after service registration. In another example, the inventive technology should be configured to allow users to opt out of providing certain data used for training AI / ML systems and / or as input during the inference phase of such systems. In yet another example, the inventive technology should be configured to allow users to choose to limit the length of time data is retained or to completely prohibit AI / ML systems from using their data. In addition to providing "opt-in" and "opt-out" options, this disclosure also contemplates providing notifications related to access to or use of personal information. For example, users may be notified when their data is input into an AI / ML system for training or inference purposes, and / or alerted when the AI / ML system generates output or makes decisions based on their data.

[0036] This disclosure recognizes that AI / ML systems should incorporate explicit limitations and / or oversight to mitigate risks that may still exist even when such systems have been designed, developed, and / or operated in accordance with industry best practices and standards. For example, outputs may be generated that could be considered erroneous, harmful, offensive, and / or biased; such outputs may not necessarily reflect the opinions or positions of the entity that developed or deployed these systems. Furthermore, in some cases, references to third-party products and / or services in these outputs should not be construed as endorsement or affiliation of the third-party products and / or services by the entity providing the AI / ML system. The generated content may be filtered to remove potentially inappropriate or dangerous material before being presented to users, while maintaining the ability for human oversight and / or to cover or correct erroneous or undesirable outputs as a safeguard.

[0037] This disclosure also anticipates that users of the AI / ML system should avoid using the service in any way that infringes upon, misappropriates, or violates the rights of any party. Furthermore, the AI / ML system should not be used for any illegal or unlawful activities, nor should it be used to develop any application or use case that will commit or assist in the committing of a crime or other tortious, illegal, or unlawful act. The AI / ML system should not violate, misappropriate, or infringe upon any party's copyright, trademark, privacy and portrait rights, trade secrets, patents, or other proprietary or statutory rights, and should be appropriately attributed to the content as required. In addition, the AI / ML system should not interfere with any security, digital signature, digital rights management, content protection, verification, or authentication mechanisms. The AI / ML system should not misrepresent machine-generated output as human-generated.

[0038] Figure 1 Example system implementations according to some aspects of the present technology are illustrated. Although the example system depicts specific system components and arrangements of such components, this description is for the purpose of facilitating the discussion of the present technology and should not be considered limiting unless specified in the appended claims. For example, some components illustrated as separate may be combined with other components, some components may be divided into separate components, some components may be absent or unnecessary, and other components may be present.

[0039] Figure 1 Example: Developer 116 develops application 120, which is subsequently published in app store 118. Users can operate client device 102 to download application 120 and run application 120 on client device 102.

[0040] More specifically, developer 116 may develop application 120 configured to utilize the base language model 122 on client device 102. In some implementations, the application may be configured to use the fine-tuned base language model by using adapter file 110. As will be explained in more detail herein, adapter file 110 may include an adapter to the base language model 122 that enables the base language model 122 to be used as the fine-tuned base language model. Adapter file 110 may also optionally include a distillation model. The distillation model is distilled from a dataset created based on the fine-tuned base language model. The distillation model is configured to generate a smaller model than the fine-tuned base language model with less computational resources, producing a response comparable to the fine-tuned base language model.

[0041] like Figure 1As illustrated, developer 116 can create application 120 with adapter file 110. Adapter file 110 can be hosted by an app store along with application 120. In some implementations, multiple versions of adapter file 110 may exist, which are related to the version of the underlying language model 122 or other software, such as the operating system on client device 102.

[0042] As will be explained in this article, application 120 and adapter file 110 may be downloaded to client device 102, but they may not all be downloaded simultaneously. The dashed lines indicate that these files may be downloaded during multiple transactions.

[0043] Client device 102 can be any computing device other than those in a data center. Some specific examples of anticipated computing devices include smartphones, tablets, spatial computing devices, smart devices (such as IoT devices with various form factors), laptops, etc. Client device 102 includes application space 108 and system space 104. Application space 108 is a sandboxed space dedicated to application processes and their files. System space 104 is a space for trusted processes (such as those that are part of the operating system).

[0044] Client device 102 includes an on-device language model, labeled as base language model 122. This language model is an AI engine that executes on the client device. As used herein, the indexing of language model, base language model, on-device language model, or fine-tuned base language model refers to the language model operating on client device 102. Furthermore, while the language model at its core may be an autoregressive transformer model composed of weights, this description anticipates that various supporting services may be used with the language model, and the indexing of the language model explicitly includes such supporting services. For example, when a cue word is passed to the language model, the cue word may first be passed to a deterministic tool that supports the language model. Alternatively, when the language model is described as creating a constrained output, that output may be created using a deterministic tool such as constrained decoding system 124. Therefore, the indexing of the language model should be interpreted as including both the language model itself and the supporting processes.

[0045] As illustrated herein, one advantage of the base language model 122 is its ability to interact with various tools, such as tools / other applications 112 and external tools 114. Tools / other applications 112 can be tools provided by the operating system, or they can be other applications, including third-party applications, which are accessible to the base language model 122 via application programming interfaces (APIs) provided by the operating system's software development kit (SDK). External tools 114 can be tools, websites, and data sources available via the Internet.

[0046] Whether the tool is a tool / other application 112 or an external tool 114, these tools are typically invoked via APIs, which are programmatic calls that usually require communication in a defined syntax. The constrained decoding system 124 is a system that supports the underlying language model 122 in constraining its output to match the defined syntax when needed.

[0047] Finally, client device 102 illustrates that application 120 on the client device may have adapter files 110 stored in application space 108. However, when such adapter files 110 need to be used by the underlying language model 122, these adapter files may be copied to system space 104, as represented as adapter 106.

[0048] This article will refer to Figure 1 This technology will be described in more detail.

[0049] Figure 2 Example routines according to some embodiments of the present technology are illustrated for using a compiler to create grammars for constraining the output of a language model (constrained decoding). Although the example routines depict a specific order of operations, this order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routines may perform their functions substantially simultaneously or in a specific order.

[0050] Because language models are not deterministic by nature, they typically do not produce the same output for the same cue word every time it appears. While this behavior realizes many of the potential benefits of language models, it is also an obstacle to using them in tasks requiring specific grammar. Constrained decoding in language models offers significant benefits by enabling these models to interact more effectively with external tools and perform a broader array of functions. By incorporating constraints during the decoding process, language models can adhere to predefined patterns, syntactic structures, or output formats necessary for interfacing with a specific application or system. This capability ensures higher accuracy and reliability when models are assigned to generate responses that must conform to stringent requirements, such as API calls, database queries, or controlled natural language commands. Therefore, constrained decoding enhances the usability of language models in practical applications by allowing them to operate seamlessly within complex workflows, perform tasks requiring precision, and increase their utility in diverse domains beyond simple text generation.

[0051] Developing patterns for guiding constrained decoding within a language model presents several challenges, and is at least a burden for application developers. These difficulties stem primarily from the need to accurately define the constraints that prescribe valid outputs while maintaining the flexibility and generalization capabilities of the language model. Creating comprehensive schemes requires a deep understanding of the task domain, the specific patterns or structures needed for successful execution, and the potential variations in the input data. Additionally, the pattern must be robust enough to adapt to edge cases and changes, which typically necessitates extensive testing and iteration. Often, these patterns are limited to representation in a single programming language, such as JSON.

[0052] To alleviate some of the burden of developing patterns for guiding constrained decoding, this technique only requires developers to provide a grammar for the expected response by providing the data type in the grammar, and the technique will interact with the compiler to generate the grammar based on the compiler's understanding of the type. This approach eliminates the need for JSON schema definition and parsing, thus ensuring that developers can easily receive well-structured output.

[0053] According to some examples, the method includes receiving instructions at box 202 for providing at least some responses using a first syntax acceptable to the type in a first programming language. For example, Figure 1 The illustrated base language model 122 can receive instructions for providing at least some responses using a first syntax that is type-acceptable in a first programming language. These instructions can be configured to appear in the form of a message or session initialization message or prompt (based on which the language model will generate responses). In some implementations, the first syntax may define one or more types in the SWIFT programming language, which is commonly used in Apple's iOS operating system. SWIFT is a strongly typed language.

[0054] A key advantage of this technology is that developers do not need to specify a programming language or first grammar. The underlying language model 122, which works with the constrained decoding system 124, identifies the language and type, and can determine the first grammar based on the type. Example instructions are able to output data in the following format:

[0055] struct Person {

[0056] var name: String

[0057] var age: Int

[0058] }

[0059] According to some examples, the method includes retrieving, at box 204, rules defining the first syntax for types in the first programming language from the compiler of the first programming language. For example, Figure 1The illustrated base language model 122 can retrieve the rules of the first syntax that define the types in the first programming language from the compiler of the first programming language.

[0060] In some implementations, the compiler is aware of the requirements for SWIFT types, such as String and Int. In some implementations, the compiler is able to convert between the SWIFT programming language and the JSON programming language, which can be useful when the language model is better suited to generating code using JSON.

[0061] Of course, as long as the compiler can provide a language-specific grammar, these examples of SWIFT and JSON can be applied to any decoded language.

[0062] According to some examples, the method includes creating a grammar at box 206 based on rules from the compiler, which defines the lexical units that can be output by the language model. For example, Figure 1 The illustrated base language model 122 can create a grammar based on rules from a compiler, which defines the lexical units that can be output by the language model. In some examples, the grammar can be a Backus-Naur Form (BNF) grammar, but this technique is not limited to this type of grammar.

[0063] An example grammar using JSON based on the SWIFT struct "Person" could be:

[0064] <person>= {"name": " <characters>", "age": <digits>}

[0065] According to some examples, the method includes generating a response to at least one prompt word at box 208. For example, Figure 1 The illustrated base language model 122 generates a response to at least one prompt word. The prompt word can be provided along with the instruction at box 202, or it can be provided in subsequent communication. Note that neither the instruction nor the prompt word needs to specify the output programming language for the language model. This complexity has been eliminated, eliminating the need for developer intervention. Instead, the language model is trained to output a code language that matches the grammar (e.g., JSON in this example). Developers only need to provide the type in the first programming language (e.g., SWIFT in this example).

[0066] When generating a response, the language model is constrained by the grammar to output valid lexical units within the first grammar. The grammar constrains the language model by masking valid lexical units from the candidate lexical unit distribution, thus limiting the language model to selecting valid lexical units from that distribution. Figures 3 to 4B The process will be described in more detail in the text.

[0067] While the language model may output lexical terms that match the grammar of a second programming language (e.g., JSON in this example), the instruction uses a first programming language (e.g., SWIFT in this example), meaning the developer intends the response to use the first programming language. Therefore, generating a response to the prompt at box 208 may also include converting an internal response using the second programming language into a response using the second programming language. In some implementations, the language model performs this conversion. In some implementations, the language model may invoke a tool that can deterministically perform the conversion. For example, in Apple iOS, there is a tool called GENERABLE for converting between JSON and SWIFT (and vice versa).

[0068] According to some examples, the method includes returning a response at box 210 that uses a first syntax acceptable in a first programming language for at least one prompt word. For example, Figure 1 The illustrated base language model 122 can return a response to at least one prompt word using a first syntax that is type-acceptable in a first programming language.

[0069] Figure 3 Example routines for creating and using prefix trees to assist a constrained decoding process, according to some embodiments of the present technology, are illustrated. Although the example routines depict a specific order of operations, this order may be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routines may perform functions substantially simultaneously or in a specific order.

[0070] Although language models can evaluate grammars during the inference process, using a deterministic process to mask invalid lexical units is significantly more efficient; by using prefix trees ( Figure 4B An example prefix tree is shown, which can make the process more efficient. The constrained decoding system 124 represents this deterministic process.

[0071] According to some examples, the method includes creating a prefix tree based on the grammar at box 302. For example, Figure 1 The illustrated constrained decoding system 124 can create a prefix tree based on the grammar.

[0072] like Figure 4A As illustrated, the grammar defines the string "hello world", and only this string is valid. For illustrative purposes, Figure 4A It is assumed that there exists a finite set of lexical units to illustrate the example in the diagram, but in reality, the set of lexical units is much larger. Based on the set of possible lexical units, a prefix tree can be created either in advance or dynamically. The prefix tree consists of branches representing valid character / lexical sequences.

[0073] Figure 4B The example shows the prefix tree after sampling the lexical "h". Now, subsequent lexical prefixes are restricted to "i" or "e". Since any lexical whose character after "h" is "i" cannot effectively achieve the goal of "hello world", we can mask the lexical "hi" and "him" (we have already masked "world", "wo", and "stop" because they do not start with "h"). Any masked lexical has a 0% probability of being a correct lexical, so it makes no sense for the language model to treat them as possible lexicals; therefore, they can be masked to prevent them from being selected.

[0074] After the prefix "e", the only other possible character prefixes in our original vocabulary are "l" and "y". Once "y" is excluded because it does not achieve the goal of "hello world", we can mask out "hey", and we are left with only one lexical for sampling: "hello".

[0075] According to some examples, the method includes selecting, at box 304, a specific branch of the prefix tree corresponding to a specific existing character sequence generated by the language model as part of the response. For example, Figure 1 The illustrated constrained decoding system 124 can select a specific branch of the prefix tree corresponding to a specific existing character sequence generated by the language model as part of the response. The specific branch corresponds to a valid candidate lexical. This process is repeated until either the constrained decoding system 124 or the underlying language model 122 selects a valid lexical.

[0076] According to some examples, the method includes masking valid candidate lexical units at box 306 to restrict the language model from selecting from valid candidate lexical units. For example, Figure 1 The illustrated constrained decoding system 124 can mask valid candidate lexical units to restrict the language model from selecting from valid candidate lexical units. The language model can only select valid candidate lexical units as the next lexical unit in the response.

[0077] In some instances (such as a simple example for the grammar of "hello world"), the constrained decoding process may only achieve one possible lexical from which the language model can select. In such instances, the method includes, at box 308, injecting a single valid candidate lexical into the language model's response when the grammar indicates the presence of such a lexical, without the language model selecting the single valid candidate lexical as part of its inference process. For example, Figure 1 The illustrated constrained decoding system 124 can inject a single valid candidate lexical into the response of the language model. This feature of the technique also achieves another efficiency, in Figure 5 This efficiency is illustrated.

[0078] Figure 5 The timeline illustrates various processes that occur during the inference operation of a language model when constrained decoding is used, according to some embodiments of the present technology.

[0079] exist Figure 5 In the diagram, the timeline progresses from left to right. As illustrated, the first process to be initiated is lexicalization process 502, in which the prompt word is lexicallyized. Lexicalization is the process of breaking down the prompt word (input text) into smaller units called lexical units. These lexical units can include words, subwords, or even characters, depending on the lexicalization method used.

[0080] After the cue words are lexicalized, the constrained sampling process 504 and the model interference process 508 start roughly simultaneously, but the constrained sampling process 504 is faster. This means that when the constrained sampling process 504 can mask all but one lexical from the invalid lexical 506 it samples, the model interference process 508 can be terminated for that lexical, and the process can jump to the selection lexical 510, which selects the only valid lexical and restarts the constrained sampling process 504 and the model interference process 508 to determine the next lexical in the sequence. This reveals that when lexical selection can be deterministically performed by the constrained sampling process 504, the constrained sampling process 504 can significantly accelerate the model interference process 508 used for language modeling.

[0081] Another advantage of this technology is that the constrained sampling process 504 is a CPU-based process. CPUs are typically more efficient, so the constrained sampling process 504 not only saves time but also conserves valuable GPU resources and power, as GPUs are relatively power-intensive.

[0082] Figure 6 Example efficiencies of some implementation schemes according to this technology are illustrated. Figure 6 In this context, the grammar requests the language model output to be a job title in string form. In this case, Erik's job title is engineer, so once it is clear that the output will begin with "e", the constrained decoding system 124 can inject the job title "engineer" without waiting for the inference process to eventually sample the same lexical.

[0083] Figure 7 Example routines for processing grammars having another grammar nested therein, according to some embodiments of the present technology, are illustrated. Although the example routines depict a particular order of operations, this order may be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routines may perform functions substantially simultaneously or in a particular order.

[0084] Sometimes, the desired structured output may include structures nested within structures. An example would be JSON code including topic tags defined by the REGEX grammar. In this example, the entire output needs to conform to the grammar defining the JSON code, and the topic tags also need to conform to the REGEX grammar. It's possible that an instance requiring both grammars to generate a response might occur when a user enters a dotted star (.*) within a REGEX string, where the REGEX string is nested within JSON code. While .* is generally allowed to represent any character within a REGEX string, due to JSON limitations, it cannot represent any character within a JSON string. Therefore, .* is not allowed within a REGEX string appearing within a JSON string. As discussed in [the article]... Figure 2 This can be established by providing language model instructions that define a second grammar within a first grammar. The first grammar is associated with the first grammar, and the second grammar is associated with the second grammar, and the second grammar is constrained by both the first and second grammars.

[0085] According to some examples, the method includes calling the first grammar at box 702 to generate the first part of the lexical unit. For example, Figure 1 The illustrated base language model 122 can invoke the first grammar to generate the first part of a lexical unit. The language model is trained to determine when it should invoke the constrained decoding system 124 to apply the first grammar, and the language model can invoke the constrained decoding system 124 to apply the first grammar when the language model crosses such a decision boundary.

[0086] According to some examples, the method includes determining, at decision box 704, after generating the first part of a lexical using the first grammar, whether to continue using the first grammar, use the second grammar, or continue generating lexicals unconstrained by the grammar. The language model is trained to repeatedly determine whether to invoke a grammar and which grammar it should invoke. In many instances, the language model may perform this determination after it generates specific lexicals that may indicate a transition. This behavior allows the language model to be particularly agile and switch between grammars.

[0087] According to some examples, the method includes, at box 706, after generating the first part of the lexical unit, calling a second grammar to generate the second part of the lexical unit. For example, Figure 1 The illustrated base language model 122 can make a determination at decision box 704, and after generating the first part of the lexical unit, call the second grammar to generate the second part of the lexical unit.

[0088] According to some examples, the method includes generating grammatically unconstrained lexical units at box 708. For example, Figure 1 The illustrated base language model 122 can generate lexical units that are not bound by grammar after making a determination at decision box 704.

[0089] Figure 8A and Figure 8B Example sequence diagrams illustrating some embodiments of the present technology are shown, illustrating instances where an application transmits instructions to a language model for generating grammatically constrained output. Although the example routines depict a specific order of operations, this order may be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routine may perform functions substantially simultaneously or in a specific order.

[0090] exist Figure 8A In this context, at least one prompt word may be a request to generate a response, wherein at least a portion of the response will be returned using the first syntax. Figure 8B At least two additional use cases are illustrated. First, the application transmits instructions for configuring the language model to generate a response using the first syntax before transmitting the prompt word that requires a response using the first syntax. Second, Figure 8B An example is provided where the language model is configured to invoke tools in communication using a first grammar to assist in generating responses to prompt words.

[0091] According to some examples, the method includes transmitting at box 802 instructions as part of at least one prompt word for providing at least some responses using a type-acceptable first syntax in a first programming language. For example, Figure 1 The illustrated application 120 may transmit instructions as part of at least one prompt word for providing at least some responses using a first syntax acceptable in a first programming language. In this example, the at least one prompt word may be a request to generate a response, wherein at least a portion of the response will be returned using the first syntax.

[0092] According to some examples, the method includes creating a grammar at box 804 based on rules collected from a compiler, which defines the lexical units that can be output by a language model. For example, Figure 1 The illustrated base language model 122 can create a grammar based on rules collected from a compiler, which defines the tokens that can be output by the language model. In this example, the base language model 122 may need to call the constrained decoding system 124 to create the grammar in order to generate a response. This step will introduce additional latency when providing the response.

[0093] According to some examples, the method includes generating a response using the first syntax at box 806. For example, Figure 1 The illustrated base language model 122 can generate a response using the first grammar. The grammar constrains the language model to output lexical units that are valid within the first grammar.

[0094] According to some examples, the method includes receiving at box 808 a response to at least one prompt word using a first syntax acceptable in a first programming language. For example, Figure 1 The illustrated application 120 can receive a response to at least one prompt word using a first syntax that is type-acceptable in a first programming language.

[0095] As introduced above, in Figure 8B In this method, at box 810, instructions are transmitted for configuring the language model to invoke a tool, wherein the instructions define a type-acceptable first syntax in a first programming language to be used when invoking the tool. These instructions may be transmitted before prompts that might require inference operations to be performed by the language model. More specifically, instructions may be transmitted when initiating a session between application 120 and the underlying language model 122, or even during configuration events that may occur when application 120 is first installed on a client device.

[0096] Although box 810 illustrates configuring the base language model 122 to both generate output using the grammar and invoke tools, these two use cases do not need to coexist. It is possible that the instructions only request the base language model 122 to provide output using the first grammar, as discussed in [the previous section]. Figure 8A As explained. When the instruction is used to generate a response using the first syntax, Figure 8A and Figure 8B The differences illustrated are only in timing. Figure 8A In the text, the instructions appear together with the prompt words, and... Figure 8B In the instruction, the basic language model 122 is configured before the first prompt word.

[0097] Tool invocation is a common use case for configuring the base language model 122 to generate output using a first syntax. Tool 830 can be a tool on client device 102, such as tool / other application 112, or a tool on the Internet, such as external tool 114. Tool 830 is typically accessible via an API defined by a specific syntax. Therefore, one use case for configuring the base language model 122 to output a response using the first syntax is to enable the base language model 122 to reliably invoke tool 830 using the first syntax.

[0098] According to some examples, the method includes creating a grammar at box 812 based on rules collected from the compiler, which defines the lexical units that can be output by the language model. This step is similar to... Figure 8A Illustrated block 804.

[0099] According to some examples, the method includes transmitting at least one prompt word at box 814. Figure 8B In this context, the prompt words appear after the base language model 122 has been configured to generate a response using the first syntax and / or to invoke tool 830. Although the configuration only needs to occur once per session, many prompt words can be delivered to take advantage of the configuration.

[0100] According to some examples, the method includes invoking a tool across a decision boundary at box 816. For example, a base language model 122 can cross the decision boundary and determine tool invocation in response to at least one prompt word. The base language model 122 is trained based on configuration information for the tool (about... Figure 9 (Further explanation) to make the decision to invoke the tool. When the base language model 122 determines that it should invoke the tool to facilitate a response to the prompt word, the base language model 122 will need to generate an API call using a first grammar at box 818 for invoking the tool. For example, the base language model 122 may generate an API call assisted by the constrained decoding system 124 to ensure that the output lexical is valid within the first grammar of the API call.

[0101] Then, the basic language model 122 can make API calls to tool 830.

[0102] According to some examples, the method includes receiving an API call using valid first syntax at box 820. For example, Figure 8B The illustrated tool 830 can receive API calls using valid first syntax. In some implementations, tool 830 is a tool provided by a third party (other than the operating system or language model publisher).

[0103] Based on some examples, the method includes responding to API calls at box 822. For example, Figure 8B The illustrated tool 830 can respond to API calls.

[0104] According to some examples, the method includes copying the response from the first tool to a context window used for the first session at box 824. For example, Figure 1 The illustrated base language model 122 can copy the response from the first tool into a context window used for the first session. This is a key improvement over the prior art, in which language models typically cannot interact directly with third-party tools.

[0105] According to some examples, the method includes using the response from the first tool at box 826 to generate a response to the first prompt word. For example, Figure 1 The illustrated base language model 122 can use the response from the first tool to generate a response to the first prompt word. Following box 824, a context window includes the first prompt word and the response from the first tool, and the base language model 122 generates the response based on the current content of the context window. In some embodiments, such as regarding... Figure 9 To elaborate further, the basic language model 122 can optionally call another tool.

[0106] Once the base language model 122 has completed generating a response to at least one prompt word, the base language model 122 can transmit the response to the application 120. As described above, in some instances, at least one prompt word may have specified a first syntax in which the base language model 122 will provide the response, and the response may be formatted using the first syntax.

[0107] According to some examples, the method includes receiving a response to at least one prompt word at box 828. For example, Figure 1 The illustrated application 120 can receive a response to at least one prompt word.

[0108] Figure 9 Example routines for invoking one or more tools by a language model, according to some embodiments of the present technology, are illustrated. Although the example routines depict a particular order of operations, this order may be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routines may perform functions substantially simultaneously or in a particular order.

[0109] According to some examples, the method includes receiving instructions at box 902 to enable the language model to invoke a first tool. For example, Figure 1 The illustrated base language model 122 may receive instructions that enable the language model to invoke a first tool. These instructions include a description of when the language model should invoke the first tool. In some embodiments, the instructions may also enable the language model to invoke additional tools.

[0110] In some implementations, the tools are supported by the device's operating system's software development kit (SDK). In other words, the SDK defines how the tools should be invoked. Additionally, these tools can be configured to utilize services provided by the operating system, as required by the SDK. As an example, if a tool wants to access user data, the tool requires permission from the operating system, which may require the user to explicitly grant permission for the tool to access such data.

[0111] Examples of instructions that enable the language model to invoke the first tool could be:

[0112] @Tool(description: "Use this tool to look up contact information")

[0113] struct ContactsTool {

[0114] func invoke(name: String) async throws -> ToolOutput {

[0115] }

[0116] }

[0117] This defines a SWIFT struct (ContactsTool) with a function invoke that takes a name as an argument. The @Tool attribute provides a description that indicates the tool is used to find contact information.

[0118] According to some examples, the method includes receiving a session initialization message at box 904. For example, Figure 1 The illustrated base language model 122 can receive a session initialization message. The session initialization message includes a toolset, which includes at least a first tool. The session initialization message enables the language model to invoke any tool in the toolset during a session. An example session initialization message could be:

[0119] let session = LanguageModelsSession(tools: [ContactsTool()])

[0120] let answer = try await session.respond(to: "What is mom's number")

[0121] This creates a session with a list of tools that includes ContactsTool (the only tool in the list in this example).

[0122] According to some examples, the method includes receiving the first cue word at box 906 during the first session. For example, Figure 1 The illustrated base language model 122 can receive the first cue word during the first session. An example of the first cue word could be:

[0123] What is my mother's phone number?

[0124] According to some examples, the method includes determining the invocation of the first tool at box 908 based on the first prompt word. For example, Figure 1 The illustrated base language model 122 can determine which tool to invoke based on the first prompt word. When the language model is configured to invoke multiple tools, the first tool is one of those tools. The language model determines which tool among the multiple tools to invoke to facilitate a response to the first prompt word. For example, the language model can output (in JSON):

[0125] Available tools:

[0126] {"name": "contacts_tool", "description": "Use this tool to look upcontact information", "parameters": {"name": "string"}}

[0127] """

[0128] """

[0129] <start_executable> ```function

[0130] [{"name": "contacts_tool", "arguments": {"name": "Mom"}}]<end_executable>

[0131] """

[0132] This JSON object describes contacts_tool, including its name, description, and expected parameters, and represents a JSON call to contacts_tool where the parameter name is set to "Mom".

[0133] According to some examples, the method includes calling the first tool at box 910. For example, Figure 1 The illustrated base language model 122 can invoke the first tool. Invocation of the first tool and responses from the first tool are subject to services provided by the device operating system, which is associated with the device on which the language model is being executed and exposes integration with the first tool.

[0134] In this example, the call to the first tool is made via an API provided by the device's operating system. Therefore, the API call needs to be converted from JSON to SWIFT, as the iOS operating system uses SWIFT for API calls. The iOS operating system has a tool called GENERABLE that can perform this conversion. Thus, the language model can interface with GENERABLE to perform the conversion. After the conversion, the call from the base language model 122 can be:

[0135] contactsTool.invoke(name: "Mom") --> Mom's contact information

[0136] This calls a function on contactsTool, with "Mom" as the parameter, designed to retrieve the mother's contact information.

[0137] As described in this article, the first tool does not require an operating system for availability. The session initialization message configures the base language model 122 to call APIs over the internet, which may already require JSON-based API calls. In this case, the base language model 122 can output and transmit API calls using JSON syntax.

[0138] According to some examples, the method includes receiving a response from the first tool at box 912. For example, Figure 1 The illustrated base language model 122 can receive a response from the first tool.

[0139] According to some examples, the method includes copying the response from the tool to a context window used for the first session at box 914. For example, Figure 1 The illustrated base language model 122 can copy the response from the tool into a context window used for the first session. The context window now includes the first prompt word and the response from the first tool.

[0140] According to some examples, the method includes determining at decision box 916 to invoke another tool. For example, Figure 1 The illustrated base language model 122 can determine to invoke another tool. Although the example above does not include another tool in the session initialization message, it is possible to include another tool in the session initialization message. The base language model 122 can be configured to invoke several tools. This configuration can occur through one or more messages.

[0141] Additionally, the language model can be given knowledge of all tools present on the computing device or available through the operating system. For example, tools can register with the operating system, which can be a source of knowledge for the language model to determine which tools are available for invocation and what syntax is required to invoke them. Through this mechanism, it may not be necessary to explicitly grant the language model access to the set of tools in the session initialization message; instead, the language model may naturally be able to discover tools through the operating system, enabling it to determine whether and when to invoke such tools.

[0142] When the underlying language model 122 is unsure whether to call another tool, the method continues to box 922.

[0143] Based on some examples, when the language model determines to invoke the second tool, the method includes invoking the second tool at box 918. The language model is trained to determine which tools to invoke and when to invoke them based on the configuration provided to it. The model does not need to be provided with a specific order in which the tools are invoked.

[0144] In some implementations, invoking a second tool may include transmitting information to the second tool based on a response from the first tool. It is possible that the base language model 122 determines that tools it has authorized access to should be serially invoked, such that the base language model can collect information from the response from the first tool and transmit at least some of that information to the second or third tool. Although Figure 9 The example illustrates how the base language model 122 serially calls the second tool, but after calling the first tool, the base language model 122 can decide to call multiple tools in parallel simultaneously.

[0145] According to some examples, the method includes receiving a response from a second tool at box 920. For example, Figure 1 The illustrated base language model 122 can receive responses from a second tool. When the base language model 122 receives a response from a second (or third, etc.) tool, the context window can be updated to include the first prompt word, the response from the first tool, and the response from the second tool.

[0146] According to some examples, the method includes using the response from the tool at box 922 to generate a response to the first prompt word. For example, Figure 1 The illustrated base language model 122 can use responses from the tool to generate a response to the first prompt word.

[0147] In addition to the constrained decoding implementation described above, this technology also benefits third-party application developers by supporting fine-tuning models and custom distillation models, which can be bundled with their applications to provide better support for those applications. More specifically, fine-tuning models can provide a degree of better or more reliable response to cue words in a specific domain associated with a given training set. Furthermore, distillation models can provide faster responses because they are typically smaller (fewer parameters) and more focused on performing well when responding to cue words from an application-specific context.

[0148] As this article will illustrate, supporting fine-tuning and distillation models within applications—especially those running on personal computing devices with on-device language models—is no easy task. Application developers need to be able to fine-tune the language model on the user's personal computing device, and the language model version may also be updated as the operating system is updated. This presents challenges for all application developers, app stores, and client devices. All three entities need to manage new assets—adapters for fine-tuning the base language model and distillation models. Some of these challenges will be discussed in detail below. Figure 10 and Figure 11 Let me elaborate.

[0149] Figure 10 Example routines for creating fine-tuning and distillation models and managing these assets by app stores and client devices are illustrated according to some embodiments of the present technology. Although the example routines depict a specific order of operations, this order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routine may perform functions substantially simultaneously or in a specific order.

[0150] Based on some examples, the method includes creating an adapter at box 1002 to the first version of the underlying language model. For example, this could be part of the process of developing an app to be offered for download through an app store. Figure 1 The illustrated developer 116 can create an adapter to the first version of the base language model. When the adapter is used with the first version of the base language model, the base language model is a fine-tuned version of the base language model used to provide responses to queries originating from the application.

[0151] An adapter is a custom weight layer that interfaces with the weights on the base language model. The combined weights of the adapter and the base language model produce weights different from those of the base language model alone, thus resulting in a fine-tuned base language model. A common type of adapter is the LoRA (Low-Rank Adaptive) adapter, which involves fine-tuning by adding a trainable rank decomposition matrix to certain layers of the base language model. This approach significantly reduces the number of parameters required for training, thereby reducing computational requirements and enabling large base language models to be adapted to specific tasks or domains with limited data or resources. In this case, the LoRA adapter does not change the weights of the base language model, which are fixed at training time, but rather adjusts the weights associated with the LoRA adapter to refine the performance of the combination of the base language model and the LoRA adapter for a specific purpose.

[0152] Since the base language model executes on the client device, this technology provides application developers with a fine-tuning toolkit (as part of an application developer toolkit that facilitates the development of applications that run in a specific environment (e.g., iOS)). This toolkit allows developers to download a version of the base language model to their client device and connect it to the training environment (an application on a powerful computing device or in the cloud) via a connection to the client device (which could be a smartphone, tablet, headset, consumer laptop, etc.). In this way, application developers can create their adapters and test the performance of their applications using the base language model running in its native environment.

[0153] According to some examples, the method includes distilling the fine-tuned base language model into a distilled model at box 1004. For example, Figure 1 The developer 116 illustrated can distill the fine-tuned base language model into a distilled model.

[0154] As described above, distilled models offer some of the benefits of fine-tuning a base language model, but are much smaller and more efficient, enabling faster response times and less power consumption during inference operations. A distilled model is a simplified version of a larger, more complex machine learning model created using a process called "model distillation." This process involves training a smaller distilled model (often referred to as the "student") to mimic the behavior of a larger, pre-trained, fine-tuned base language model (the "teacher"). The goal is to retain as much of the original model's accuracy and performance as possible while reducing its size and computational requirements. This is particularly useful for deploying models in resource-constrained environments, such as mobile devices or embedded systems.

[0155] In some implementations, similar to fine-tuning toolkits, tools can be provided to developers to assist in creating distillation models. In some examples, the tools may be able to automatically generate distillation models from a set of sample prompts that the application might pass to the distillation model.

[0156] According to some examples, the method includes submitting the app, along with the adapter and distillation model, to the app store at box 1006. For example, Figure 1 The illustrated developer 116 can submit their app to the app store along with an adapter and a distillation model. The adapter and distillation model are provided in a different container than the app. In some examples, the developer also includes a metadata file defining the compatibility of the adapter with the first version of the underlying language model. Note that the app does not need to include an adapter or distillation model to take advantage of the language model on the device. An adapter or distillation model is only needed if the app developer wants to benefit from fine-tuning the underlying language model. Even if the app developer wants to benefit from fine-tuning the underlying language model, the app developer does not need to include a distillation model; including only the adapter is sufficient, but the app developer will not gain any additional benefits from using a distillation model.

[0157] According to some examples, the method includes receiving an application 1008 with an adapter and a distillation model. For example, Figure 1 The example app store 118 can receive apps with adapters and distillation models.

[0158] Once an app is hosted in an app store, it becomes available for download by client devices. According to some examples, the method includes requesting the app download at box 1010. For example, Figure 1 The illustrated client device 102 can request to download the application.

[0159] According to some examples, the method includes, at box 1012, providing the application to the first device in response to a request to download the application. For example, Figure 1 The illustrated app store 118 can provide an app to a first device in response to a request to download the app. In some implementations, app store 118 only initially provides the app to client device 102 and does not immediately provide a container including adapters and / or distillation models, but may provide the container when the app is downloaded. Not providing the app download to the container can be beneficial by storing these components of the app in a separate container. More specifically, many apps will have functionality that does not require the use of the language model on the device, so downloading the container before the user attempts to use the language model on the device through the app may be wasteful. Additionally, the versions of the language model and adapter on the device may change over time, and actively downloading potentially unused versions of the adapter or distillation model may be wasteful.

[0160] According to some examples, the method includes downloading the application-associated adapter and / or distillation model at box 1014. For example, Figure 1 The illustrated client device 102 can download a container (adapter file 110) that includes an adapter associated with the application and / or a distillation model.

[0161] Before downloading the adapter, app store 118 can check which version of the underlying language model the client device 102 is running. Because the adapter is specific to a particular underlying language model, app store 118 can store multiple versions of the adapter for the app. The app store can use the metadata file provided with the adapter in the container to determine which underlying language model the adapter is suitable for.

[0162] According to some examples, the method includes, at box 1016, providing an adapter and a distilled model to the first device when the first device is running a first version of the base language model. For example, Figure 1 The illustrated app store 118 can provide the first device with an adapter and distillation model configured for the first version of the base language model when the first device is running the first version of the base language model.

[0163] Figure 11 Examples of some implementations of this technology are shown, which reside in application space 108 and are copied to system space 104 for use in the inference process, and include adapters and distillation models.

[0164] Once the adapter and / or distillation model have been obtained from, as per... Figure 10 The files downloaded from the app store reside in the operating system's application space 108. However, the device's language model (base language model 122) resides in the operating system's system space 104. Application space 108 is a sandboxed environment for applications and their files, while system space 104 is a privileged space for trusted system processes. Processes running in system space 104 cannot directly access files in application space 108, and vice versa. Therefore, to utilize adapters and / or distillation models, these files need to be copied to system space 104.

[0165] Figure 11 The adapter file 110 is shown after it has been downloaded from app store 118. Adapter file 110 includes metadata 1106, adapter 1102, and distillation model 1104. Metadata 1106 describes the adapter files and which version of the underlying language model they are compatible with. Adapter 1102 is a set of weights and, in some implementations, a LoRA adapter. Distillation model 1104 is typically two files. One small file constitutes the model, and the other file is the weights of the distillation model. As described above, the distillation model is not required (therefore, any further indexing of the distillation model assumes its existence).

[0166] A system process called the model manager (not shown) is responsible for copying the adapter 1102 to system storage 1108 and for copying the distillation model 1104 to compiler 1112, which compiles the distillation model 1104 before storing the compiled distillation model 1104 to system storage 1108. In some implementations, the model manager may copy these files on demand as the interference process requires them. More specifically, the distillation model 1104 and the adapter 1102 are retained in the application space without being copied to system storage 1108 until application 120 initializes the underlying language model 122. Figure 12 It begins at this step in the process, and therefore... Figure 11 Further description will be available in [the following text is incomplete and likely refers to a separate event or description]. Figure 12 It takes place within the context of [the event / event].

[0167] Figure 12 Example routines for using adapters and distillation models in the inference process, according to some embodiments of the present technology, are illustrated. Although the example routines depict a specific order of operations, this order may be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in different orders that do not substantially affect the functionality of the routine. In other examples, different components of the example device or system implementing the routine may perform functions substantially simultaneously or in a specific order.

[0168] According to some examples, the method includes receiving a prompt word from the application at box 1202. For example, Figure 1 The illustrated base language model 122 can receive prompt words from the application.

[0169] As described above, since the base language model 122 has received the prompt that the adapter 1102 and distillation model 1104 (if they exist) will be needed, the method includes copying the adapter 1102 and distillation model 1104 from an application-accessible storage location to a secure storage location 1108 that is accessible by system processes but not by the application at box 1204.

[0170] More specifically, adapter 1102 is directly copied into system memory 1108 and is immediately ready to be copied by the model manager into base model memory 1110 for use with the base language model in inference process 1114. However, when the inference process begins, distillation model 1104 is being distilled.

[0171] According to some examples, the method includes performing a first part of the inference process at box 1206 by fine-tuning the base language model based on a first cue word. For example, fine-tuning the base language model may perform the first part of the inference process based on a first cue word of the fine-tuned base language model. For clarity, fine-tuning the base language model is a combination of base language model 122 and adapter 1102.

[0172] According to some examples, the method includes compiling the distillation model on the computing device before using it at box 1208. This compilation is performed at least partially concurrently with the first part of the inference process. Once the distillation model is compiled, the compiled distillation model can be loaded into GPU memory and is ready for use in the inference process 1114.

[0173] According to some examples, the method includes using a compiled distillation model at box 1210 to perform the second part of the inference process. The compiled distillation model can output a draft response as the result of the second part of the inference process.

[0174] According to some examples, the method includes evaluating the draft response at decision box 1212 by fine-tuning the underlying language model. For example, Figure 1 The illustrated base language model 122 can be used to evaluate draft responses by fine-tuning the base language model. This technique, which enables the distillation model 1104 to perform inference operations when available, leverages the efficiency of the distillation model. The base language model can then be fine-tuned to examine the answers provided by the distillation model. Even though fine-tuning the base language model requires more computational resources to run, it runs much faster than generating lexical units from scratch when evaluating the output from the distillation model, thus preserving most of the efficiency gains of using the distillation model for the inference process.

[0175] The reason for using a fine-tuned base language model to check draft responses is that a distillation model may have already been created without extensive training. At least, using a fine-tuned base language model in this way can also provide confidence in the distillation model. After the distillation model has generated enough draft responses without requiring correction, the use of a fine-tuned base language model can be discontinued to check the output of the distillation model. This technique of checking the output of a distillation model using a fine-tuned base language model is called speculative decoding.

[0176] According to some examples, the method includes delivering the second response to the application at box 1214 when the fine-tuned base language model classifies the second response as acceptable.

[0177] According to some examples, the method includes revising the draft response into a revised response at box 1216 when the fine-tuning of the underlying language model does not consider the second response acceptable.

[0178] In some implementations, and where the user consents to the use of their data in this manner, prompts and responses from the distillation model, as well as the final responses delivered back to application 120 in response to the prompts, may be stored. This data can be used to further train the distillation model into a better model, and may be a distillation model specific to the type of prompts provided by application 120 for the user account.

[0179] Figure 13A , Figure 13B and Figure 13C Example transformer architectures according to some embodiments of the present technology are illustrated. Examples of ML models using transformer neural networks (e.g., transformer architecture 1300) may include, for example, generative pre-trained transformer (GPT) models and bidirectional encoder representation (BERT) models from transformers. Figure 13A , Figure 13B and Figure 13C The converter architecture 1300 illustrated includes an input 1302, an input embedding block 1304, a positioning encoder 1306, an encoder 1308 including an encoding block 1310, a decoder 1312 including a decoding block 1314, a linear block 1316, a softmax block 1318, and an output probability 1320.

[0180] Input embedding block 1304 is used to provide a representation for words. For example, the embedding can be used for text analysis. According to some non-limiting examples, the representation is a real-valued vector that encodes the meaning of words in a way that makes words expected to be closer in the vector space similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from a vocabulary are mapped to real-valued vectors. According to some non-limiting examples, input embedding block 1304 can be a learned embedding to transform input and output lexical units into dimensional vectors, for example, having the same dimension as the localization encoding.

[0181] The localization code 1306 provides information about the relative or absolute localization of terms in the sequence. According to some non-limiting examples, the localization code 1306 can be provided by adding the localization code to the input embedding at the input to the encoder 1308 and decoder 1312. The localization code has the same dimension as the embedding, thus enabling the summation of the embedding and the localization code. Several ways exist for implementing the localization code, including learning and fixing. For example, sine and cosine functions with different frequencies can be used. That is, each dimension of the localization code corresponds to a sine curve. Other techniques for conveying localization information can also be used, as will be understood by those skilled in the art. For example, learned localization embeddings can be used alternatively to obtain similar results. The advantage of using a sine localization code instead of a learned localization code is that doing so allows the model to extrapolate to sequence lengths longer than those encountered during training.

[0182] Encoder 1308 can use stacked self-attention and pointwise fully connected layers. Encoder 1308 can be a stack of N identical layers (e.g., N=6), and each layer can be a coding block, such as... Figure 13B The coding block 1310 is illustrated here. Each coding block 1310 has two sub-layers: (i) a first sub-layer has a multi-head attention block 1322, and (ii) a second sub-layer has a feedforward block 1326, which can be a position-by-position fully connected feedforward network. The feedforward block 1326 may use rectified linear units (ReLUs).

[0183] Encoder 1308 uses residual connections around each of the two sublayers, followed by summation and normalization blocks 1324 that perform normalization. For example, the output of each sublayer could be LayerNorm(x+Sublayer(x)). To facilitate these residual connections, all sublayers in the model, as well as the embedding layers, produce output data with the same dimension.

[0184] Similar to encoder 1308, decoder 1312 uses stacked self-attention and pointwise fully connected layers. Decoder 1312 can also be a stack of M identical layers (e.g., M=6), and each layer can be a decoding block, such as... Figure 13B The decoding block 1312 is illustrated in the diagram. In addition to the two sub-layers found in the encoding block 1310 (i.e., the sub-layer with multi-head attention block 1322 and the sub-layer with feedforward block 1326), the decoding block 1314 may also include a third sub-layer that performs multi-head attention on the output of the encoder stack. Similar to encoder 1308, decoder 1312 uses residual connections around each of these sub-layers before performing layer normalization. Additionally, the sub-layer with multi-head attention block 1322 can be modified in the decoder stack to prevent localization from focusing on subsequent localizations. This masking, combined with the fact that the output embedding is offset by a locality, ensures that the prediction for locality i can depend only on known output data at localities less than i.

[0185] Linear block 1316 can be a learned linear transformation. For example, when transformer architecture 1300 is used to translate from a first language to a second language, linear block 1316 can project the output from the last decoding softmax block 1318 onto word scores for the second language (e.g., scores for each unique word in the target vocabulary) at each location in the sentence. For example, if the output sentence has seven words and the provided second language vocabulary has 10,000 unique words, then 10,000 scores are generated for each of those seven words. The scores indicate the probability that each word in the vocabulary will appear at that location in the sentence.

[0186] Then, the Softmax block 1318 transforms the scores from the linear block 1316 into output probabilities 1320 (which total 1.0). At each location, the index provides the word with the highest probability, and this index is then mapped to the corresponding word in the vocabulary. These words then form the output sequence of the transformer architecture 1300. The softmax operation is applied to the output from the linear block 1316 to transform the original numbers into output probabilities 1320 (e.g., lexical probabilities).

[0187] Figure 14 This is a system diagram illustrating some embodiments of the device 1400 according to the present technology. Although this example system depicts specific system components and the arrangement of such components, this depiction is for the purpose of facilitating the discussion of the present technology and should not be considered limiting unless specified in the appended claims. For example, some components illustrated as separate may be combined with other components, some components may be divided into separate components, some components may be absent or unnecessary, and other components may be present.

[0188] Device 1400 can perform various operations, including image processing. For this and other purposes, device 1400 may include an image sensor 1401, a system-on-a-chip 1402, a system memory 1417, a permanent storage device 1416, a motion sensor 1419, and a display 1410, as well as other components.

[0189] Image sensor 1401 is a component for capturing image data and may be embodied, for example, as a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, camera, camcorder, or other device. Image sensor 1401 generates raw image data, which is transmitted to system-on-chip 1402 for further processing. In some embodiments, the image data processed by system-on-chip 1402 is displayed on display 1410, stored in system memory 1417, persistent storage device 1416, or transmitted to a remote computing device via a network connection. The raw image data generated by image sensor 1401 may be in Bayer color filter array (CFA) style (hereinafter also referred to as "Bayer style").

[0190] Flash controller 1405 is a component used to control variable characteristics of flash unit 1404. Some attributes of the adjustable flash unit 1404 profile include flash duration, flash intensity, flash spectrum, and angular profile. For example, some flash unit 1404 devices may include flash units with adjustable intensity, and some flash unit devices include multiple flash units that may have different emission spectra, which can be activated independently to control the angular profile or spectrum of light emitted from the flash units. Angular profile refers to the pattern and spread of light emitted from the flash unit over a given area, and how this spread changes at different angles relative to the flash unit. This can include how the intensity and distribution of light change as it moves laterally from a central axis located directly in front of the flash unit.

[0191] Motion sensor 1419 is a component or set of components used to sense the motion of device 1400. Motion sensor 1419 can generate sensor signals indicating the orientation and / or acceleration of device 1400. The sensor signals are transmitted to system-on-chip 1402 for various operations, such as rotating an image displayed on display 1410 and tracking the motion of image sensor 1401 during image capture.

[0192] Display 1410 is a component for displaying images generated by system-on-chip 1402. Display 1410 may include, for example, a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) device. Based on data received from system-on-chip 1402, display 1410 may display various images, such as menus, selected operating parameters, images captured by image sensor 1401 and processed by system-on-chip 1402, and / or other information received from the user interface (not shown) of device 1400.

[0193] System memory 1417 is a component used to store instructions executed by system-on-chip 1402 and to store data processed by system-on-chip 1402. System memory 1417 can be embodied in any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUSDRAM (RDRAM), static RAM (SRAM), or combinations thereof. In some embodiments, system memory 1417 may store pixel data or other image data or statistical values ​​in various formats. System memory 1417 can be accessed by many components of system-on-chip 1402, including but not limited to central processing unit 1406, graphics processing unit 1412, and neural engine 1420.

[0194] Persistent storage device 1416 is a component used to store data in a non-volatile manner. Even when power is unavailable, persistent storage device 1416 retains the data. Persistent storage device 1416 may be embodied as read-only memory (ROM), NAND or NOR flash memory, or other non-volatile random access memory devices.

[0195] The system-on-chip 1402 is embodied as one or more integrated circuit (IC) chips and performs various data processing procedures. The system-on-chip 1402 may include an image signal processor 1403, one or more central processing units 1406, a network interface 1407, a sensor interface 1408, a display controller 1409, one or more graphics processing units 1412, a memory controller 1413, a video encoder 1414, a storage controller 1415, one or more neural engines 1420, various other input / output (I / O) I / O interfaces 1411 and a bus 1418, and other components. Some components of the system-on-chip 1402 may be directly connected to the system memory 1417, while other components are connected to other components via the bus 1418. The system-on-chip 1402 may include... Figure 14 The components shown have more or fewer components.

[0196] The image signal processor 1403 (ISP) is hardware that performs various stages of the image processing pipeline. In some embodiments, the image signal processor 1403 may receive raw image data from the image sensor 1401 and process the raw image data into a form that can be used by other sub-components of the system-on-chip 1402 or components of the device 1400. The image signal processor 1403 may perform various image processing operations, such as image panning, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as described below. Figure 2 Detailed description.

[0197] The Central Processing Unit 1406 (CPU) can be embodied using any suitable instruction set architecture and can be configured to execute instructions defined in that instruction set architecture. The CPU 1406 can be a general-purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, RISC, ARM, or MIPS ISA, or any other suitable ISA. Although Figure 14 A single CPU is illustrated, but the system-on-a-chip 1402 may include multiple CPUs. In a multiprocessor system, each CPU may implement the same ISA, but this is not required.

[0198] The graphics processing unit 1412 (GPU) is a graphics processing circuit for executing graphics data. For example, the GPU can render objects to be displayed into a frame buffer (e.g., a frame buffer that includes pixel data for the entire frame). The graphics processing unit 1412 may include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of some graphics operations.

[0199] The Always-On Processor (AOP) 1422 acts as a component within the device that enables the execution of the real-time operating system, even when the Central Processing Unit 1406 is powered down or in sleep mode. As a distinct energy-efficient processor, the AOP operates autonomously relative to the Central Processing Unit 1406, ensuring that specific functionalities remain accessible even in a "power-down" state. To manage various sensors and chips within the device, including motion sensors, Bluetooth, and ultra-wideband chips, the AOP facilitates seamless connectivity and operation. The AOP has the ability to wake up the Central Processing Unit 1406 as needed, effectively conserving battery life. The AOP 1422 is capable of independently performing critical tasks, ensuring uninterrupted functionality.

[0200] The Neural Engine 1420 includes one or more processing cores optimized for machine learning tasks, including training and inference. The Neural Engine 1420 enables rapid processing of artificial intelligence (AI) and machine learning (ML) operations. Optimized for tasks such as advanced image processing, natural language processing, and pattern recognition, the Neural Engine 1420 significantly improves the efficiency and speed of AI-related processes. The architecture of the Neural Engine is designed to support a wide range of machine learning models while being highly energy efficient, thereby enhancing the user experience through faster, more responsive applications and functionality that rely on AI and ML technologies.

[0201] I / O interface 1411 is hardware, software, firmware, or a combination thereof used for interfacing with various input / output components in device 1400. I / O components may include devices such as keyboards, buttons, audio devices, and sensors such as GPS sensors. I / O interface 1411 processes data for transmitting data to or processing data received from such I / O components.

[0202] Network interface 1407 enables data to be exchanged between device 1400 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data can be received from other devices via network interface 1407 and stored in system memory 1417 for subsequent processing (e.g., via networks such as those described below). Figure 2 The image signal processor 1403 discussed herein (including its back-end interface) and display. The network may include, but is not limited to, a local area network (LAN) (e.g., Ethernet or a corporate network) and a wide area network (WAN). Image data received via the network interface 1407 may undergo image processing by the image signal processor 1403.

[0203] Sensor interface 1408 is a circuit used to interface with motion sensor 1419. Sensor interface 1408 receives sensor information from motion sensor 1419 and processes the sensor information to determine the orientation or movement of device 1400.

[0204] Display controller 1409 is a circuit used to transmit image data to be displayed on display 1410. Display controller 1409 receives image data from image signal processor 1403, central processing unit 1406, graphics processing unit 1412 or system memory 1417, and processes the image data into a format suitable for display on display 1410.

[0205] The memory controller 1413 is a circuit for communicating with the system memory 1417. The memory controller 1413 can read data from the system memory 1417 for processing by the image signal processor 1403, the central processing unit 1406, the graphics processing unit 1412, or other sub-components of the system-on-a-chip 1402. The memory controller 1413 can also write data received from various sub-components of the system-on-a-chip 1402 into the system memory 1417.

[0206] The video encoder 1414 is hardware, software, firmware, or a combination thereof for encoding video data into a format suitable for storage in permanent storage device 1416, or for transferring data to network interface 1407 for transmission over a network to another device.

[0207] In some implementations, one or more components of the system-on-a-chip 1402, or some of the functionality of these components, may be executed by software components running on the image signal processor 1403, the central processing unit 1406, or the graphics processing unit 1412. Such software components may be stored in system memory 1417, persistent storage device 1416, or in another device communicating with device 1400 via network interface 1407.

[0208] Image or video data can flow through various data paths within the system-on-chip 1402. In one example, raw image data may be generated by the image sensor 1401 and processed by the image signal processor 1403 before being transferred to the system memory 1417. After the image data is stored in the system memory 1417, it may be accessed by the graphics processing unit 1412, the neural engine 1420, and / or the video encoder 1414 for encoding or display 1410.

[0209] In another example, image data is received from a source other than image sensor 1401. For example, video data may be streamed, downloaded, or otherwise communicated to system-on-chip 1402 via a wired or wireless network. Image data may be received via network interface 1407 and written to system memory 1417 via memory controller 1413. The image data may then be retrieved from system memory 1417 and processed by image signal processor 1403, graphics processing unit 1412, or neural engine 1420. The image data may then be returned to system memory 1417.

[0210] Figure 15 An example of a computing system 1500 is shown, which may be, for example, composed of Figure 1 Any computing device, or any component thereof, of any engine, service, or component exemplified.

[0211] In some embodiments, the computing system 1500 is a single device or a distributed system in which the functions described herein can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform one or more of the functions described for use by the component. In one embodiment, these components can be physical devices or virtual devices.

[0212] In some implementations, the computing system 1500 may include one or more computing resources supplied from a "cloud computing" provider, such as AMAZON ELASTIC COMPUTE CLOUD ("AMAZON EC2") provided by AMAZON Inc. of Seattle, Washington; SUN CLOUD COMPPUTER UTILITY provided by SUN MICROSYSTEMS of Santa Clara, California; AZURE provided by MICROSOFT of Redmond, Washington; and GOOGLE CLOUD PLATFORM provided by ALPHABET of Mountain View, California.

[0213] Example computing system 1500 includes at least one processing unit (CPU or processor) 1504 and a connection 1502 that couples various system components, including system memory 1508, such as read-only memory (ROM) 1510 and random access memory (RAM) 1512, to processor 1504. Memory 1508 may be a volatile or non-volatile memory device and may be a hard disk or other type of non-transitory computer-readable medium that can store data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state memory devices, digital universal optical discs, cartridges, random access memory (RAM), read-only memory (ROM), and / or some combination of these devices.

[0214] The memory 1508 may include software services, servers, logic, etc., which enable the system to perform functions when the code defining such software is executed by the processor 1504. In some embodiments, hardware services that perform specific functions may include software components stored in a computer-readable medium and combined with necessary hardware components (such as the processor 1504, connection 1502, output device 1522, etc.) to perform the functions.

[0215] The computing system 1500 may include a cache of a high-speed memory 1506 that is directly connected to, close to or integrated into the processor 1504.

[0216] Connection 1502 can be a physical connection via a bus, or a direct connection to processor 1504, such as in a chipset architecture. Connection 1502 can also be a virtual connection, a networking connection, or a logical connection.

[0217] Processor 1504 may include any general-purpose processor and hardware or software services stored in memory 1508, which are configured to control processor 1504 and, in the case of software instructions being incorporated into the actual processor design, a dedicated processor. Processor 1504 can essentially be a completely standalone computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric. Processor 1504 can be physical or virtual.

[0218] To enable user interaction, the computing system 1500 includes an input device 1526, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1500 may also include an output device 1522, which can be one or more of a plurality of output mechanisms known to those skilled in the art. In some instances, a multi-mode system allows the user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1524, which typically controls and manages user input and system output. Operation is not limited to any particular hardware arrangement; therefore, the basic features described herein can be readily replaced with improved hardware or firmware arrangements after such arrangements are developed.

[0219] In some implementations, computing system 1500 may refer to a combination of a personal computing device and components hosted in a data center, wherein both the computing device and the components in the data center may have processors, caches, memory, storage devices, etc.

[0220] For clarity, in some instances, this technology may be presented as comprising individual functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software, or combinations of hardware and software.

[0221] Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services, or individually or in combination with other devices. In some embodiments, a service may be software residing in the memory of a client device and / or one or more servers of a content management system, and performing one or more functions while the processor executes the software associated with the service. In some embodiments, a service is a program or set of programs that performs a specific function. In some embodiments, a service may be considered a server. The memory may be a non-transitory computer-readable medium.

[0222] In some implementations, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams. However, by reference, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signal itself.

[0223] The methods described in the examples above can be implemented using computer-executable instructions stored in or otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device, or otherwise configure such devices to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), firmware, or source code. Examples of computer-readable media that can be used to store the instructions, the information used, and / or information generated during the methods according to the described examples include disks or optical discs, solid-state storage devices, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0224] Devices implementing the methods disclosed herein may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, etc. The functionality described herein may also be embodied in peripheral devices or expansion cards. As another example, this functionality may also be implemented in different processes executed between different chips on a circuit board or within a single device.

[0225] Instructions, media for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are components for providing the functionality described in these disclosures.

[0226] While various examples and other information are used to interpret aspects within the scope of the appended claims, the claims should not be construed as limiting based on specific features or arrangements in such examples, as those skilled in the art will be able to extend these examples to various specific embodiments. Although the subject matter has been described in language specific to structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or behaviors described. For example, such functionality may be distributed or performed differently in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

[0227] aspect:

[0228] This technology includes a computer-readable storage medium for storing instructions, and a system for performing any of the methods embodied in the instructions set forth in the various aspects of this technology presented below:

[0229] Aspect 1: A method comprising: receiving, by a language model, instructions for providing at least some responses using a first syntax acceptable to a type in a first programming language; receiving, by the language model, at least one prompt word from an application, wherein at least a portion of the response to the at least one prompt word will use the first syntax acceptable to the type in the first programming language; retrieving, by the language model, rules defining the first syntax for the type in the first programming language from a compiler of the first programming language; creating, by the language model, a grammar based on the rules from the compiler, the grammar defining lexical units that can be output by the language model; and returning, by the language model, a response to the at least one prompt word using the first syntax acceptable to the type in the first programming language.

[0230] Aspect 2: According to the method of Aspect 1, wherein the language model generates output using a second programming language when receiving the at least one prompt word and transmitting at least one response using the first programming language, the method comprising: wherein the rules defining a first grammar for the type in the first programming language further include rules for converting between the type in the first programming language and a second grammar in the second programming language, wherein the grammar is created according to the rules for converting between the type in the first programming language and the second grammar in the second programming language, and the grammar defines lexical units that can be output by the language model and are valid within the second grammar acceptable to the second programming language; generating an internal response to the at least one prompt word by the language model, wherein the language model is constrained by the grammar to output the lexical units valid within the second grammar; and converting the internal response to the response to the at least one prompt word using the rules for converting between the type in the first programming language and the second grammar in the second programming language.

[0231] Aspect 3: The method according to any one of Aspects 1 to 2, wherein the at least one prompt word does not mention the second programming language.

[0232] Aspect 4: The method according to any one of Aspects 1 to 3, wherein the language model has been trained to provide a response in which code using the second programming language should be used when the at least one prompt word does not explicitly request an alternative programming language by name.

[0233] Aspect 5: The method according to any one of Aspects 1 to 4, wherein the language model is executed on a personal computing device.

[0234] Aspect 6: The method according to any one of Aspects 1 to 5, the method further comprising: generating a response to the at least one prompt word by the language model, wherein the language model is constrained by the grammar to output the lexical units that are valid within the first grammar.

[0235] Aspect 7: The method according to any one of Aspects 1 to 6, wherein the grammar constrains the language model by masking effective lexical units from the candidate lexical unit distribution, thereby restricting the language model to select the effective lexical units from the candidate lexical unit distribution.

[0236] Aspect 8: The method according to any one of Aspects 1 to 7, further comprising: wherein the first grammar is implemented in a prefix tree, wherein the prefix tree includes branches representing valid character sequences; selecting a specific branch of the prefix tree corresponding to the specific existing character sequence based on a specific existing character sequence generated by the language model as part of the response, wherein the specific branch corresponds to a valid candidate lexicon; masking the valid candidate lexicon to restrict the language model from selecting from the valid candidate lexicon, thereby enabling the language model to select only the valid candidate lexicon as the next lexicon in the response.

[0237] Aspect 9: The method according to any one of Aspects 1 to 8, further comprising: when the grammar indication has a single valid candidate lexical, injecting the single valid candidate lexical into the response of the language model if the language model does not select the single valid candidate lexical as part of the inference process of the language model.

[0238] Aspect 10: The method according to any one of Aspects 1 to 9, wherein the grammar is a Backus-Noor Normal Form (BNF) grammar.

[0239] Aspect 11: The method according to any one of aspects 1 to 10, the method further comprising: wherein the instructions for providing at least some responses employing the first syntax further define a second syntax within the first syntax.

[0240] Aspect 12: The method according to any one of Aspects 1 to 11, wherein the first grammar is associated with the first grammar and the second grammar is associated with the second grammar, wherein the second grammar is constrained by the first grammar and the second grammar.

[0241] Aspect 13: The method according to any one of Aspects 1 to 12, the method further comprising: invoking the first grammar by the language model to generate a first part of a lexical; and, after generating the first part of the lexical, invoking the second grammar by the language model to generate a second part of the lexical.

[0242] Aspect 14: The method according to any one of Aspects 1 to 13, the method further comprising: after generating a first part of a lexical using the first grammar, determining whether to continue using the first grammar, use a second grammar, or continue generating lexicals that are not bound by the grammar.

[0243] Aspect 15: The method according to any one of aspects 1 to 14, wherein the at least one prompt word includes the instruction for providing the response using the first syntax acceptable to the type in the first programming language.

[0244] Aspect 16: The method according to any one of Aspects 1 to 15, wherein the instruction is provided before receiving the at least one prompt word, wherein the instruction effectively configures the language model to perform tool invocation, wherein the tool invocation employs the first syntax, and wherein the at least one prompt word causes the language model to determine to perform the tool invocation by generating the tool invocation employing the first syntax.

[0245] Aspect 17: The method according to any one of Aspects 1 to 16, the method further comprising: transmitting instructions from an application to a language model for providing at least some responses using a first syntax acceptable to a type in a first programming language; transmitting at least one prompt word from the application to the language model, wherein the response to the at least one prompt word will use the first syntax acceptable to the type in the first programming language, wherein the language model automatically creates a grammar according to rules collected from a compiler, the grammar defining lexical units that can be output by the language model; and receiving a response to the at least one prompt word using the first syntax acceptable to the type in the first programming language from the application.

[0246] Aspect 18: The method according to any one of Aspects 1 to 17, the method further comprising: receiving a first prompt word by a language model during a first session; determining, by the language model, to invoke a first tool based on the first prompt word; invoking the first tool by the language model; receiving a response from the first tool by the language model; copying the response from the first tool to a context window for the first session, wherein the context window includes the first prompt word and the response from the first tool; and generating a response to the first prompt word by the language model using the response from the first tool.

[0247] Aspect 19: The method according to any one of Aspects 1 to 18, the method further comprising: before receiving the first prompt word, receiving by the language model an instruction for enabling the language model to invoke the first tool, wherein the instruction includes a description of when the language model should invoke the first tool.

[0248] Aspect 20: The method according to any one of Aspects 1 to 19, wherein the first tool is supported by a software development kit of a device operating system, wherein the invocation of the first tool and the response from the first tool are subject to services provided by the device operating system, which is associated with the device executing the language model on it and exposes integration with the first tool.

[0249] Aspect 21: The method according to any one of Aspects 1 to 20, the method further comprising: receiving a session initialization message by the language model, wherein the session initialization message includes a toolset, the toolset including at least the first tool, the session initialization message enabling the language model to invoke any tool in the toolset during the session.

[0250] Aspect 22: The method according to any one of aspects 1 to 21, wherein the language model is configured to invoke a plurality of tools, wherein the first tool is one of the plurality of tools, the method comprising: determining by the language model which of the plurality of tools to invoke to facilitate a response to the first cue word.

[0251] Aspect 23: The method according to any one of Aspects 1 to 22, the method further comprising: after receiving a response from the first tool, the language model invoking a second tool as one of the plurality of tools, wherein invoking the second tool includes transmitting information based on the response from the first tool to the second tool; the language model receiving the response from the second tool; copying the response from the second tool to a context window for the first session, whereby the context window includes the first prompt word, the response from the first tool, and the response from the second tool; and the language model using the responses from the first tool and the second tool to generate a response to the first prompt word.

[0252] Aspect 24: The method according to any one of aspects 1 to 23, wherein the invocation of the first tool is performed via an API provided by the device operating system.

[0253] Aspect 25: The method according to any one of Aspects 1 to 24, wherein the invocation of the first tool is made via an API available on the Internet, wherein the API is invoked using structured JSON code.

[0254] Aspect 26: The method according to any one of Aspects 1 to 25, further comprising: as part of a process of developing an application to be provided for download through an app store, creating an adapter for a first version of a base language model, whereby, when the adapter is used with the first version of the base language model, the base language model is a fine-tuned base language model for providing a response to a query originating from the application; distilling the fine-tuned base language model into a distilled model; and submitting the application, along with the adapter and the distilled model, to the app store.

[0255] Aspect 27: The method according to any one of Aspects 1 to 26, the method further comprising: submitting a metadata file to the app store, the metadata file defining the compatibility of the adapter with the first version of the underlying language model.

[0256] Aspect 28: The method according to any one of Aspects 1 to 27, wherein the adapter and the distillation model are provided in a different container from the application, thereby allowing the application to be downloaded separately by the user, leaving the adapter and the distillation model for later retrieval.

[0257] Aspect 29: The method according to any one of Aspects 1 to 28, further comprising: receiving an application from an app store accompanied by an adapter and a distillation model, wherein when the adapter is used with a first version of the base language model, the base language model is a fine-tuned base language model for providing responses to queries originating from the application, wherein the distillation model is distilled from a dataset created based on the fine-tuned base language model; providing the application to a first device in response to a request to download the application; and in a subsequent transaction, providing the adapter and the distillation model to the first device while the first device is running the first version of the base language model.

[0258] Aspect 30: The method according to any one of Aspects 1 to 29, the method further comprising: downloading an adapter associated with an application by a computing device, wherein the adapter is specific to a first version of a base language model, wherein the adapter is a weight layer that, when combined with weights of the first version of the base language model, produces a fine-tuned base language model for processing cue words from the application; receiving cue words from the application; and processing the cue words in an inference operation using the fine-tuned base language model to produce a response to the cue words.

[0259] Aspect 31: The method according to any one of Aspects 1 to 30, the method further comprising: downloading a distillation model from the computing device, wherein the distillation model is distilled from a dataset created based on the fine-tuned base language model, wherein the distillation model is a smaller model than the fine-tuned base language model configured to generate a response comparable to the fine-tuned base language model using fewer computing resources; and using the distillation model to process a second cue word from the application to generate a second response.

[0260] Aspect 32: The method according to any one of aspects 1 to 31, the method further comprising: before using the adapter and the distillation model, copying the adapter and the distillation model from a storage location accessible by the application to a secure storage location accessible by a system process but not accessible by the application.

[0261] Aspect 33: The method according to any one of aspects 1 to 32, the method further comprising: compiling the distillation model on the computing device before using the distillation model.

[0262] Aspect 34: The method according to any one of aspects 1 to 33, the method further comprising: evaluating the second response by the fine-tuned base language model; and delivering the second response to the application when the fine-tuned base language model classifies the second response as acceptable.

[0263] Aspect 35: The method according to any one of Aspects 1 to 34, the method further comprising: evaluating the second response by the fine-tuned base language model; and when the fine-tuned base language model does not consider the second response acceptable, modifying the second response into a modified response by the fine-tuned base language model; and delivering the modified response to the application.

[0264] Aspect 36: The method according to any one of aspects 1 to 35, wherein the underlying language model is stored and executed on the computing device that serves as a user device.

[0265] Aspect 37: The method according to any one of Aspects 1 to 36, wherein the underlying language model and the application are stored and executed on the computing device.

[0266] Aspect 38: The method according to any one of Aspects 1 to 37, the method further comprising: downloading a distillation model from the computing device, wherein the distillation model is distilled from a dataset created based on the fine-tuned base language model; copying the adapter and the distillation model from a storage location accessible by the application to a secure storage location accessible by a system process but inaccessible by the application before using the adapter and the distillation model; performing a first portion of an inference process based on a first prompt word by the fine-tuned base language model; compiling the distillation model on the computing device to produce a compiled distillation model before using the distillation model, at least partially concurrent with the first portion of the inference process; performing a second portion of the inference process using the compiled distillation model, wherein a draft response is the result of the first portion and the second portion of the inference process; evaluating the draft response by the fine-tuned base language model; and transmitting the draft response to the application after the evaluation performed by the fine-tuned base language model.

[0267] Aspect 39: The method according to any one of Aspects 1 to 38, the method further comprising: processing a set of prompt words by the fine-tuned base language model to generate a response, wherein the set of prompt words is derived from a specific application operated by a specific user account; storing the set of prompt words and the corresponding responses; automatically creating a distillation model when the set of prompt words and the corresponding responses exceed a threshold size, wherein the distillation model is a smaller model than the fine-tuned base language model configured to generate a response comparable to the fine-tuned base language model using fewer computational resources; receiving a second prompt word; and using the distillation model to generate a second response to the second prompt word.

[0268] Aspect 40: The method according to any one of aspects 1 to 39, the method further comprising: determining that the base language model has been updated to a second version of the base language model; and downloading a second adapter specific to the second version of the base language model.

[0269] Aspect 41: A non-transitory computer-readable medium comprising instructions that, when executed, effectively cause at least one processor to perform the method according to any one of aspects 1 to 39.

[0270] Aspect 42: A system comprising a non-transitory computer-readable medium according to aspect 41 and at least one processor.< / digits> < / characters> < / person>

Claims

1. A method, the method comprising: The language model receives the first cue word during the first session; The language model invokes the first tool based on the first prompt word; The language model receives the response from the first tool; The response from the first tool is copied to a context window used for the first session, whereby the context window includes the first prompt word and the response from the first tool; The language model uses the response from the first tool to generate a response to the first prompt word.

2. The method according to claim 1, further comprising: Before receiving the first prompt word, the language model receives instructions that enable the language model to invoke the first tool, wherein the instructions include a description of the circumstances under which the language model should invoke the first tool.

3. The method according to any one of claims 1 to 2, wherein the first tool is supported by a software development kit of a device operating system, wherein invoking the first tool and the response from the first tool are subject to services provided by the device operating system, which is associated with the device executing the language model on it and exposes integration with the first tool.

4. The method according to any one of claims 1 to 2, further comprising: The language model receives a session initialization message, wherein the session initialization message includes a toolset, the toolset including at least the first tool, and wherein the session initialization message enables the language model to invoke any tool in the toolset during the first session.

5. The method of claim 4, wherein the language model is configured to invoke a plurality of tools, wherein the first tool is one of the plurality of tools, and wherein the method further comprises: The language model determines which of the plurality of tools to invoke to facilitate a response to the first prompt word.

6. The method according to claim 5, further comprising: After receiving the response from the first tool, the language model invokes a second tool, which is one of the plurality of tools, wherein invoking the second tool includes transmitting information based on the response from the first tool to the second tool; The language model receives the response from the second tool; The response from the second tool is copied to the context window used for the first session, whereby the context window includes the first prompt, the response from the first tool, and the response from the second tool; The language model uses the responses from the first tool and the second tool to generate a second response to the first prompt word.

7. The method according to any one of claims 1 to 2, wherein the invocation of the first tool is performed via an API provided by the device operating system.

8. The method according to any one of claims 1 to 2, wherein the invocation of the first tool is performed via an API available on the Internet, wherein the API is invoked using structured JSON code.

9. A computing system, the computing system comprising: At least one processor; and A memory that stores instructions that, when executed by the at least one processor, cause the computing system to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium comprising instructions stored thereon, wherein the instructions, when executed by at least one processor of a computing system, cause the computing system to perform the method according to any one of claims 1 to 8.

11. A computing system, the computing system comprising: Components for performing the method according to any one of claims 1 to 8.

12. A computer program product comprising one or more programs configured to be executed by at least one processor of a computing system, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 8.