Virtual batch in large language model inference

Abstract

This document describes systems and techniques for virtual batches in large language model (LLM) inference. An LLM deployed at least in part on an electronic device generates a dependency graph for a plurality of inferred lems. The inference lemma may be based on the same input, different inputs, or a mixture of both. The dependency graph indicates sequential dependencies or otherwise logical dependencies for each inferred lexical element. The LLM may further generate a plurality of virtual batches based on the dependency graph. The plurality of virtual batches includes a masked portion that indicates a location in one or more virtual batches of the plurality of virtual batches that is not referenced by the active lemma.

Description

Background Technology

[0001] Large Language Models (LLMs) provide predictive outputs (inferences) based on parsing inputs such as user text, audio, video, and image inputs. Inferences are generated sequentially, meaning each predicted segment (“lexical”) starts from the previous lexical. This does not allow for inference optimization, which has led to the use of batching in inference. Batches are concurrently generated inferences that can be based on common inputs, dissimilar inputs, or a combination of both. Each batch represents an independent inference from the LLM. Multiple inference batches are computationally expensive, making batch inference LLMs cumbersome or impossible to deploy on consumer computing devices. Summary of the Invention

[0002] This document describes a system and technique for virtual batches in large language model (LLM) inference. An LLM, at least partially deployed on an electronic device, generates a dependency graph for multiple inferred lexics. Inferred lexics can be based on the same input, different inputs, or a mixture of both. The dependency graph indicates the sequential dependencies or other logical dependencies of each inferred lexiconic word. The LLM can further generate multiple virtual batches based on the dependency graph. These multiple virtual batches may include masked portions indicating positions in one or more of these virtual batches that do not have active lexiconic word references.

[0003] In some aspects, an electronic device is disclosed, comprising one or more processors and a memory storing instructions. When accessed by the one or more processors, these instructions cause the one or more processors to use an LLM to generate a dependency graph for a plurality of lexical units. The dependency graph includes one or more index tags for each of the plurality of lexical units and a relevance tag for each of the one or more index tags. The instructions further cause the one or more processors to generate a plurality of virtual batches based on the dependency graph, each of the plurality of virtual batches comprising an independent inference; and to select one or more of the plurality of virtual batches as the final inference.

[0004] In some aspects, a method is disclosed that includes generating a plurality of lexical units using an LLM. The method further includes generating a dependency graph from the LLM, the dependency graph including one or more index tags for each of the plurality of lexical units and a relevance tag for each of the one or more index tags. The method further includes generating a plurality of virtual batches based on the dependency graph, each of the plurality of virtual batches including an independent inference. The method further includes selecting one or more virtual batches from the plurality of virtual batches as the final inference by the LLM.

[0005] In some aspects, a non-transitory computer-readable medium is disclosed, comprising instructions that, when accessed by one or more processors, cause the one or more processors to generate a plurality of lexical units using an LLM. The method further includes generating a dependency graph from the LLM. The dependency graph includes one or more index tags for each of the plurality of lexical units and a relevance tag for each of the one or more index tags. The instructions further cause the one or more processors to generate a plurality of virtual batches based on the dependency graph, each of the plurality of virtual batches including an independent inference; and to select one or more of the plurality of virtual batches as the final inference.

[0006] In some aspects, a computer programming product is disclosed, comprising a memory storing instructions that, when accessed by one or more processors, cause the one or more processors to generate a plurality of lexical units using an LLM. These instructions further cause the one or more processors to generate a dependency graph. The dependency graph includes one or more index tags for each of the plurality of lexical units and relevance tags for each of the one or more index tags. These instructions further cause the one or more processors to generate a plurality of virtual batches based on the dependency graph, each of the plurality of virtual batches comprising an independent inference; and to select one or more of the plurality of virtual batches as the final inference.

[0007] This invention provides a simplified concept of virtual batches in large language model inference, which is further described below in the detailed description and illustrated in the accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0008] Referring to the following figures, details of one or more aspects of the system and techniques for virtual batching in large language model inference are described in this document: Figure 1 An example environment is shown in which techniques for virtual batching in large language model inference can be implemented; Figure 2 This illustrates virtual batching for implementing large language model inference. Figure 1 Examples of electronic devices; Figures 3A to 3C An array of dependency graphs is shown for implementing virtual batches in large language model inference; Figures 4A to 4D An example of virtual batch generation with a mask is shown; Figure 5 Example trainers for LLM are shown, such as those used in virtual batches in large language model inference. Figure 6 An example transformation in the language space of the input tensor components is shown; Figure 7 An example method for implementing virtual batches in large language model inference is shown.

[0009] Using the same number in different instances can indicate similar features or components. Detailed Implementation

[0010] Overview

[0011] The proliferation of artificial intelligence (AI), particularly large language models (LLMs), has transformed personal digital assistants, automation, novel code generation, and other areas of modern computing. The development of LLM use cases has led to a demand for better prediction and inference from LLMs. Traditionally, LLMs parse input cues and produce outputs (predictions, inferences, etc.). One way to improve the output of an LLM is to generate multiple inferences / outputs from one or more inputs (one or more text inputs, one or more image inputs, a mixture of text and image inputs, etc.). Each of these inferences can be independent, allowing the LLM to select from multiple inferences as the final inference / output. In some cases, each of the multiple inferences includes the one or more inputs. However, this approach results in a significantly higher computational cost than simply generating a single inference / output. This increased computational cost (processor time / cycles, memory usage, cache usage, etc.) can prove prohibitively expensive on many electronic devices (mobile electronics, smartphones, virtual reality (VR) or augmented reality (AR) goggles, smartwatches, etc.).

[0012] As the utility and functionality of LLMs increase, cramming more robust implementations (e.g., LLMs that employ multiple batches for inference generation) into more resource-intensive computing devices prevents most users from fully utilizing the increased capabilities of modern LLMs. For example, an LLM that includes generating multiple inference batches can only be deployed on a server due to the computational cost of generating and maintaining those batches, and may not be fully accessible to users with smartphones not connected to that server. In another example where an LLM that includes generating multiple inference batches can only be deployed on a server, a user with a smartphone can connect to the server, but the increased processing time / computational cost compared to traditional LLMs can lead to a poor user experience, decreased functionality, or other undesirable results. In some cases, the act of generating and / or resolving multiple inference batches becomes computationally prohibitive until sufficient advantages are achieved compared to traditional LLM processing.

[0013] This document describes techniques and systems for virtual batches in large language model inference. These techniques and systems use the generated dependency graph to order and associate multiple lexical units. Multiple virtual batches are generated based on the dependency graph. In some respects, each of these virtual batches represents an independent inference. The LLM selects one or more virtual batches from the multiple virtual batches as the final prediction. In some respects, this final prediction is configured for the output.

[0014] As mentioned in this disclosure, lexical terms refer to inferred lexical terms generated by LLM. In some examples, lexical terms are in textual form (e.g., words, parts of grammar, word fragments). In some examples, lexical terms are mathematical constructs in a language space (e.g., tensors). In some aspects, lexical terms may be referred to as embeddings. In some examples, the input is not text input (audio input, image input, etc.). In such examples, lexical terms may be soft lexical terms. In some examples, soft lexical terms are embedded but at least partially dynamic lexical terms. In some examples, soft lexical terms are lexical terms with additional labels, tags, information, etc.

[0015] In some respects, generating a dependency graph uses fewer computational resources than generating separate physical batches. For example, multiple physical batches each occupy a finite amount of space on memory such as cache memory. In some examples, a virtual batch comprises a single physical batch. A single physical batch uses less cache space than splitting it into several physical batches. In some examples, the dependency graph comprises more than one physical batch, but at least one of these physical batches comprises more than one virtual batch, and this still represents a resource saving.

[0016] In some examples, virtual batches are not generated through construction, but rather logically extrapolated from the dependency graph. In this way, the dependency graph can still leverage the advantages of multiple inferences without simultaneously incurring the resource / computational costs of using multiple inferences. The advantages of using virtual batches in LLM inference include lower computational costs, lower memory usage, and the ability to deploy multi-inference models on less resource-intensive devices such as smartphones.

[0017] Operating environment

[0018] The following discussion describes an operating environment, the technologies that can be employed within that operating environment, and various apparatuses or systems in which the operating environment can be embodied. In the context of this disclosure, reference to the operating environment is made by way of example only.

[0019] Figure 1 An example environment 100 is shown, in which techniques for implementing virtual batches in large-scale language model inference can be implemented. Typically, environment 100 includes an electronic device 102. The electronic device 102 in the illustrated example is a smartphone, but it should be noted that other electronic devices can be used equivalently. Electronic device 102 includes an instantiated LLM (not shown). Inputs can be given to the LLM, such as input prompts 104. Input prompts 104 in... Figure 1 The input prompt 104 is shown as a user input suggestion, but according to some examples, it can be the product of a machine or machine algorithm. Input prompt 104 is shown as text input, but other input types (audio, video, images, etc.) can also be used equivalently. In some respects, LLM can provide a response 106.

[0020] In some examples, electronic device 102 may be an assistant device (e.g., Google® Nest® Hub; Google® Nest® Hub Max), a home automation controller (e.g., a controller for alarm systems, thermostats, lighting systems, door locks, power doors, etc.), a gaming device (e.g., a gaming system, game controller, data glove, etc.), a communication device (e.g., a smartphone such as a Google® Pixel® phone, such as a cellular phone, mobile phone, cordless phone, portable phone, wireless phone, etc.), a wearable device (e.g., a smartwatch, smart glasses, earphones, smart helmet, VR headset, AR goggles, smart ring, etc.), a vehicle (car, electric scooter, automated vehicle, etc.), and / or another computing device (e.g., a tablet computer, phablet computer, laptop computer, etc.). As another example, electronic device 102 with an assistant application or program (e.g., an AI assistant) may audibly convey information to the user. In some implementations, the battery management system audibly conveys notification information to the user and lists actions the user can take, such as ordering a new battery or obtaining disposal information. In some implementations, electronic device 102 listens to responses from a user, such as the user's selection of one or more of a listed action, and responds accordingly (e.g., obtaining the disposal option and audibly communicating the disposal option to the user).

[0021] In some examples, response 106 is based on data stored in the memory of electronic device 102. According to some examples, response 106 is based on one or more capabilities of electronic device 102. In some examples, response 106 is based on data stored remotely from electronic device 102 (via a remote server connected via a wireless communication link, the Internet, etc.). In some examples, response 106 is generated using only the resources of electronic device 102 (one or more processors of electronic device 102, the memory of electronic device 102, etc.), the resources of a remote device, or both.

[0022] In some aspects, response 106 is based on one or more inferences among a plurality of inferences generated by the LLM based on input prompt 104. In some aspects, the plurality of inferences may be a plurality of virtual batches. As outlined in this disclosure, a plurality of virtual batches may be generated from a dependency graph, including generating virtual batches via logical extrapolation without generating physical batches. The dependency graph includes one or more index tags for each of the plurality of lexics and relevance tags for each of the one or more index tags. In some aspects, the plurality of lexics is part of a plurality of generated inferences. Response 106 is based on one of the plurality of virtual batches, i.e., one of the plurality of virtual batches selected by the LLM. In some examples, the LLM compares the plurality of virtual batches and makes a selection based on the comparison. In some examples, response 106 is based on more than one selection from the plurality of virtual batches. In some examples, operations are performed on the selected virtual batches, transforming them into the final form of response 106.

[0023] Example device

[0024] Figure 2 This illustrates virtual batching for implementing large language model inference. Figure 1 Examples of electronic devices 102 include smartphones 102-1, tablets 102-2, desktop computers 102-3, laptops 102-4, servers 102-5 (including server arrays), smart monitors or TVs 102-6, smartwatches 102-7, earphones (e.g., true wireless earphones) 102-8, VR goggles 102-9, AR headsets 102-10, smart glasses 102-11, smart helmets 102-12, smart vehicles 102-13, home hub devices 102-14, and headsets 102-15. Although not shown, electronic devices 102 can also be implemented as mobile communication devices, client devices, home automation and control systems, entertainment systems, personal media devices, health monitoring devices, drones, cameras, internet-connected home appliances supporting wireless internet access and browsing, IoT devices, security systems, etc. It should be noted that the electronic device 102 can be wearable, non-wearable but mobile, or relatively stationary (e.g., electrical appliances). The electronic device 102 may include components designed for clarity or visual simplicity. Figure 2 Components or interfaces omitted in the text.

[0025] As shown in the figure, electronic device 102 includes one or more processors 202 and memory 204 (e.g., computer-readable medium). The one or more processors 202 may include any suitable single-core or multi-core processor (application processor (AP), digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU), etc.). The one or more processors 202 may be configured to execute instructions or commands stored in memory 204. Memory 204 may be stored in one or more non-transitory storage devices (e.g., random access memory (RAM, dynamic RAM (DRAM), non-volatile RAM (NVRAM), or static RAM (SRAM) etc.), read-only memory (ROM), flash memory, hard drive, solid-state drive (SSD), or any type of medium suitable for storing electronic instructions), each of which is coupled to a computer system bus. The term "coupled" can refer to two or more elements in direct contact (physical, electrical, magnetic, optical, etc.), or to two or more elements that are not in direct contact but still cooperate and / or interact with each other.

[0026] In some examples, memory 204 includes instructions 206. Instructions 206 may be in the form of executable code, one or more applications, software, etc. In some examples, memory 204 further includes cache 208. According to some examples, cache 208 is a virtual memory partition of memory 204. In some examples, cache 208 is a physical partition of memory 204.

[0027] In some examples, electronic device 102 includes one or more modules 210. In some aspects, module 210 may be based on one or more capabilities of electronic device 102. Examples of modules 210 include one or more sensor modules 212, one or more input modules 214, one or more communication modules 216, and one or more other modules 218. The one or more sensor modules 212 may include input sensors, capacitive sensors, infrared sensors, or optical sensors. Data based on any one of the one or more sensor modules 212 can be used to parse input (e.g., Figure 1 The prompt input 104) is used as the output for the generated product (e.g., Figure 1 The response 106) can be used as a basis, as an action against the output, as input or part of the input, or as any other aspect of a virtual batch in large language model inference. Similarly, data based on any one of the input modules 214 can be used to parse the input (e.g., Figure 1 The prompt input 104) is used as the output for the generated product (e.g., Figure 1The response (106) is based on, as an action against the output, as input or part of the input, or as any other aspect of virtual batches in large language model inference.

[0028] One or more communication modules 216 may include wired or wireless connection interfaces, radios, connection protocols, etc. One or more other modules 218 may include other aspects of the electronic device 102 (e.g., screens, microphones, or other capabilities of the electronic device 102) not shown for clarity. One or more communication modules 216, one or more other modules 218, or both may also be used to parse input (e.g., Figure 1 The input prompt 104) is used as the generated output (e.g., Figure 1 The response 106) serves as the basis for actions against output, as the basis for input or a portion of input, or as any other aspect of virtual batches in large language model inference. One or more communication modules 216 enable the transmission of device data (e.g., received data, transmitted data, or other information as described herein) and can provide connectivity to one or more networks and other devices connected thereto. Examples of one or more communication modules 216 include near field communication (NFC) transceivers, wireless personal area network (WPAN) radios compliant with various IEEE 902.15 (Bluetooth®) standards, wireless local area network (WLAN) radios compliant with any of the various IEEE 902.11 (WiFi®) standards, wireless wide area network (WWAN) radios for cellular phones (compliant with 3GPP), wireless metropolitan area network (WMAN) radios compliant with various IEEE 902.16 (WiMAX®) standards, infrared (IR) transceivers compliant with the Infrared Data Association (IrDA) protocol, and wired local area network (LAN) Ethernet transceivers. Device data transmitted via one or more communication modules 216 may be packetized or framed depending on the communication protocol or standard followed by which the electronic device 102 is communicating. One or more communication modules 216 may include interfaces for communicating via a local network, private network, intranet, Internet, or wireless network (e.g., WLAN, cellular network, or WPAN).

[0029] Electronic device 102 may further include and / or be operatively coupled to LLM 222. For example, LLM 222 may be stored on memory 204 (e.g., as part of instruction 206). In another example, LLM 222 is stored remotely from electronic device 102 and accessed via one or more communication modules 216. LLM 222 includes one or more of the following: parameters 224, language space 226, machine learning (ML) model 228, fine-tuning (FT) 230, one or more action modules 232, and one or more interface modules 234.

[0030] In some respects, parameter 224 controls the behavior of LLM 222. For example, LLM 222 can accept prompts as input (e.g., Figure 1 (Input hint 104). In some aspects, LLM 222 may use parameters 224 and language space 226 to parse the input. In some examples, LLM 222 also uses ML model 228 to further parse the input and / or generate output. In some examples, LLM 222 uses FT 230. For example, FT 230 can be used to grant a general LLM subject-specific expertise. As an example, FT 230 may include user-specific data. In some examples, FT 230 includes one or more low-rank adaptive (LoRA), retrieval-enhanced generation (RAG), or other techniques known in the art for fine-tuning or otherwise modifying an existing LLM (e.g., LLM 222).

[0031] One or more action modules 232 may allow the LLM 222 to perform actions. For example, if the parsing of input indicates an expectation of performing an action, the LLM 222 may use one or more action modules 232 to perform the expected action. For example, consider the user input prompt “cancel my meeting tomorrow”. The LLM 222 may parse the user input prompt (using parameters 224, language space 226, etc.) and determine that user calendar data is needed to fully parse the user input prompt and generate output. One or more action modules 232 may include calendar functionality from the electronic device 102 and may provide user calendar data to the LLM 222. The LLM 222 may further parse the user calendar data and determine that the specific meeting is the “my meeting” mentioned by the user. One or more action modules 232 may cancel the specific meeting and provide a cancellation confirmation to the LLM 222. The LLM 222 may use the cancellation confirmation as at least a partial basis for the output. For example, the output could be in the form of "I have canceled your meeting with the builder tomorrow. Is there anything else you need?"

[0032] Other actions may be accessible to one or more action modules 232, including vehicle control, information retrieval, communication, capabilities of electronic device 102 (e.g., module 210), and similar actions. In some examples, one or more action modules 232 utilize application programming interfaces (APIs) from one or more applications available to electronic device 102 or another device. In some examples, LLM 222 may create new action modules based on known or implicit capabilities. It should be understood that the one or more action modules 232 (including associated actions) listed herein are intended as examples and should not be considered limiting. Other actions and / or action modules 232 not listed may be similarly employed by LLM 222 using the methods outlined in this disclosure.

[0033] In some respects, one or more interface modules 234 provide interfacing between the LLM 222 and other devices, robots, etc. For example, the LLM 222 can use one or more interface modules 234 to connect to different LLMs deployed on a remote device. For example, different LLMs can access restricted data that the LLM 222 itself cannot access. In another example, one or more interface modules 234 can import sensor data from one or more sensor modules 212 of the electronic device 102. For example, the LLM 222 can use one or more interface modules 234 to obtain user facial expressions as at least a portion of the input, the user facial expressions being based on camera data from cameras of one or more input modules 214.

[0034] Example dependency graph

[0035] Figure 3A An example dependency graph array 300A is shown for implementing virtual batches in large language model inference. The dependency graph array 300A has indices 0 to 15, each index having an associated relevance value. In some respects, the index and the relevance value can be represented by ordinal pairs. For example, index 0 has a relevance value of -1, which can be represented by the ordinal pair (0, -1).

[0036] In some respects, the dependency graph array 300A mapping is derived by LLM (e.g., Figure 2 The dependency relationships of multiple terms inferred from the LLM (222). The dependency graph array 300A can be used to generate output from the LLM (e.g., Figure 1 (Response 106). Each index in the index represents the position of a term in the multiple terms on the dependency graph array 300A, and each relevance value in the relevance values ​​represents the index of the previous term of the term in the multiple terms.

[0037] As previously outlined, index 0 of dependency array 300A includes the ordinal pair (0, -1). The -1 relevance value indicates that the term indicated by index 0 has no predecessor and is the start of the first virtual batch. Index 1 includes the ordinal pair (1, 0), indicating that the term indicated by index 1 is concerned with the term indicated by index 0. The first association arrow 302A shows the mapping from index 1 (ordinal pair (1, 0)) to index 0. Index 2 includes the ordinal pair (2, 1), indicating that the term indicated by index 2 is concerned with the term indicated by index 1. The second association arrow 304A shows the mapping from index 2 to index 1. Index 3 includes the ordinal pair (3, 2), indicating that the term indicated by index 3 is concerned with the term indicated by index 2. The third association arrow 306A shows the mapping from index 3 to index 2. Index 4 includes the ordinal pair (4, 3), indicating that the term indicated by index 4 is concerned with the term indicated by index 3. The fourth associative arrow 308A shows the mapping from index 4 to index 3.

[0038] It should be noted that although the lexical unit indicated by index 1 is considered to be concerned with the lexical unit indicated by index 0, this does not mean that the lexical unit indicated by index 1 is specifically concerned with the lexical unit indicated by index 0. As used in this disclosure, the concern of a first lexical unit with a second lexical unit may include the first lexical unit concerning itself and / or concerning any lexical unit concerned by the second lexical unit. For example, if the second lexical unit concerns a third lexical unit, which in turn concerns a fourth lexical unit, then the first lexical unit may concern one or more of the first, second, third, and fourth lexical units. Associated arrows (e.g., first associated arrow 302A) indicate the concern of one lexical unit with another lexical unit (e.g., first associated arrow 302A shows the lexical unit indicated by index 1 concerning the lexical unit indicated by index 0).

[0039] Index 6 has an ordinal pair (6, 4) indicating that the lexicon associated with index 6 focuses on the lexicon associated with index 4, as shown by the fifth association arrow 310A. Similarly, index 11 has an ordinal pair (11, 6) indicating that the lexicon associated with index 11 focuses on the lexicon associated with index 6, as shown by the sixth association arrow 312A. Similarly, index 14 has an ordinal pair (14, 11) indicating that the lexicon associated with index 14 focuses on the lexicon associated with index 11, as shown by the seventh association arrow 314A. In this example, the first ordered set of focused lexicons is {0, 1, 2, 3, 4, 6, 11, 14}. In some examples, the first ordered set is the first terminal lexicon chain (e.g., a complete first virtual batch). In other examples, the first ordered set is part of an incomplete first virtual batch and contains future lexicons that will focus on the lexicon associated with index 14.

[0040] Figure 3B This illustrates the virtual batch mapping using dependency array 300B. Dependency array 300B and... Figure 3A The dependency array 300A is the same, but in Figure 3B The text shows the relationship with... Figure 3A The virtual batch paths are different from the virtual batch paths. As in... Figure 3A In the example, index 0 includes the ordinal pair (0, -1). Index 1 includes the ordinal pair (1, 0), indicating that the lexical indicated by index 1 is related to the lexical indicated by index 0. The first association arrow 302B shows the mapping from index 1 (ordinal pair (1, 0)) to index 0. The second association arrow 304B shows the mapping from index 2 to index 1, and the third association arrow 306B shows the mapping from index 3 to index 2.

[0041] Similar to index 4, index 5 has a relevance value of 3, thus giving it the ordinal pair (5, 3). The fourth association arrow 308B illustrates the mapping from index 5 to index 3. Index 7 has the ordinal pair (7, 5), which indicates that the lexicon associated with index 7 focuses on the lexicon associated with index 5, as shown by the fifth association arrow 310B. Similarly, index 12 has the ordinal pair (12, 7), which indicates that the lexicon associated with index 12 focuses on the lexicon associated with index 7, as shown by the sixth association arrow 312B. In this example, the second ordered set of focused lexicons is {0, 1, 2, 3, 5, 7, 12}. In some examples, the second ordered set is a second terminal lexicon chain (e.g., a complete second virtual batch). In other examples, the second ordered set is part of an incomplete second virtual batch and contains future lexicons that will focus on the lexicon associated with index 12.

[0042] For example, consider Figure 3A The first ordered set and the second ordered set. Note that both have the same prefix, {0, 1, 2, 3}. As these examples show, two or more virtual batches can share the same prefix. In other examples, two or more virtual batches can share the same part (in Figure 3A and Figure 3B (As shown as a prefix in the examples), such as the middle part, the terminal part, or any combination of parts. Use a dependency graph (dependency array 300B, Figure 3A The dependency array 300A, etc., generates virtual batches, allowing for a smaller storage and computational footprint than physical batch processing.

[0043] Figure 3C This illustrates the virtual batch mapping using dependency array 300C. Dependency array 300C and... Figure 3A The dependency array 300A and Figure 3B The dependency array 300B is the same, but in Figure 3C The text shows the relationship with... Figure 3A and Figure 3B The virtual batch paths are different from the virtual batch paths. Index 8 includes the ordinal pair (8, -1). The ordinal pair (8, -1) indicates that the term associated with index 8 does not concern itself with any other term associated with dependency array 300C (although, for example, in some examples, another term not indicated by dependency array 300C may have the same form as one or more of the multiple terms, but this same form is accidental and not part of the logical construction of dependency array 300C). Index 9 includes the ordinal pair (9, 8), indicating that the term indicated by index 9 concerns itself with the term indicated by index 8. The first association arrow 302C shows the mapping from index 9 to index 8. The second association arrow 304C shows the mapping from index 10 to index 9, the third association arrow 306C shows the mapping from index 13 to index 10, and the fourth association arrow 308C shows the mapping from index 15 to index 13.

[0044] In this example, the third ordered set of terms of interest is {8, 9, 10, 13, 15}. In some examples, the third ordered set is a chain of third terminal terms (e.g., a complete third virtual batch). In other examples, the third ordered set is part of an incomplete third virtual batch and contains future terms that will focus on the term associated with index 15. The third ordered set does not focus on terms associated with index 15. Figure 3A The first ordered set in the example or Figure 3B The second ordered set in the example contains any lexical units associated with it (again, except where two lexical units happen to have the same form).

[0045] It should be noted that, although in Figure 3A , Figure 3B and Figure 3C China has already presented Figure 3A Dependency graph array 300A Figure 3B Dependency graph array 300B and / or Figure 3CThe dependency graph array 300C is shown as an array of ordinal pairs, but this is not necessary. A dependency graph can be equivalently a 1-dimensional array of single values ​​(e.g., with implicit indices), a matrix of dependencies greater than 2 dimensions (e.g., triples), etc. Dependency graph arrays 300A, 300B, and / or 300C are used for illustrative purposes and should not be considered restrictive. In some examples, although the dependency graph is ordered, the order of the dependency graph does not need to indicate the corresponding order in the memory storing the dependency graph (e.g., a key-value (KV) cache).

[0046] Example Virtual Batch

[0047] Figures 4A to 4D An example virtual batch with a mask is shown generating 400A to 400D. In some respects, this can be achieved from a dependency graph (e.g., Figures 3A to 3C The dependency graph arrays 300A to 300C generate virtual batches 400A to 400D. In some examples, the dependency graph can be generated from virtual batches 400A to 400D. In some examples, virtual batches 400A to 400D are implicitly generated from the dependency graph without creating any physical virtual batches. It should be noted that although the example virtual batches 400A to 400D may suggest that they are generated row by row in sequential order, this is not necessarily the case. This is for ease of understanding and not as a limitation. Any row shown in any virtual batch generation of virtual batches 400A to 400D can be generated in any order. Furthermore, virtual batches 400A to 400C show two rows generated at a time. This is also for ease of understanding and not as a limitation. Other numbers of simultaneous row generation, such as 16 rows, 32 rows, or 1 row, can be used similarly without departing from the basic concept.

[0048] Figure 4A Example virtual batch generation 400A is shown. The virtual batch is shown in grid 402A, labeled with numeric columns 0 to 15 and Roman numeral rows I to XVI. Columns 0 to 15 are associated with dependency graph array 300 (shown as...). Figures 3A to 3CThe dependencies 300A to 300C correspond to indices 0 to 15. Rows and columns of grid 402A will be referred to by ordinal pairs. For example, consider row I, column 0 of grid 400A. The ordinal pair used for this row and column combination is (I, 0). The ordinal pair (I, 0) of grid 402A is shaded with a slash to indicate that the ordinal pair (I, 0) of grid 402A is active. The ordinal pair (II, 1) of grid 402A has a corresponding attention arrow 404A. Attention arrow 404A indicates that ordinal pair (II, 1) pays attention to ordinal pair (II, 0). In some respects, this can be understood as row II corresponding to the attention mask when processing the term at index 1. Specifically, the term associated with index 1 in the dependency graph array 300 focuses on the term associated with index 0, so the unit indicated by the ordinal pairs (II, 0) and (II, 1) is active (unmasked).

[0049] Figure 4B An example virtual batch generation 400B is shown. This example virtual batch generation is... Figure 4A The example virtual batch generation is a continuation of 400A. Rows III and IV of grid 402B (in some respects, this grid is...) Figure 4A The grid 400A (a continuation) is filled. Attention arrow 404B shows ordinal pair (III, 1) focusing on ordinal pair (III, 0), which in some respects shows the lexicon associated with index 1 in dependency graph array 300 focusing on the lexicon associated with index 0. Attention arrow 406B shows ordinal pair (III, 2) focusing on ordinal pair (III, 1), which in some respects shows the lexicon associated with index 2 in dependency graph array 300 focusing on the lexicon associated with index 1. Attention arrow 408B shows ordinal pair (IV, 1) focusing on ordinal pair (IV, 0), which in some respects shows the lexicon associated with index 1 in dependency graph array 300 focusing on the lexicon associated with index 0. Attention arrow 410B shows ordinal pair (IV, 2) focusing on ordinal pair (IV, 1), which in some respects shows the lexicon associated with index 2 in dependency graph array 300 focusing on the lexicon associated with index 1. Attention arrow 412B shows that ordinal pair (IV, 3) focuses on ordinal pair (IV, 2), and in some respects, this shows that the lexicon associated with index 3 in the dependency graph array 300 focuses on the lexicon associated with index 2.

[0050] The example virtual batch generates ordinal pairs 400A and 400B without any masking. Consider the ordinal pairs of the dependency graph array 300 in the form of (index, relevance value), where the ordinal pairs (0, -1) to (4, 3) show the ordered set of indices {0, 1, 2, 3, 4} with the corresponding ordered set of relevance values ​​{-1, 0, 1, 2, 3}. Figure 4C An example virtual batch generation 400C is shown, which is an example virtual batch generation of... Figure 4A and Figure 4B The example virtual batch generates continuations of 400A and 400B. Consider the ordinal pair (5, 3) of dependency graph array 300. The correlation value 3 is the same as the correlation value in the ordinal pair (4, 3) of dependency graph array 300. This will cause the behavior of virtual batch generation 400C to change relative to the behavior of 400A and 400B.

[0051] Rows V and VI of grid 402C (in some respects, the grid is respectively) Figure 4A and Figure 4B (A continuation of grids 402A and 402B) is filled. Attention arrow 404C shows ordinal pair (V, 1) focusing on ordinal pair (V, 0), which in some respects shows the lexicon associated with index 1 in dependency graph array 300 focusing on the lexicon associated with index 0. Attention arrow 406C shows ordinal pair (V, 2) focusing on ordinal pair (V, 1), which in some respects shows the lexicon associated with index 2 in dependency graph array 300 focusing on the lexicon associated with index 1. Attention arrow 408C shows ordinal pair (V, 3) focusing on ordinal pair (V, 2), which in some respects shows the lexicon associated with index 3 in dependency graph array 300 focusing on the lexicon associated with index 2. Attention arrow 410C shows ordinal pair (V, 4) focusing on ordinal pair (V, 3), which in some respects shows the lexicon associated with index 4 in dependency graph array 300 focusing on the lexicon associated with index 3. This completes the ordered index set {0, 1, 2, 3, 4} corresponding to the ordered relevance value set {-1, 0, 1, 2, 3}.

[0052] Attention arrow 412C shows ordinal pair (VI, 1) focusing on ordinal pair (VI, 0). In some respects, this shows that the lexicon associated with index 1 in dependency graph array 300 focuses on the lexicon associated with index 0. Attention arrow 414C shows ordinal pair (VI, 2) focusing on ordinal pair (VI, 1). In some respects, this shows that the lexicon associated with index 2 in dependency graph array 300 focuses on the lexicon associated with index 1. Attention arrow 416C shows ordinal pair (VI, 3) focusing on ordinal pair (VI, 2). In some respects, this shows that the lexicon associated with index 3 in dependency graph array 300 focuses on the lexicon associated with index 2.

[0053] Attention arrow 418C shows that ordinal pair (VI, 5) focuses on ordinal pair (VI, 3). In some respects, this shows that the term associated with index 5 in dependency graph array 300 focuses on the term associated with index 3. It should be noted that ordinal pair (VI, 4) is not shown as focusing on any object, thus making it a masked ordinal pair (as indicated by dotted padding). In some respects, masked ordinal pairs indicate that the term corresponding to the column index is not actively referenced by the corresponding virtual batch. For example, consider two virtual batches constructed from the active units of rows V and VI. In this example, row V is a virtual batch of terms associated with ordinal indices {0, 1, 2, 3, 4}, and row VI is a virtual batch of terms associated with ordinal indices {0, 1, 2, 3, 5}.

[0054] Figure 4D Example virtual batch generation 400D for the entire dependency graph array 300, including full masking, is shown. The example dependency graph array 300 has two indices (index 0 and index 8) with corresponding relevance values ​​of -1, meaning that in this example, at least two virtual batches must exist, since a relevance value of -1 indicates no dependency / attention. More than one virtual batch can exist starting from the same -1 relevance value, as in the case shown in grid 402D for all virtual batches used for the dependency graph array 300. For example, consider rows XIII and XV. Row XIII has a set of first unmasked indices {0, 1, 2, 3, 5, 7, 12}, and row XV has a set of second unmasked indices {0, 1, 2, 3, 4, 6, 11, 14}. Two virtual batches are generated by associating the unmasked indices with corresponding terms from the dependency graph array 300. The XVI row also contains virtual batches represented by a third set {8, 9, 10, 13, 15} of non-masked indices.

[0055] Although examples of three virtual batches corresponding to the first, second, and third sets of unmasked indices have been shown, other virtual batch configurations are possible based on the example dependency graph array 300. For example, consider line XI. Line XI can construct a fourth set of unmasked indices {8, 9, 10}, which, while a subset of the third set of unmasked indices, is still unique. In this way, a total of 16 virtual batches can be generated from the dependency graph array 300 by employing a unique sequence of terms indicated by the active units of each line. Furthermore, in some examples, more than 16 virtual batches can be created from the dependency graph 300 by employing partial portions of one or more lines, other novel combinations between different lines, etc. The examples shown are intended to help illustrate the concepts, not to limit their scope. It should be noted that the construction of physical virtual batches (e.g., batches whose data is stored in a KV cache or other memory) is not necessary for constructing virtual batches, because the information used for virtual batches is stored in a dependency graph array 300, where the index is implicit. In some examples, this dependency graph array 300 can be a 1D array with 16 members. In some examples, Figure 4D The virtual batch shown does not need to be complete, as additional calculations are still possible.

[0056] Large Language Models (LLM)

[0057] Generally speaking, LLM is a type of artificial intelligence (AI). LLM (for example, Figure 2 LLMs (222) are trained on large amounts of data to provide foundational capabilities that are frequently used and reused by fine-tuning for specific applications and tasks. In contrast, other software applications are typically built and trained on specific data for each use case. In this way, LLMs are considered a type of foundational model.

[0058] Some LLMs use machine learning (ML) computer models that can parse language and provide context-aware outputs, such as mimicking human responses. This mimicry of human responses is often directed at prompts, such as questions asked by the user. For example, the prompt "ask how to get to the train station in French" could be used as a prompt for an LLM to provide translation services, i.e., a human response in French to an English prompt.

[0059] As an example, consider Figure 5 The figure illustrates a trainer 500, through which an LLM (e.g., virtual batch 400 in Figure 4) is trained for virtual batches in LLM inference. Figure 2(LLM 222). Trainer 500 receives training data as training input (e.g., input 502). This training data can be of many different types (e.g., labeled text and prediction data). Figure 5 In the example shown, training input 502 is a phrase, although it can alternatively be a word, a long text paragraph (e.g., a book, article, or webpage), or any other data containing comprehensible text. In some examples, the text comes from a screen or image capture. In a process known as “lexicalization,” trainer 500 breaks down training input 502 into lexical units labeled lexical units 502-1, 502-2, 502-3, and 502-4. Here, training input 502 has a missing next word, labeled blank 502-5. The goal of trainer 500 is to predict blank 502-5.

[0060] The trainer 500 encodes tokens (502-1, 502-2, etc.) into the input tensor through a mapping process. In 504, for example, the word "It" is mapped to the input tensor in 502-1. The first component of 504, 504-1, maps the word “s” to the input tensor. The second component of 504, 504-2, maps the word "character" to the input tensor. The third component of 504 is 504-3, and the word “ize” is mapped to the input tensor. The fourth component of 504 is 504-4. Although the lexical units “It” 504-1 and “s” 504-2 are shown as two parts of the word “It's,” other mapping schemes exist (e.g., mappings based on discrete words or phonemes). In some cases, the ML model or ML component training 500 performs training input 502 to the input tensor. 504 (e.g., feature extraction convolutional neural networks (CNN)) lexicalization and / or mapping. Lexicalized training input 502 to input tensor The mapping of 504 can involve a lookup table that maps each possible lexical unit (e.g., 502-1, 502-2, etc.) to a known tensor object in the language space of the training data. In some examples, the mapping of lexical unit 502 is referred to as an embedding.

[0061] Transformer 506 will input tensors 504 is used as input, and the purpose is to transform the input tensor 504 transformation into the transformed tensor 508 is used to predict the blank 502-5. The transformation process is mathematically represented as follows:

[0062] In Equation 1 T This represents transformer 506. The tensor after transformation. 508 includes components 508-1, 508-2, 508-3, 508-4, and 508-5. Component 508-1 is the transformation of component 504-1 through transformer 506 (similarly for component pairs 508-2 / 504-2, 508-3 / 504-3, and 508-4 / 504-4). Component 508-5 corresponds to the blank 502-5, and therefore component 508-5 is a prediction of the blank 504-5. Besides the contextualization of components 504-1 to 504-4, the final transformed tensor is derived as part of the transformation process. 508 component 508-5.

[0063] In some examples, the final transformed tensor The 508-5 component is a multi-component tensor. For example, the second transformed tensor... (Not shown) Different transformations can be performed as follows. (Not shown) to generate:

[0064] Multiple transformed tensors can be generated (e.g., the final transformed tensor). The second transformed tensor In some examples, multiple transformed tensors can be compared by an LLM (e.g., virtual batch 400 in Figure 4), and based on this comparison, one or more of the multiple transformed tensors can be selected for output.

[0065] Input (e.g., input tensor) Training input 504 and / or training input 502 typically includes multiple lexical units. For example, training input 502 includes lexical units 502-1 through 502-4. Trainer 500 transforms a single training input (e.g., training input 502) into multiple training inputs. For example, by removing lexical unit 502-4, and when training input 502 asks trainer 500 to predict lexical unit 502-4, the blank 502-5 is "left-shifted," thereby creating a new training input from the original training input 502. Since the value used for lexical unit 502-4 is known in this example, the new input is a labeled input, which allows the new input to be used by supervised ML training algorithms (it should be noted that such input can also be used by unsupervised ML training algorithms). In this way, a single text containing multiple lexical units (e.g., a book, research paper, etc.) is used as multiple training inputs for trainer 500.

[0066] Figure 6The input tensor component 604-1 is shown (e.g., Figure 5 Input tensor Example transformation 600 in language space 602-1 (components 504-1). Language space 602-1 is a multidimensional mathematical space that includes specific language components coded as tensors within the multidimensional mathematical space. The term "tensor" is a mathematical object of any dimension, including scalar, vector, and matrix quantities. Therefore, language space 602-1 is a mathematical vocabulary, and mapped lexical units (e.g., Figure 5 Lexical unit 502-1 is a lexical unit that has been translated into a mathematical vocabulary. For ease of explanation, language space 602-1 is... Figure 6 The vectors are shown as having orthogonal basis vectors. , and The language space is a three-dimensional space. However, this should not be considered a limitation. In general, the language space 602-1 has the dimension of the mapped lexical units from the input tensor. For example, Figure 5 Input tensor 504 (its tensor components 504-1 to 504-4 each contain...) n (number of members) corresponds to n 3D language space.

[0067] The input tensor component 604-1 is plotted in the language space 602-1. Figure 6 The input tensor component 604-1 is shown as a vector in three-dimensional space. In some examples, the drawing is the product of a lookup table, CNN feature mapping, or any other mapping from lexical units to the language space 602-1. The input tensor component 604-1 is transformed by transformation 600. Consider the same language space 602-2 as the language space 602-1 and the same input tensor component 604-2 as the input tensor component 604-1. Transformation 600 is based on transformation operators 606 and 608 and is performed by the transformer. Transformation operators 606 and 608 are shown as vector addition operators, resulting in a remapped tensor 610.

[0068] As an illustration of this transformation, let the input tensor component 604-2 represent the mapped (e.g., translated into a mathematical vocabulary of language space 602-2) lexical units of "rodent", and generate transformation operators 606 and 608 by contextualizing the mapped lexical units "large" and "eared" from the input cue including the phrase "large-eared rodent". Contextualization is defined as characterizing the input cue (e.g., Figure 5The correlation between “rodent,” “large,” and “eared” in input 502: This corresponds to how a speaker of the language of the input prompt would understand the word “rodent” when it appears in the input prompt along with “large” and “eared.” In this illustration, the transformed tensor 610 is mapped to the region of the language space 602-2 containing the word “chinchilla.”

[0069] Although the transformation from input tensor component 604-2 to the transformed tensor 610 has been shown as two transformations using transformation operators 606 and 608, this should not be considered a limitation. Any number of transformation operations can be employed, including more than two or a single transformation operation. Transformation operators (e.g., transformation operator 606) can also take forms other than vector / tensor addition, including, for example, multiplication (e.g., scaling, matrix multiplication, dot product, cross product, tensor product, etc.), normalization, orthogonalization, or any combination of these or other transformation operations known to those skilled in the art. Therefore, Figure 6 The transformation operators 606 and 608 are intended to be illustrative, not restrictive.

[0070] Example Method

[0071] Method 700 is shown as a set of boxes specifying operations that are performed, but are not necessarily limited to the sequence or combination of operations shown for performance by the corresponding boxes. Furthermore, any of one or more operations may be repeated, combined, rearranged, or linked to provide a wide range of additional and / or alternative methods. In the sections discussed below, reference may be made to any of the foregoing figures or processes described in detail in other figures, references to which are made by way of example only. The technology is not limited to being performed by one or more entities operating on a device.

[0072] Generally, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., a fixed logic circuit system), manual processing, or any combination thereof. Some operations of the example methods can be described in the general context of a computer program product (e.g., executable instructions stored on a computer-readable storage medium located locally and / or remotely on a computer processing system), and implementations can include software applications, programs, functions, etc. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0073] Figure 7 An example method 700 for implementing virtual batches in large language model inference is shown. At 702, multiple lexical units are generated by an LLM (e.g., LLM 222). In some examples, these multiple lexical units include inferences about the input (e.g., input prompt 104). In some aspects, this inference can be a prediction based on the input. In some examples, the input is one or more text inputs, one or more image inputs, one or more video inputs, one or more audio inputs, or any combination of these inputs.

[0074] At position 704, a dependency graph is generated by LLM. In some examples, this dependency graph includes one or more index tags for each of the plurality of tokens and a relevance tag for each of the one or more index tags. In some examples, the dependency graph is an array of dependencies (e.g., Figure 3A The dependency graph array (300A) is shown. In some examples, the dependency array consists of ordinal pairs of index and relevance tags. In some examples, for each ordinal pair, the relevance tag is less than the index tag. According to some examples, the relevance tag indicates the index tag. According to some examples, the index tag is determined by the position of the relevance tag in the dependency array. According to some examples, the dependency array is linear.

[0075] At position 706, multiple virtual batches are generated by LLM (e.g., Figure 4C The example generates virtual batches (400C). In some aspects, multiple virtual batches are generated based on a dependency graph. For example, each of the multiple virtual batches includes independent inferences. In some examples, the multiple virtual batches include a single physical batch with a batch length. According to some examples, the multiple virtual batches include multiple physical batches, each of which has a final length, where the final lengths are combined to form the batch length. In some examples, the batch length is based on the size of the cache memory. In some examples, a first portion of a first batch in the multiple virtual batches is the same as a second portion of a second batch in the multiple virtual batches, and the first batch is different from the second batch.

[0076] In some examples, one or more virtual batches among the plurality of virtual batches include one or more masked markers at one or more positions within the plurality of virtual batches, the one or more masked markers being configured to indicate that the one or more positions within the plurality of virtual batches are unrelated to any of the plurality of tokens. According to some examples, two or more virtual batches among the plurality of virtual batches share the same input, which includes a subset of the plurality of tokens. In some examples, two or more virtual batches among the plurality of virtual batches share a first input, and one or more other virtual batches among the plurality of virtual batches include a second input. This one or more other virtual batches are different from any of the virtual batches among the two or more virtual batches among the plurality of virtual batches, and the first input is different from the second input. In some examples, the first input, the second input, or both are part of a previous input or part of another virtual batch.

[0077] At 708, the LLM selects one or more virtual batches from the plurality of virtual batches. At 710, at least one selected virtual batch is configured for output (e.g., to...). Figure 1 The output 106 of the electronic device 102). For example, a module of the electronic device (e.g., Figure 2 The electronic device 102 (module 210) is used to process at least one virtual batch in the virtual batch. In some examples, the configuration is made by the LLM element ( Figure 2 The action module 232 and interface module 234 of the LLM 222 are used to execute this. In some examples, configuring at least one selected virtual batch in the virtual batch for output includes using at least one selected virtual batch in the virtual batch as at least partially as a second input for the LLM or for another LLM. According to some examples, the output is one or more of an action, response, information, correspondence, or suggestion.

[0078] Proceeding from 706 to 712, multiple virtual batches are compared by the LLM. For example, each of these virtual batches may be assigned a fitness score or value, and these fitness scores or values ​​may be compared. In another example, one or more subsequent batches may be generated, and the multiple virtual batches are compared based on compatibility with these one or more subsequent batches. In some examples, the comparison of virtual batches includes generating multiple fitness scores, where each of the multiple virtual batches is associated with one or more of these fitness scores. Method 700 proceeds from 712 to 708.

[0079] Throughout this disclosure, it is described that a computing system (e.g., computing device 102) can analyze information associated with a user (e.g., Figure 1 Examples of input prompts (104); for example, input prompt 104 could be text from a messaging application (e.g., from an instantiated chat application). In addition to the above description, users may be provided with controls allowing them to choose whether and when the systems, programs, and / or features described herein can collect information (e.g., information about a user's social networks, social actions, social activities, or profession, user preferences, user's current location), and whether content or communications are sent to the user from a server. Computing systems can be configured to use information only after the computing system receives explicit permission from the user to use the data. For example, in cases where an application of the computing system contains private messaging data used as information, users may be provided with the opportunity to provide input to control whether a program or feature of the computing system can collect and utilize information. Furthermore, individual users may have ongoing control over what programs can or cannot process information. Additionally, before the collected information is transmitted, stored, or otherwise used, it may be preprocessed in one or more ways to remove personally identifiable information. For example, private messaging data may allow the removal of personally identifiable profiles, names, and / or faces. Therefore, users can control whether information about themselves and their devices is collected, and if such information is collected, how the computing system and / or remote computing system can use it.

[0080] Additional examples

[0081] This paper describes various examples, including a first example method (Example 1), which includes generating multiple lexical units using a large language model (LLM). The method further includes generating a dependency graph from the LLM, the dependency graph including one or more index tags for each of the multiple lexical units and a relevance tag for each of the one or more index tags. The method further includes generating multiple virtual batches based on the dependency graph, each of the multiple virtual batches including an independent inference. The method further includes selecting one or more virtual batches from the multiple virtual batches as the final inference by the LLM.

[0082] Example 2: As described in Example 1, wherein the dependency graph is an array of dependencies configured as ordinal pairs of the index tag and the relevance tag. For each ordinal pair, the relevance tag is less than the index tag.

[0083] Example 3: The method described in Example 2, wherein the index marker is determined by the position of the correlation marker in the dependency array.

[0084] Example 4: The method described in Example 3, wherein the dependency array is a linear array.

[0085] Example 5: The method as described in any of the preceding examples, wherein the plurality of virtual batches comprises a single physical batch having a batch length.

[0086] Example 6: The method as described in any one of Examples 1 to 4, wherein the plurality of virtual batches comprises a plurality of physical batches, each of the plurality of physical batches having a final length, wherein the final lengths are combined to form a batch length.

[0087] Example 7: The method as described in any one of Examples 5 or 6, wherein the batch length is based on the size of the cache memory.

[0088] Example 8: The method as described in any one of Examples 5 or 6, wherein the batch length is dynamic.

[0089] Example 9: The method as described in Example 1, wherein one or more virtual batches of the plurality of virtual batches include one or more masked markers at one or more locations in the plurality of virtual batches, the one or more masked markers being configured to indicate that the one or more locations in the plurality of virtual batches are not related to any of the plurality of lexical units.

[0090] Example 10: The method as described in any of the preceding examples further includes comparing a plurality of virtual batches by the LLM, wherein selecting one or more of the plurality of virtual batches as the final inference configured for output is at least partially based on the comparison.

[0091] Example 11: The method as described in Example 10, wherein the comparison of the plurality of virtual batches includes generating a plurality of fitness scores. Each of the plurality of virtual batches is associated with one or more fitness scores of the plurality of fitness scores.

[0092] Example 12: The method as described in any of the preceding examples, wherein two or more of the plurality of virtual batches share the same input, the same input comprising a subset of the plurality of lexical units.

[0093] Example 13: The method as described in any of the preceding examples, wherein the plurality of lexical units are based on one or more text inputs, one or more image inputs, one or more video inputs, one or more audio inputs, or any combination of these inputs.

[0094] Example 14: The method as described in any of the preceding examples, wherein two or more of the plurality of virtual batches share a first input, and one or more other virtual batches of the plurality of virtual batches include a second input. The one or more other virtual batches are different from any of the two or more virtual batches in the plurality of virtual batches, and the first input is different from the second input.

[0095] Example 15: The method as described in Example 14, wherein the first input, the second input, or both are part of a previous input or part of another virtual batch.

[0096] Example 16: The method as described in any of the preceding examples, wherein a first portion of a first batch of the plurality of virtual batches is identical to a second portion of a second batch of the plurality of virtual batches, and the first batch is different from the second batch.

[0097] Example 17: The method as described in any of the preceding examples further includes configuring at least one selected virtual batch in the virtual batches for output.

[0098] Example 18: An electronic device including one or more processors and a memory storing instructions that, when accessed by the one or more processors, cause the one or more processors to perform any of the methods described in Examples 1 to 17.

[0099] Example 19: A non-transitory computer-readable medium storing instructions that, when accessed by one or more processors, cause the one or more processors to perform any of the methods described in Examples 1 to 17.

[0100] Example 20: A computer program product including instructions that, when accessed by one or more processors, cause the one or more processors to perform any of the methods described in Examples 1 to 17.

[0101] in conclusion

[0102] As used herein, the phrase “at least one of” or “one or more of” in a list of items refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination of multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0103] Although the concept of virtual batches in large language model inference has been described in technical and / or system-specific language, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific techniques or methods described. Rather, the specific techniques and methods are disclosed as exemplary implementations of virtual batches for large language model inference.

Claims

1. A method comprising: Generate multiple lexical units using a large-scale language model (LLM); The dependency graph generated by the LLM includes: One or more index tags for each of the plurality of lexical units; and A relevance tag for each of the one or more index tags; Multiple virtual batches are generated based on the dependency graph, each of the multiple virtual batches including independent inference; and The LLM selects one or more virtual batches from the plurality of virtual batches as the final inference.

2. The method of claim 1, wherein: The dependency graph is an array of dependencies configured as ordinal pairs of index tags and relevance tags; and For each ordinal pair, the correlation marker is less than the corresponding index marker.

3. The method as described in claim 2, wherein, The index marker is determined by the position of the correlation marker in the dependency array.

4. The method as described in any of the preceding claims, wherein: One or more of the plurality of virtual batches include one or more masked markers at one or more locations in the plurality of virtual batches; and The one or more masked markers are configured to indicate that the one or more positions in the one or more virtual batches are not related to any of the multiple tokens.

5. The method as described in any one of the preceding claims, wherein, The multiple virtual batches include a single physical batch with a batch length.

6. The method of claim 5, wherein, The batch length is based on the size of the cache memory.

7. The method of claim 5, wherein, The batch length is dynamic.

8. The method as described in any of the preceding claims, further comprising comparing the plurality of virtual batches by the LLM, wherein, The selection of one or more virtual batches from the plurality of virtual batches as the final inference is based at least in part on the comparison.

9. The method of claim 8, wherein: The comparison of the plurality of virtual batches includes generating a plurality of fitness scores; and Each of the plurality of virtual batches is associated with one or more fitness scores among the plurality of fitness scores.

10. The method as described in any of the preceding claims, wherein, Two or more of the plurality of virtual batches share the same input, which includes a subset of the plurality of lexical units.

11. The method as described in any of the preceding claims, wherein, The multiple lexical units are based on: One or more text inputs; One or more image inputs; One or more video inputs; One or more audio inputs; or Any combination of these inputs.

12. The method as described in any of the preceding claims, wherein: Two or more of the multiple virtual batches share the first input; One or more other virtual batches among the plurality of virtual batches include a second input; The one or more other virtual batches are different from any of the two or more virtual batches in the plurality of virtual batches, and The first input is different from the second input.

13. The method as claimed in any of the preceding claims, further comprising configuring at least one selected virtual batch in the virtual batches for output.

14. An electronic device comprising: One or more processors; as well as A memory storing instructions that, when accessed by the one or more processors, cause the one or more processors to perform any one of the methods described in claims 1 to 13.

15. A computer program product comprising instructions that, when accessed by one or more processors, cause the one or more processors to perform any one of the methods described in claims 1 to 13.

Images