Computing architecture with model core and fine-tuning portion

Abstract

Methods and systems related to a custom computing architecture are disclosed herein. The disclosed computing architecture includes a model core and a fine-tuning portion. The model core stores a set of parameters of an ML model. The fine-tuning portion stores a set of fine-tuning values for fine-tuning the ML model. The fine-tuned ML model is a fine-tuned version of the ML model. The model core can be fixed during manufacturing of the custom computing architecture. The fine-tuning portion can be fixed after the model core is fixed. The model core can be less configurable than the fine-tuning portion. The set of parameters of the ML model can be defined during manufacturing of the computing architecture. The set of fine-tuning parameters for fine-tuning the ML model can be defined after the set of parameters of the ML model is defined.

Description

[0001] Interactive reference for related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 613,041, filed December 20, 2023, entitled “Computing Architecture with Model Core and Fine-Tuning Portion,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to a computing architecture designed to accelerate a variety of computing tasks by modifying small, configurable parts of the same computing architecture.

[0003] background Over the past few years, we have seen rapid advancements in dedicated computing architectures for applications such as cryptography, cloud computing, machine learning, and others. Computing architectures are constantly evolving to serve applications—for example, parallelizing complex computations and performing specific computations more efficiently—but the computational demands of these applications are growing at an even faster pace. This has led to a significant increase in the need for computing resources. For instance, large machine learning (ML) models, such as large language models (e.g., generative pre-trained transformer (GPT) models), can include hundreds of billions of trainable parameters. This requires significantly more processing power and complexity, thus creating a greater demand for high-performance hardware to enable computing architectures to meet the processing needs of various applications, such as ML models. Summary of the Invention

[0004] To address the aforementioned drawbacks, this paper discloses a computing architecture and method for accelerating various computing tasks by modifying small configurable portions of the computing architecture. The computing architecture includes: a hardwired model core configured to store a set of parameters for a machine learning (ML) model; and a programmable fine-tuning portion configured to store a set of fine-tuning parameters for fine-tuning the ML model. The fine-tuned ML model is a fine-tuned version of the ML model. In some embodiments, the hardwired model core includes a mask read-only memory storing the set of parameters for the ML model, and the programmable fine-tuning portion includes a programmable read-only memory storing the set of fine-tuning parameters for the ML model. In some embodiments, the computing architecture is a multi-core processor.

[0005] In other embodiments, the computing architecture includes: a model core configured to store a set of parameters of a machine learning (ML) model in a first memory; a programmable fine-tuning portion configured to store a set of fine-tuning parameters for fine-tuning the ML model in a second memory; and an inference engine configured to use the set of fine-tuning parameters on the ML model to generate inference from the fine-tuned ML model. The fine-tuned ML model is a fine-tuned version of the ML model, and the first memory has a higher density than the second memory. In some embodiments, the inference engine is further configured to use the set of parameters on the ML model and the set of fine-tuning parameters on the ML model to generate the inference from the fine-tuned ML model. In some embodiments, the first memory is a mask read-only memory, and the second memory is an electrically programmable read-only memory. In some embodiments, the set of fine-tuning parameters forms a low-rank adaptation adapter for the ML model, and the set of fine-tuning parameters replaces the corresponding set of parameters of the ML model in the fine-tuned ML model. In some embodiments, the computing architecture is a multi-core processor.

[0006] In some embodiments, the method includes fabricating a computational architecture having a model core. The model core stores a set of parameters for a machine learning (ML) model in a first memory. The method also includes programming a fine-tuning portion of the computational architecture to form a programmable fine-tuned portion of the computational architecture. The programmable fine-tuned portion stores a set of fine-tuned parameters for fine-tuning the ML model in a second memory. The fine-tuned ML model is a fine-tuned version of the ML model. The first memory has a higher density than the second memory. In some embodiments, the first memory is a mask read-only memory, and the second memory is an electrically programmable read-only memory. In some embodiments, the fine-tuned parameter set forms a low-rank adaptation adapter for the ML model. The fine-tuned parameter set replaces a corresponding set of parameters in the fine-tuned ML model. In some embodiments, the computational architecture is a multi-core processor. In some embodiments, the fine-tuned parameter set for the ML model is used to generate inference from the fine-tuned ML model. In some embodiments, the fine-tuned parameter set is generated by a Parametric Efficient Fine-Tuning (PEFT) routine for the model.

[0007] The above and other preferred features, including various novel details of embodiments and combinations of elements, will now be described in more detail with reference to the accompanying drawings and pointed out in the claims. It should be understood that particular methods and apparatus are shown by way of illustration only and are not intended to be limiting. As will be understood by those skilled in the art, the principles and features explained herein can be used in various and many embodiments. Brief description of the attached diagram The disclosed embodiments have advantages and features that will become more apparent from the detailed description, claims, and accompanying drawings (or figures). A brief description of the drawings is given below.

[0009] Figure 1 An exemplary diagram of the overall computing architecture according to some embodiments is shown.

[0010] Figure 2 Exemplary options for how an inference engine can generate inference using a model core and fine-tuning portions, according to some embodiments, are shown.

[0011] Figure 3 A flowchart of a set of methods for providing a computing architecture, according to some embodiments, is shown.

[0012] Figure 4 A block diagram of an example computer system, according to some embodiments, that can be used to implement the techniques described herein is shown.

[0013] Detailed description The accompanying drawings (Figures) and the following description are for illustrative purposes only and relate to preferred embodiments. It should be noted that, based on the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily identified as feasible alternatives that can be used without departing from the claimed principles.

[0014] Detailed reference will now be made to embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that similar or identical reference numerals may be used in the drawings where feasible, and may indicate similar or identical functionality. The drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0015] Overview While advancements in computing architecture can enhance performance and efficiency, a mismatch between these advancements and the rapidly growing demands of applications can lead to bottlenecks in the computing process (e.g., memory bandwidth, I / O throughput) and increased costs, driving innovation in both hardware and software. This paper discloses methods and systems involving customized computing architectures. Computing architectures can be customized for specific computing tasks. Using the methods disclosed herein, a single computing architecture design can include a core for efficiently accelerating a first computing task and more configurable portions that can be modified to allow the architecture to accelerate a second computing task. The first and second computing tasks can be related, and the second computing task can be selected from many related computing tasks that can be accelerated using the same computing architecture design by modifying the configurable portions in different ways.

[0016] In some embodiments, the computational task may be related to an application that presents inference from a large ML model. While these applications demonstrate significant benefits, training state-of-the-art large ML models can cost millions of dollars, and the resulting models can be highly specialized for a given task. Therefore, these models are often retrained for different purposes rather than retrained from scratch. In this context, a method called Parametric Efficient Fine-Tuning (PEFT) has been developed that allows targeted modifications to large ML models for specific applications without having to retrain all of the model's parameters. For example, a user could take a GPT model designed for general dialogue and modify it specifically for Chinese-to-English translation. Retraining using PEFT is more efficient than retraining all of the model's parameters and is several orders of magnitude more efficient than training the model from scratch.

[0017] The embodiments of the computational architecture disclosed herein can be beneficially applied to the applications described above because the core of the computational architecture can be a model core associated with a model, and a more configurable portion can be a fine-tuning portion associated with a fine-tuning method for modifying the model for a given application. The model core typically refers to the main architecture and parameters of a pre-trained model that encapsulates the model's learned representations and knowledge. The model can be a large-parameter ML model, such as a GPT model, BERT, or ViT-22B. The model core may store parameters defining the model and / or be configured to perform computations necessary to derive inference from the model. The fine-tuning circuitry may store parameters generated by PEFT routines used for the model, and / or be configured to perform computations necessary for the PEFT routines, and / or be configured to perform computations necessary to derive inference from the fine-tuned model. Inference is the practical application of an ML model after it has been trained or fine-tuned, and includes the process of using the ML model to generate predictions based on patterns learned during training or fine-tuning. For fine-tuned models, inference specifically refers to applying task-specific adaptations (learned during fine-tuning) to new data.

[0018] The configurability of the model core may be less than that of the fine-tuning section. The model core may be fixed before the fine-tuning section is fixed. In some embodiments, the model core may be a hardwired model core. The model core may be implemented as part of the computing architecture, which is fixed when the computing architecture is manufactured and completed for deployment. The characteristics of a part of the computing architecture can be fixed in various ways, such as by setting values ​​in read-only memory or programmable read-only memory. As used herein, the term "fabricated" refers to the point in the manufacturing process where the computer chip (e.g., silicon substrate) of the computing architecture is processed in a manufacturing plant, and "final test and customization" refers to the point in the manufacturing process where the programmable read-only memory of the computing architecture is being programmed and / or the firmware of the computing architecture (if present) is being loaded into the computing architecture.

[0019] The tuning section can be more configurable than the hardwired model core. This tuning section can be fixed after the model core is fixed. The tuning section can be a programmable tuning section. For example, the tuning section can be programmed when the computing architecture is deployed and operational, allowing its characteristics to be set by the user operating the computing architecture, while the model core can be fixed during device manufacturing and during final customization before the computing architecture is shipped to the user. As another example, the tuning section can be programmed during final customization before the computing architecture is sent to the customer, and the model core can have its characteristics fixed when the computing architecture is manufactured.

[0020] In some embodiments, the parameters generated by the PEFT routine are stored in a fine-tuning section, which renders some parameters in the model core redundant. The computational architecture can be designed to ignore redundant or redundant parameters when inference is presented from the fine-tuned model. When the computational architecture presents such inference, these redundant parameters represent wasted memory consumption of the model core. However, given the significant difference between space and power consumption in less configurable circuitry systems, the resources attributable to redundant parameters are relatively small. Furthermore, fine-tuned models based on the same model core can be applied to a wide range of applications and share most of the core model. Thus, a single computational architecture design can be modified in a low-cost manner for many different applications using the fine-tuning sections disclosed herein. This can be several orders of magnitude less expensive than providing a completely custom computational architecture for each of those applications, even considering the fact that a single chip design may have slightly redundant functionality.

[0021] System Implementation Plan Figure 1An exemplary diagram of an overall computing architecture 100 according to some embodiments is shown. The computing architecture 100 can be implemented in various ways. In some embodiments, the computing architecture 100 may be a dedicated architecture designed to accelerate a specific workload in a given application, such as generating inference from an ML model or generating hashes based on a cryptographic algorithm. In other embodiments, the computing architecture 100 may be implemented in a data center (e.g., in a set of servers), in an edge environment (e.g., a mobile data center), on a client device (e.g., a mobile phone or wearable device), or on an Internet of Things (IoT) device (e.g., a sensor). In still other embodiments, the computing architecture 100 may also be implemented on a stationary device (e.g., a base station), a mobile vehicle (e.g., an autonomous car), or a group of vehicles (e.g., a swarm of autonomous drones).

[0022] In some embodiments, computing architecture 100 may be implemented via a single computing node or a group of computing nodes operating collaboratively. For example, computing architecture 100 may be implemented as a single application-specific integrated circuit (ASIC), a single-core processor, a multi-core processor, or a network of processors. Computing architecture 100 may also be implemented on a single substrate, on multiple substrates packaged together in a single package, on multiple packages on a common backplane, on one or more servers, and in one or more data centers. In other embodiments, computing architecture 100 may be implemented on multiple computing nodes, such as multiple chips or one or more wafer-level integrated circuits.

[0023] When computing architecture 100 is implemented as a set of computing nodes, computing architecture 100 may include a network, such as a network on a chip (NoC) for multi-core processors. It should be noted that the term "NoC" does not imply that all cores of the processor are on a single semiconductor substrate. More precisely, NoC can be implemented on various interconnect chips. These chips may be integrated into a single package or may be integrated into different packages. These chips may be connected together via a network on different chips and on a common backplane such as a printed circuit board, interposer, or silicon mesh. These chips may also be on different support structures, such as different printed circuit boards or silicon meshes. The network linking computing nodes may include server tiers, rack tiers, and / or inter-datacenter and intra-datacenter tiers. The network may also include any form of interconnect mesh and / or any scale of communication from within the chip to the Internet.

[0024] exist Figure 1In this architecture, the computing architecture 100 may include a model core 101, a fine-tuning section 102, and an inference engine 105. The fine-tuning section 102 may be a programmable fine-tuning section. In some embodiments, the model core 101 may store a set of parameters 103 of the ML model, and the fine-tuning section 102 may store a set of fine-tuning parameters 104 for fine-tuning the ML model. The fine-tuned ML model may be a fine-tuned version of the ML model.

[0025] In some embodiments, inference engine 105 may use model core 101 and fine-tuning portion 102 to generate inference output 106 from input 107. For example, inference engine 105 may generate output 107 in the form of categories of input images (e.g., input 107). In this case, the model being executed by inference engine 105 may be an image classifier. However, it should be noted that inference engine 105 may execute any ML model, such as large language models (LLM), natural language processing (NLP) models, variational autoencoders (VAE), generative adversarial networks (GAN), long short-term memory networks (LSTM), recurrent neural networks (RNN), convolutional neural networks (CNN), transformer models, autoencoders, and / or any other ML model defined by a large number of parameters. Furthermore, the ML model does not necessarily have to be an artificial neural network. The methods disclosed herein are also applicable to reinforcement learning models and other types of models. In other embodiments, alternative computation engines may be used to replace inference engine 105 for different workloads. The alternative computation engines may use model core 101 and fine-tuning portion 102 to perform these different workloads.

[0026] In some embodiments, model core 101 may be fixed during the manufacturing of the custom computing architecture 100 or during final testing and customization. Fine-tuning portion 102 may be fixed at a slightly later time than model core 101 is fixed. In some embodiments, fixing model core 101 may include setting the value of model parameter 103 in the memory of model core 101. The memory may be a first memory 108, which may be read-only memory (ROM), once-programmable (OTP) read-only memory (PROM), electrically programmable read-only memory (EPROM), reprogrammable read-only memory, electrically erasable programmable read-only memory (EEPROM), or another type of memory. Fixing fine-tuning portion 102 may involve setting the value of fine-tuning parameter 104 in the memory of fine-tuning portion 102. The memory may be a second memory 109, which may be ROM, OTP PROM, PROM, reprogrammable read-only memory, EPROM, EEPROM, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, or another type of memory.

[0027] As used herein, the term "fixed" refers to a point in a circuit module where its parameters are locked so that these parameters are set and cannot be changed without reprogramming. For example, a laser-fused PROM is fixed once programmed and its bits are fused or cut. As another example, a mask ROM is fixed when the layers defining it have been applied to the chip's fabrication process. As yet another example, an embedded system module is fixed once the firmware has been loaded into the module by programming its non-volatile memory.

[0028] As used herein, the term “manufacturing” refers to the point in the manufacturing process where the computer chip (e.g., silicon substrate) of computing architecture 100 is processed in a manufacturing plant, and the term “final testing and customization” refers to the point in the manufacturing process where the programmable read-only memory of computing architecture 100 is being programmed and / or the firmware of computing architecture 100 (if present) is being loaded into the computing architecture.

[0029] The model core 101 and the fine-tuning portion 102 can be fixed in various ways. For example, when manufacturing the top layer of one or more chips, the model core 101 implemented on one or more chips can be fixed during manufacturing. In this example, model parameters 103 can be stored in a mask ROM. The mask ROM can store data through different connection forms formed by wires in a configurable mask during the manufacturing process. The mask ROM can store data in different transistor forms, which may or may not be used by the configurable implanted mask during the manufacturing process. For example, in the case of a mask ROM implementation or an OTP implementation, the model core 101 fixed in this way cannot be modified after formation. In this case, the model core 101 is referred to as a hard-wired model core. The fine-tuning portion 102 can be fixed after the model core 101 is fixed. For example, if the model core 101 is fixed during manufacturing, the fine-tuning portion 102 can be fixed during final testing and customization. As another example, if the model core 101 is fixed during final testing and customization, the fine-tuning portion 102 can be fixed when the computing architecture 100 is deployed and in operation.

[0030] In some embodiments, the configurability of model core 101 may be less than that of fine-tuning portion 102. Model core 101 may be hardwired and fixed during device manufacturing, while fine-tuning portion 102 may be programmable and fixed during final testing and customization. Fine-tuning portion 102 may be fixed after manufacturing, such as during OTP programming during final testing and customization, and is referred to herein as a programmable fine-tuning portion. For example, model parameters 103 may be stored in a mask ROM, and fine-tuning parameters 104 may be stored in an OTP memory for writing parameters when computing architecture 100 is completed for delivery to the user. The set of parameters 103 of the ML model may be defined during the manufacturing of computing architecture 100. The set of fine-tuning parameters 104 used to fine-tune the ML model may be defined during the programming of computing architecture 100.

[0031] Model parameters 103 and fine-tuning parameters 104 can take various forms. The two sets of parameters can be of different data types, such as 8-bit integers, 16-bit floating-point numbers, or various other data types. The sets of parameters (e.g., 103, 104) can be of the same data type, or they can be of different data types. In some embodiments, model parameters 103 may include all the parameters necessary to define a large ML model. For example, if the large ML model is a GPT-3 model, then model parameters 103 may include all the parameters necessary to generate inference from GPT-3 (i.e., more than 150 billion parameters). Fine-tuning parameters 104 may include parameters generated by a PEFT routine that operates on model parameters 103 or by certain other routines used to generate parameters to fine-tune the model. The number of fine-tuning parameters 104 may be less than the number of model parameters 103.

[0032] As discussed above, fine-tuning parameter 104 can be the parameters of a fine-tuned ML model, which can be a fine-tuned version of the ML model defined by model parameters 103. In some embodiments, fine-tuning parameter 104 can be selected to replace specific parameters of model core 101, or can be selected to augment the parameters of model core 101, where model core 101 stores model parameters 103. Fine-tuning parameter 104 often augments the parameters of a trained ML model in two ways. One or more new layers (e.g., task-specific fine-tuning layers) can be added to the ML model to adapt it to a specific task, or low-rank fine-tuning parameter metrics can be added in parallel with model core 101 (e.g., the weight matrix of the ML model) so that input can be passed through both the model core and the low-rank matrix to produce a combined output that retains the features of model core 101 while enhancing task-specific performance.

[0033] In some embodiments, the fine-tuning section 102 may include data indicating which parameters are being replaced in model parameters 103 and / or how the fine-tuning parameter 104 is intended to be used to augment model parameters 103. For example, this data may be explicitly stored at the address of the set of parameters in model parameters 103 that will be replaced by a particular fine-tuning parameter 104. Alternatively, the data may be implicitly stored by storing the fine-tuning parameter 104 intended for a particular adapter at the location intended for the design and integration of the fine-tuning section 102 and the inference engine 105.

[0034] The model core 101 and the fine-tuning section 102 can use different types of memory to store parameters. In some embodiments, the model core 101 can store a set of parameters of the ML model (e.g., model parameters 103) in a first memory 108, and the fine-tuning section 104 can store a set of fine-tuning values ​​(e.g., fine-tuning parameters 104) for fine-tuning the ML model in a second memory 109. The first memory 108 may have a higher density than the second memory 109. Specifically, the first memory 108 may be a higher-density and less configurable memory, while the second memory 109 may be a lower-density and more configurable memory. For example, the first memory 108 may be a mask ROM, and the second memory 109 may be a flash EPROM. Therefore, the base model (e.g., model core 101) can be stored in high-density memory, and a large-volume chip can be produced using the base model while the fine-tuning section 102 is modified to adapt the base model for a specific use case. Figure 1 In the example, the base model (e.g., model core 101) can be a general image classifier, and the fine-tuning part (e.g., 102) can fine-tune the base model to improve its performance (e.g., when classifying black and white images).

[0035] The fine-tuning parameter 104 can be generated in various ways. In some embodiments, the fine-tuning parameter 104 may be generated by a separate architecture that executes fine-tuning routines for model parameter 103. The fine-tuning parameter 104 may then be loaded into the computing architecture 100, for example, by programming into a second memory 109 (e.g., non-volatile flash memory). In other embodiments, the fine-tuning section 102 may include a logic circuitry system to execute fine-tuning routines on model parameters (e.g., 103) stored in model core 101. However, given the computational requirements for running standard fine-tuning routines, particularly when the computing architecture 100 is implemented as a multi-core processor or a discrete integrated circuit, it is likely more efficient to run the routines externally and load the fine-tuning parameters into the fine-tuning section 102.

[0036] Fine-tuning routines may include adjusting model core 101 to fit a specific task or dataset while minimizing the number of parameters that need to be updated to achieve better performance. This approach is particularly suitable for scenarios with limited data or computational resources. Specific fine-tuning routines may vary based on the architecture of the model core, the characteristics of the task at hand, and / or the dataset. In some embodiments, a fine-tuning routine may be a PEFT routine (e.g., a low-rank adaptation (LoRA) routine). PEFT identifies the set of parameters in an ML model that needs to be replaced to optimize the ML model for a specific application. PEFT routines can build adapters that work with the ML model to optimize it for a specific application. PEFT routines can build adapters that replace parts or the entire ML model to optimize it for a specific application. PEFT routines may typically generate parameters and any additional data needed to generate a fine-tuned ML model, which is a fine-tuned version of the ML model associated with model core 101.

[0037] Model core 101 and fine-tuning section 102 can be designed in conjunction with inference engine 105 to generate inference from a fine-tuned ML model, depending on the type of fine-tuning routine applied. For example, elements of computing architecture 100 can be designed to replace a portion of parameter 103 from model core 101 with parameter 104 from fine-tuning section 102. Because the memory storing model parameter 103 may be non-erasable, this can be achieved by modifying the address table used to access model parameter 103 to include the address in fine-tuning section 102 that replaces parameter 104. As another example, elements of computing architecture 100 can be configured to modify the instructions executed by inference engine 105 to include adapters or replace portions of the fine-tuned ML model when generating inference from the fine-tuned ML model. Inference engine 105 can be designed to execute two different graphs using stored instructions, one for the ML model and one for the fine-tuned ML model, the stored instructions defining the graphs by the order of operations and the addresses of the required parameters for those operations.

[0038] Elements of the computational architecture 100 may also be configured to use only the original ML model for inference when no fine-tuning routines are performed or when the original model is required at a specific time. Specific state registers in the computational architecture 100 may be configured to place the computational architecture 100 in a mode where a fine-tuned version of the model will be used to generate inference, or in a mode where the model will be used to generate inference. In some embodiments, the same computational architecture may be designed to operate using multiple fine-tuned versions of the model that have been fine-tuned for different applications, and the state registers may be configured to determine which of those multiple fine-tuned versions are applied to generate inference. In these embodiments, the computational architecture 100 may include multiple fine-tuning portions (e.g., multiple copies of fine-tuning portion 102) dedicated to a particular fine-tuning version. Alternatively, the same fine-tuning portion 102 may include different memories or different segments of the same memory to store fine-tuning parameters 104 for different fine-tuning versions and logic for selecting the appropriate fine-tuning parameters 104 for a given fine-tuning version.

[0039] Figure 2 Different options for generating inference using model core 101 and fine-tuning portion 102 according to some embodiments are illustrated. Model layer 200 is an illustration of how input 202 to the ML model can be used to generate layer output 201 using a set of model parameters 203 defining the layer. Input 202 can be the output of a previous layer, and output 201 can be the input to the next layer. Variations of the ML model from which model layer 200 takes to form a fine-tuned model are described in the following paragraphs. Fine-tuning portion 102 of computational architecture 100 may store fine-tuning parameters 104 to augment or otherwise modify the ML model. Alternatively or in combination, fine-tuning parameters 104 may include logic for modifying the ML model. For example, fine-tuning portion 102 may store a set of fine-tuning parameters 104, and this set of fine-tuning parameters 104 may form a low-rank adaptation adapter for the ML model, as described below in the examples of fine-tuning model layers 210 and 220. Alternatively or in combination, the fine-tuning section 102 may store a set of fine-tuning parameters 104, and the set of fine-tuning parameters 104 may replace the corresponding set of parameters of the ML model in a fine-tuned version of the ML model, as shown in the example below in the fine-tuning model layer 230.

[0040] In some embodiments, a fine-tuned model may include the entire original ML model or a portion of the original ML model along with augmentations (e.g., adapters). Fine-tuned model layer 210 is a layer derived from such a fine-tuned model. Fine-tuned model layer 210 includes a set of model parameters 203 from the original ML model along with a low-rank adapter 211. Fine-tuned model layer 210 can thus apply input 202 to both the set of model parameters 203 and the low-rank adapter 211, and then combine the outputs of both 203 and 211 to produce layer output 212. In this approach, the low-rank adapter 211 has significantly fewer parameters than the set of model parameters 203, making it more efficient to retrain the model by modifying only the parameters of the low-rank adapter 211 than to retrain the entire model. In some embodiments of fine-tuned model layer 210, fine-tuning portion 102 may store parameters defining the low-rank adapter 211. Fine-tuning portion 102 may also store information identifying which layers of the ML model should be augmented by including adapters such as the low-rank adapter 211. Furthermore, in some embodiments, the fine-tuning section 102 may include logic for performing computations required to apply input 202 to the low-rank adapter 211. For example, the inference engine 105 may be a hardwired logic system, such as a systolic array, designed to execute ML models. The fine-tuning section 102 may include logic for acquiring layer input values ​​(e.g., input 202) and activation values ​​generated from applying input 202 to the model parameter set 203, applying the input to the low-rank adapter 211, and constructing output 212 for the next layer of the ML model. Logic for modifying the original ML model to produce a fine-tuned model and parameter values ​​may be configurable in the fine-tuning section 102.

[0041] In some embodiments, the fine-tuning model can be a simplified replacement for the original model. For example, the simplified replacement may include a low-rank approximation of the original model. Fine-tuning model layer 220 is a layer derived from such a fine-tuned model. Fine-tuning model layer 220 includes a low-rank approximation 221 of a set of model parameters 203 that can be used to generate layer output 221 from input 202. In this approach, the low-rank approximation 221 has significantly fewer parameters than the set of model parameters 203, making it more efficient to retrain the model by modifying only the parameters of the low-rank approximation 221 than to retrain the entire model. In embodiments according to fine-tuning model layer 220, fine-tuning portion 102 may store the parameters defining the low-rank approximation 221. Alternatively, fine-tuning portion 102 may store the parameters that define the low-rank approximation 221 in conjunction with model core 101. For example, fine-tuning portion 102 may store replacement parameters for the parameters in the set of model parameters 203. Additionally or in combination, the fine-tuning section 102 may store identification information about which parameters from the model parameter set 203 should not be utilized to form a low-rank approximation 221. The fine-tuning section 102 may also store information identifying which layers of the ML model should be augmented using the low-rank approximation 221. Furthermore, in some embodiments, the fine-tuning section 102 may include logic for performing the computations required to apply the input 202 to the low-rank approximation 221. For example, the inference engine 105 may be a hardwired logic system, such as a systolic array, designed to perform ML modeling. The fine-tuning section 102 may include logic for acquiring layer input values ​​(e.g., input 202), applying the input to the low-rank approximation 221, and constructing the output 221 for the next layer of the model. Logic for modifying the model to produce the manner in which the fine-tuned model is adopted, as well as parameter values, may be configurable in the fine-tuning section 102. While an example of a low-rank approximation has been used in this example, the low-rank approximation 221 may be replaced by any simplified version of the model parameter set 203 that is easier to train than the model parameter set 203.

[0042] In some embodiments, fine-tuning the model may include a set of fine-tuning parameters that replaces the corresponding set of parameters of the ML model. Fine-tuning model layer 230 is a layer derived from such a fine-tuning model. Fine-tuning model layer 230 includes a set of fine-tuning parameters 231 of model parameters 203 that can be used to generate layer output 231 from input 202. Figure 2In this method, the fine-tuning parameter set 231 is marked by slashes through the corresponding parameters of the ML model. The number of replacement parameters is much smaller than the number of parameters in model parameter set 203, making it more efficient to retrain the model by modifying only the replacement parameters than to retrain the entire model. In embodiments according to fine-tuning model layer 230, fine-tuning portion 102 may store replacement parameters and identification information for the corresponding parameters to be replaced in model parameter set 203. For example, a position may be explicitly stored by referring to the structure of model parameter set 203 or by referring to the address of a memory (e.g., first memory 108) in model core 101 that stores the model parameter set, identifying the address of the corresponding model parameter. Alternatively, for example, a position may be implicitly stored by storing the value at a specific position in second memory 109 or by storing the replacement value in a data structure having the same size as parameter set 203 so that the data structure can be used as a mask to replace the corresponding value. For example, fine-tuning portion 102 may store replacement parameters for the parameters in model parameter set 203. One advantage of this approach is that the logic required to execute the fine-tuned model will be similar to that required to execute the original model, except for the logic used to retrieve replacement values.

[0043] Figure 3 A flowchart 300 illustrates a set of methods for providing a computing architecture according to some embodiments. Figure 3 The document describes the core design method 300 and two optional distribution methods 310 and 320. Distribution method 310 and distribution method 320 are mutually exclusive.

[0044] The core design method 300 includes step 301 of manufacturing a computational architecture with a model core, wherein the model core stores a set of parameters of an ML model in a first memory. The model core may be model core 101. The first memory may be memory 108. The core design method 300 also includes step 302 of programming a fine-tuning portion of the computational architecture, thereby forming a programmable fine-tuning portion of the computational architecture. The fine-tuning portion may be fine-tuning portion 102. In response to steps 301 and 302 being performed, the programmable fine-tuning portion may store a set of fine-tuning values ​​for fine-tuning the ML model in a second memory. The second memory may be memory 109. The fine-tuned ML model may be a fine-tuned version of the ML model. The first memory may have a higher density than the second memory. The core design method 300 may continue to step 303, wherein inference is generated using the fine-tuned version of the ML model. The fine-tuning parameters determined after inference generation (e.g., 104) may then be uploaded to a device delivered to the user to accommodate the computational needs of one or more tasks customized by the user, as shown in step 304. The generation of inference and fine-tuning parameters are detailed in step 321 below.

[0045] In some embodiments, step 301 may be performed by the manufacturer of the computing architecture (e.g., at a manufacturing facility for semiconductor chips). Steps 303 and 304 may be performed by the user of the computing architecture after it has been deployed for use with a fine-tuned version of the ML model. Step 302 may be performed by the manufacturer of the computing architecture or by the user of the computing architecture, depending on which distribution method is used.

[0046] In some embodiments, a distribution method 320, selected as needed, may be applied. Distribution method 320 includes a step 321 of shipping the device to a user. This step 321 may be performed after manufacturing the model core in step 301 and before programming the fine-tuning portion in step 302. Using this method, a manufacturer can design a computing architecture with a core model and distribute the computing architecture to a user. The user can then modify the computing architecture to produce inference for a specific application using a fine-tuning model specified by the user when fine-tuning the portion 302. In these embodiments, the user may be responsible for performing the training routines necessary to modify the ML model for a given application. This approach can be advantageous because the user will not need to provide the manufacturer with training data for fine-tuning the ML model, thereby improving the security of the training data. Additionally, both the user and the manufacturer can benefit from the lower cost per computing architecture due to the combination of a large, inexpensive, non-configurable core model with a small and therefore inexpensive, configurable fine-tuning portion. Furthermore, in this case, the user may not be an end-user but rather a distributor focused on producing computing architectures for a specific application. This distribution method 320 allows both distributors and original manufacturers to benefit from developing specialized expertise within the distribution chain.

[0047] In some embodiments, a distribution method 310 may be applied as needed. Distribution method 310 includes a step 311 of receiving an order for a fine-tuned model and a step 312 of shipping the device to the user. Step 311 of receiving an order for the fine-tuned model 311 may be performed after manufacturing the model core in step 301. Step 312 of shipping the device to the user 312 may be performed after step 301 of manufacturing the model core and after step 302 of programming the fine-tuned portion. In these embodiments, the manufacturer may be responsible for performing the training routines necessary to modify the ML model for a given application. This approach can be advantageous because the manufacturer can maintain inventory of components with a shared core and can use that same core design to serve many different users or different user applications by modifying more configurable portions of the design. In these embodiments, users may not even be aware that they are ordering the same components for different applications. Instead, all users see a cost reduction in the combination of a large, inexpensive, non-configurable core model and a small, and therefore inexpensive, configurable fine-tuned portion.

[0048] Additional considerations While this specification has been described in detail with respect to specific embodiments of the invention, it should be understood that modifications, variations, and equivalents of these embodiments will readily occur to those skilled in the art upon receiving the foregoing understanding. Any of the method steps discussed above may be implemented by a processor operating on a computer-readable, non-transitory medium storing instructions for those method steps. The computer-readable medium may be memory within a personal user device or network-accessible memory. These and other modifications and variations of the invention may be practiced by those skilled in the art without departing from the scope of the invention as more specifically set forth in the claims.

[0049] Figure 4 This is a block diagram of an example computer system 400 that can be used to implement the techniques described herein. General-purpose computers, network devices, mobile devices, or other electronic systems may also include at least a portion of system 400. System 400 includes a processor 410, memory 420, storage device 430, and input / output device 440. Each of components 410, 420, 430, and 440 may be interconnected, for example, using system bus 440. Processor 410 is capable of processing instructions for execution within system 400. In some embodiments, processor 410 is a single-threaded processor. In some embodiments, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored in memory 420 or stored on storage device 430.

[0050] Memory 420 stores information within system 400. In some embodiments, memory 420 is a non-transitory computer-readable medium. In some embodiments, memory 420 is a volatile memory cell. In some embodiments, memory 420 is a non-volatile memory cell.

[0051] Storage device 430 provides mass storage for system 400. In some embodiments, storage device 430 is a non-transitory computer-readable medium. In various embodiments, storage device 430 may include, for example, a hard disk drive, an optical disk drive, a solid-state drive, a flash memory drive, or some other mass storage device. For example, the storage device may store long-term data (e.g., database data, file system data, etc.). Input / output device 440 provides input / output operations for system 400. In some embodiments, input / output device 440 may include one or more network interface devices, such as Ethernet cards; serial communication devices, such as RS-232 ports; and / or wireless interface devices, such as 802.4 cards, 3G wireless modems, or 4G wireless modems. In some embodiments, input / output devices may include drive devices configured to receive input data and send output data to other input / output devices (e.g., keyboard, printer, and display device 460). In some examples, mobile computing devices, mobile communication devices, and other devices may be used.

[0052] In some implementations, at least a portion of the methods described above can be implemented by instructions that, when executed, cause one or more processing devices to perform the processes and functions described above. These instructions may include, for example, interpreted instructions, such as script instructions, or executable program code, or other instructions stored in a non-transitory computer-readable medium. Storage device 430 may be implemented in a distributed manner via a network, such as a server cluster or a widely distributed group of servers, or may be implemented in a single computing device.

[0053] Although already Figure 4 The example processing system described herein may be implemented in other types of digital electronic circuit systems, in the form of tangible computer software or firmware, in computer hardware (including the structures disclosed herein and their structural equivalents), or in a combination thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-volatile program carrier for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof.

[0054] The term "system" can encompass all kinds of devices, apparatuses, and machines used for processing data, including (as an example) programmable processors, computers, or multiple processors or computers. A processing system may include special-purpose logic circuit systems, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a processing system may also include program code that creates the execution environment for the computer program in question, such as program code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0055] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or program code) may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages) and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., stored in one or more scripts in a markup language file), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file that stores portions of one or more modules, subroutines, or program code). A computer program may be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected by a communications network.

[0056] The programs and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and producing outputs. The programs and logic flows can also be executed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0057] A computer suitable for executing computer programs may include (as an example) a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random access memory, or both. A computer typically includes a central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or transfer data to such mass storage devices or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game control panel, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0058] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including (by example) semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0059] To provide interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) for the user to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending files to and receiving files from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser.

[0060] Embodiments of the subject matter described in this specification can be implemented in computing systems that include back-end components, such as data servers, or middleware components, such as application servers, or front-end components, such as client computers having a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more of these back-end components, middleware components, or front-end components. Components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0061] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact through communication networks. The relationship between clients and servers is established by computer programs running on various computers that have client-server relationships with each other.

[0062] While this specification contains numerous specific implementation details, these details should not be considered as limiting the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in this specification in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be about sub-combinations or variations of sub-combinations.

[0063] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0064] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous. Additional steps or stages may be provided, or steps or stages may be eliminated from the described process. Therefore, other embodiments are within the scope of the claims.

[0065] the term The wording and terminology used in this article are for descriptive purposes and should not be considered restrictive.

[0066] The terms “approximately,” “approximately equal to,” and other similar phrases (e.g., “X has approximately the value of Y” or “X is approximately equal to Y”) used in this specification and claims shall be understood to mean that a value (X) is within a predetermined range of another value (Y). Unless otherwise indicated, the predetermined range may be positive or negative 20%, 10%, 5%, 3%, 1%, 0.1%, or less than 0.1%.

[0067] Unless explicitly indicated to the contrary, the indefinite articles “a” and “an” as used in this specification and claims shall be understood to mean “at least one”. The phrase “and / or” as used in this specification and claims shall be understood to mean “any one or both” of the elements so combined, i.e., elements that are combined in some cases and not combined in others. Multiple elements listed using “and / or” shall be interpreted in the same manner, i.e., “one or more” of the elements so combined. Other elements may exist as needed, besides those specifically identified by the “and / or” clause, whether related to or unrelated to those specifically identified elements. Thus, as a non-limiting example, when referring to “A and / or B” in conjunction with open-ended terms such as “comprising”, in one embodiment may refer only to A (including elements other than B as needed); in another embodiment, only to B (including elements other than A as needed); in yet another embodiment, both A and B (including other elements as needed); and so on.

[0068] As used in this specification and claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” should be interpreted as inclusive, i.e., including multiple elements or a list of elements and, if necessary, at least one, or more than one, of any unlisted items. Only words that explicitly indicate the opposite, such as “only one of…” or “exact one of…” or, when used in a claim, “consisting of…”, will refer to including multiple elements or exactly one element in a list of elements. Generally, when placed before exclusive terms such as “any one,” “one of…,” “only one of…” or “exact one of…”, the word “or” should be interpreted only as indicating an exclusive alternative (i.e., “one or the other but not both”). When used in a claim, “consisting substantially of…” should have its ordinary meaning as used in the field of patent law.

[0069] As used in this specification and claims, the phrase "at least one" referring to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the element list, but does not necessarily include each element specifically listed in the element list and at least one of each element, nor does it exclude any combination of elements in the element list. This definition also allows for the presence of elements other than those specifically identified in the element list referred to by the phrase "at least one," regardless of whether they are related to or unrelated to those specifically identified elements. Thus, as a non-limiting example, "at least one of A and B" (or equivalently "at least one of A or B" or equivalently "at least one of A and / or B") in one embodiment may refer to at least one (or more than one, if needed) A without B (and, if needed, include elements other than B); in another embodiment, it refers to at least one (or more than one, if needed) B without A (and, if needed, include elements other than A); in yet another embodiment, it refers to at least one (or more than one, if needed) A and at least one (or more than one, if needed) B (and, if needed, include other elements); and so on.

[0070] The use of “including,” “containing,” “having,” “containing,” “involving,” and their variations is intended to cover the items listed thereafter and any additional items.

[0071] The use of ordinal numbers such as "first," "second," and "third" to modify claim elements does not imply any priority, precedence, or order of action of one claim element relative to another element, or the temporal order of the actions of the method of execution. Ordinal numbers are used only as labels to distinguish one claim element with a certain name from another element with the same name (but using ordinal numbers) to differentiate claim elements.

[0072] In consideration, the numerical values ​​presented herein (e.g., in tables, graphs, or line graphs) represent minimum or maximum values ​​within a range for the corresponding parameter. Therefore, when added to a claim, the numerical values ​​provide support for the expression of the scope of the claim, based on the teachings herein, which may be above or below the numerical value. Where not included in a claim, the numerical values ​​presented herein should not be considered limiting in any way.

[0073] The words and expressions used herein are for descriptive purposes and are not restrictive, and their use is not intended to exclude any equivalents of the features or portions thereof shown and described. Furthermore, while certain embodiments of the invention have been described, other embodiments incorporating the concepts disclosed herein will be apparent to those skilled in the art without departing from the spirit and scope of the invention. Features and functions of various embodiments can be configured in various combinations and arrangements, all of which are considered to be within the scope of the disclosed invention. Therefore, the described embodiments should be considered illustrative in all respects and not restrictive. Moreover, the configurations, materials, and dimensions described herein are intended to be illustrative and in no way restrictive. Similarly, although physical explanations have been provided for illustrative purposes, they are not intended to bind or limit any particular theory or mechanism to the claims consistent therewith.

Claims

1. A computing architecture, comprising: The hardwired model core is configured to store the parameter set of a machine learning (ML) model; as well as The programmed fine-tuning section is configured to store a set of fine-tuning parameters for fine-tuning an ML model, wherein the fine-tuned ML model is a fine-tuned version of the ML model.

2. The computing architecture according to claim 1, wherein: The hardwired model core includes a first memory for storing the parameter set of the ML model, and The programmed fine-tuning portion includes a second memory that stores the set of fine-tuning parameters for the ML model.

3. The computing architecture according to claim 2, wherein: The first memory includes a mask read-only memory, and The second memory includes one or more of static random access memory (SRAM), dynamic random access memory (DRAM), or other programmable read-only memory.

4. The computing architecture according to claim 2, wherein the first memory has a higher density than the second memory.

5. The computing architecture according to claim 1, wherein the computing architecture is a multi-core processor.

6. A computing architecture, comprising: The core of the model is configured to store the set of parameters of the machine learning (ML) model in the first memory; The programmed fine-tuning section is configured to store a set of fine-tuning parameters for fine-tuning the ML model in a second memory, wherein the fine-tuned ML model is a fine-tuned version of the ML model, and the first memory has a higher density than the second memory. as well as An inference engine configured to use the set of fine-tuned parameters on the ML model to generate inference from the fine-tuned ML model.

7. The computing architecture of claim 6, wherein the inference engine is further configured to use the set of parameters for the ML model and to use the set of fine-tuning parameters for the ML model to generate the inference from the fine-tuned ML model.

8. The computing architecture according to claim 6, wherein: The first memory is a mask read-only memory. The second memory is an electrically programmable read-only memory, and The electrically programmable read-only memory includes at least static random access memory (SRAM) or dynamic random access memory (DRAM).

9. The computational architecture of claim 6, wherein the set of fine-tuning parameters forms a low-rank adaptation adapter for the ML model.

10. The computing architecture of claim 6, wherein the set of fine-tuning parameters replaces the corresponding set of parameters of the ML model in the fine-tuning ML model.

11. The computing architecture of claim 6, wherein the computing architecture is a multi-core processor.

12. A method comprising: A computing architecture with a model core is fabricated, wherein the model core stores the parameter set of a machine learning ML model in a first memory; as well as The fine-tuning portion of the computing architecture is programmed to form the programmed fine-tuning portion of the computing architecture. in: The programmed fine-tuning section stores the set of fine-tuning parameters used to fine-tune the ML model in the second memory. The fine-tuned ML model is a fine-tuned version of the ML model, and The first memory has a higher density than the second memory.

13. The method according to claim 12, wherein: The first memory is a mask read-only memory. The second memory is an electrically programmable read-only memory, and The electrically programmable read-only memory includes at least static random access memory (SRAM) or dynamic random access memory (DRAM).

14. The method of claim 12, wherein the set of fine-tuning parameters forms a low-rank adaptation adapter for the ML model.

15. The method of claim 12, wherein the set of fine-tuning parameters replaces the corresponding set of parameters of the ML model in the fine-tuning ML model.

16. The method of claim 12, wherein the set of fine-tuning parameters is selected to expand the set of parameters of the ML model.

17. The method of claim 12, wherein the computing architecture is a multi-core processor.

18. The method of claim 12, wherein the set of fine-tuning parameters is generated by a Parametric Efficient Fine-Tuning (PEFT) routine for the model.

19. The method according to claim 12, wherein: The core of the model is implemented on at least one chip, and The core of the model is manufactured before the programming of the fine-tuning section.

20. The method of claim 10, wherein the set of fine-tuning parameters for the ML model is used to generate inference from the fine-tuned ML model.

21. The method of claim 10, wherein the corresponding fine-tuned version of the ML model is fine-tuned for the corresponding application.

22. The method of claim 19, wherein the status register is configured to determine the version of the fine-tuned ML model used to generate inference.

Images