Model customization of transducers to improve efficiency

Abstract

Embodiments of the present disclosure include systems and methods for providing model customization of transducers to improve efficiency. A first set of settings for a transformer model is received. Based on the first set of settings, a second set of settings of the transformer model is determined. A transformer model is configured and trained using the first set of settings and the second set of settings. Configuring and training the transformer model using the first set of settings and the second set of settings allows training the transformer model using a reduced amount of computational resources.

Description

Background Art

[0001] The present disclosure relates to computing hardware. More specifically, the present disclosure relates to techniques for training neural networks.

[0002] Natural language understanding (NLU) is a subfield of natural language processing (NLP) in artificial intelligence that addresses the computer's understanding of the structure and meaning of human language. NLU enables speech technology, search engines, and machine translation to infer the user's meaning regardless of how it is expressed.

[0003] Neural networks are machine learning models that power NLU applications. A neural network can be trained for a specific purpose by running a collection of data through it, comparing the results from the neural network with known results, and updating the network based on the differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the present disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0005] Figure 1 A system for providing model customization of transformers to improve efficiency is shown in accordance with some embodiments.

[0006] Figure 2 Language model loss as a function of sparsity according to some embodiments is shown.

[0007] Figure 3 Language model loss as a function of model density according to some embodiments is shown.

[0008] Figure 4 An example of determining model settings according to some embodiments is shown.

[0009] Figure 5 Another example of determining model settings according to some embodiments is shown.

[0010] Figure 6 Another example of determining model settings according to some embodiments is shown.

[0011] Figure 7 Another example of determining model settings according to some embodiments is shown.

[0012] Figure 8 A process for providing model customization of a transformer according to some embodiments is shown.

[0013] Fig. 9Depicted is a simplified block diagram of an example computer system in accordance with some embodiments.

[0014] Fig.10 A neural network processing system is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0015] In the following description, for the purpose of explanation, many examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. These examples and details should not be interpreted as undue limitations on the elements of the claims or the subject matter claimed for protection as a whole. It will be apparent to those skilled in the art that, based on the language of different claims, the subject matter claimed for protection may include some or all of the features in these examples, individually or in combination, and may also include modifications and equivalents of the features and techniques described herein.

[0016] Techniques for providing model customization of a transformer to improve efficiency are described herein. In some embodiments, a computing system may receive a first model setting set of a transformer model. Based on the first model setting set, the computing system determines a second model setting set of the transformer model. The transformer model can be configured and trained using the first model setting set and the second model setting set. The computing system may determine different second model setting sets for different first model setting sets. For example, when the first model parameter set includes a model topology (e.g., number of layers, hidden dimension size, etc.) and the number of tokens used to train the transformer model, the computing system may determine the density level of the parameters used in the transformer model. As another example, if the computing system receives the number of non-zero parameters in the defined transformer model and the number of tokens used to train the transformer model as the first model setting set, the computing system may determine the number of layers, hidden dimension size, and density level of the transformer model. In the case where the computing system receives as a first set of model settings a defined density level, a ratio of the hidden dimension size of the transformer model to the number of layers in the transformer model, and a number of tags to train the transformer model, the computing system may determine the number of parameters for the transformer model and the hidden dimension size and number of layers for the transformer model. If the computing system receives a defined model topology and a defined density value for the first set of model settings, the computing system may determine the number of tags to train the transformer model.

[0017] The techniques described in this application provide multiple benefits and advantages over conventional methods of training transformer models. For example, applying sparsification techniques to the parameters of the transformer model allows the transformer model to be trained using fewer computational resources, but maintaining the same / similar amount of loss. Conventional methods that do not utilize sparsification techniques on the parameters of the transformer model achieve the same / similar amount of loss, but utilize more computational resources to train the transformer model.

[0018] Figure 1 A system 100 for providing model customization of a transformer to improve efficiency according to some embodiments is shown. As shown, the system 100 includes a client device 105, a computing system 110, and (multiple) artificial intelligence (AI) processors 135. The client device 105 is configured to interact with the computing system 110. For example, a user of the client device 105 can provide a first set of model settings for a transformer model to the computing system 110. In turn, the client device 105 receives a second set of model settings from the computing system 110. Then, the user of the client device 105 provides the first set of model settings and the second set of model settings to the computing system 110 to configure the transformer model and train the transformer model.

[0019] like Figure 1 As shown, the computing system 110 includes a model setup manager 115, a model manager 120, a transformer model storage device 125, and a training data storage device 130. The transformer model storage device 125 stores the transformer model, and the training data storage device 130 stores the training data for training the transformer model. In some embodiments, the transformer model is a machine learning model that includes a set of layers and a self-attention mechanism (e.g., self-attention heads). In some such embodiments, each layer of the transformer model includes a set of self-attention heads. In some embodiments, the storage devices 125 and 130 are implemented in a single physical storage device, while in other embodiments, the storage devices 125 and 130 may be implemented across several physical storage devices. Although Figure 1 Storage devices 125 and 130 are shown as part of computing system 110 , but one of ordinary skill in the art will appreciate that in some embodiments, transformer model storage device 125 and / or training data storage device 130 may be external to computing system 110 .

[0020] The model settings manager 115 is configured to manage model settings for the transformer model. For example, the model settings manager 115 may receive a first set of model settings (e.g., from the client device 105). In response, the model settings manager 115 determines a second set of model settings. In some cases, the model settings manager 115 sends the second set of model settings to the client device 105. In other cases, the model settings manager 115 sends the first set of model settings and the second set of model settings to the model manager 120 for further processing.

[0021] In some embodiments, the model settings manager 115 determines a second set of model settings for a given first set of model settings by introducing parameter sparsity as a variable for configuring the transformer model and utilizing the efficiency gained from parameter sparsity to determine other model settings. The sparsity scaling principle will now be explained to demonstrate the efficiency gained from parameter sparsity. Figure 2 The language model loss as a function of sparsity according to some embodiments is shown. Specifically, Figure 2 A diagram 200 conceptually depicting the language model loss as a function of model parameter sparsity is shown. As shown, the diagram 200 includes a dense Pareto boundary 205, which shows the relationship between the non-zero parameters (excluding the embedding parameters) in the transformer model and the language model loss of the transformer model after training. As the number of non-zero parameters decreases, the language model loss increases. The diagram 200 also includes a sparse Pareto boundary 210. The sparse Pareto boundary 210 shows the relationship between the non-zero parameters (excluding the embedding parameters) in the transformer model that has been sparse and the language model loss of the sparse transformer model after training. As shown, a dense transformer model with a given number of non-zero parameters and a sparse transformer model including fewer non-zero parameters than the dense transformer model reach the same language model loss. In fact, the sparse transformer model can achieve the same language model loss as the corresponding dense transformer model, but the sparse transformer model can do this using fewer computing resources. As shown in graph 200 , efficiency gain 215 may refer to the difference between non-zero parameters in a sparse Transformer model and a corresponding dense Transformer model for a given language model loss.

[0022] Figure 3 The language model loss as a function of model density according to some embodiments is shown. Specifically, Figure 3 A chart 300 conceptually illustrating language model loss as a function of model density is shown. As shown, chart 300 includes three regions 305-315. In region 305 (also referred to as the low error platform), the sparse transformer model has the same / similar accuracy as the corresponding dense transformer model. In region 310 (also referred to as the power law region), on a logarithmic scale, there is a linear correlation between language model loss and model density (e.g., density = 1-sparsity). The transition point from the low error platform to the power law region can be defined as a critical density level. In region 315 (also referred to as the high error platform), the sparse transformer model has the same / similar accuracy as the initialized dense transformer model.

[0023] To quantify the efficiency gain of the converter model, the following formula (1) is used:

[0024]

[0025] Where N total is the total number of parameters in the Dense Transformer model excluding vocabulary parameters and positional parameters, α N As N total The power law exponent of the function used to scale the dense loss, L dense is of size N total The loss of the converter model, N c Is L dense 、N total and α N In some embodiments, N c =8.8×10 13 non-embedded parameters, α N = 0.076. In some embodiments, N total can be estimated as 12*H 2 *n layer , where H is the hidden dimension size of the Transformer model, and n layer is the depth (e.g., number of layers) of the Transformer model.

[0026] For the purpose of quantifying the efficiency gain, region 315 will be ignored. Regions 305 and 310 in graph 300 may be modeled using equation (2) below:

[0027]

[0028] where d is the density level of the transformer model, d cr is the critical density level mentioned above, β is a constant equal to the value 4, γ is the slope in the sparse power law region mentioned above, L sparse is the loss of the transformer model after it has been thinned to a density level d. Here, the value of d can be between [0-1], where a density of 1 indicates zero sparsity (e.g., the model is dense). Equation (2) can be rewritten as the following equations (3)-(6):

[0029]

[0030] Next, the efficiency gain can be defined according to the following equation (7):

[0031]

[0032] Where N'total is the total number of parameters in the converter model excluding embedded parameters, and effgain is the efficiency gain. Equation (7) can be rewritten as the following equations (8) and (9):

[0033]

[0034] Now assume that γ and d cr Regardless of the density level (d) of the model, effgain can be maximized according to the following equations (10) and (11):

[0035]

[0036] The optimal density level can be determined using equations (12) and (13) below:

[0037]

[0038] d opt =d cr for γ≤α N andγ≥2α N

[0039] where d opt is the optimal density level for the transformer model. The optimal density level varies depending on the model topology (e.g., number of layers, hidden dimension size, ratio of number of layers to hidden dimension size (also called aspect ratio), etc.). In some embodiments, γ is a function of the number of layers in the transformer model, the hidden dimension size, and the number of tags used to train the transformer model. Such a function can be modeled using the following equation (14):

[0040]

[0041] Among them, α γ =0.002, β n =0.089, β h =0.041, β t = 0.127, H is the hidden dimension size of the transformer model, and T is the number of tags used to train the transformer model. In some embodiments, d cr is the transformer model width (e.g., hidden dimension size) and aspect ratio (e.g., H / n layer ). The aspect ratio may control the y-intercept (rather than the slope) on a log-log scale. In some embodiments, the slope may be modeled by analyzing a transformer model with a fixed aspect ratio. Once the slope is quantified, the y-intercept may be modeled by analyzing several data points with different aspect ratios (e.g., keeping the slope fixed between different fits) using the following equation (15):

[0042] Make d cr >d random

[0043] The model manager 120 is responsible for managing the transformer model. For example, the model manager 120 may receive a first set of model settings and a second set of model settings (e.g., from the client device 105, from the model settings manager 115, etc.). In response, the model manager 120 generates, configures, and trains the transformer model based on the received first set of model settings and the second set of model settings. The model manager 120 may train the transformer model using (multiple) AI processors 135 and training data retrieved from the training data storage device 130. After the transformer model is trained, the model manager 120 may store the trained transformer model in the transformer model storage device 125.

[0044] The AI ​​processor(s) 135 are hardware configured to implement and execute the transformer model. The AI ​​processor(s) 135 may include a graphics processor (GPU), an AI accelerator, or other digital processor optimized for AI operations. For example, the AI ​​processor(s) 135 may receive the transformer model and a training data set. In response, the AI ​​processor 135 trains the transformer model using the training data set.

[0045] Now refer to Figure 4-Figure 7 1 and 2 to describe several example operations. Specifically, these example operations illustrate how the model settings manager 115 can determine different sets of model settings for different given sets of model settings. Figure 4A first example of determining model settings according to some embodiments is shown. For this example, the model settings manager 115 receives a model topology settings set 405 for a transformer model and a number of marker settings 410 for training the transformer model (e.g., from a client device 105, a service or application operating on a computing system 110 or other computing device / system, etc.). In some embodiments, the model topology settings set 405 may include the depth of the transformer model (e.g., the number of layers in the transformer model) and the hidden dimension size of the transformer model. When the model settings manager 115 receives these model settings, the model settings manager 115 uses equations (12) and (13) to determine an optimal density level 415 for the transformer model. The model settings manager 115 sends the model topology settings set 405, the number of marker settings 410, and the optimal model density level 415 to the model manager 120. Upon receiving these settings, the model manager 120 generates and configures a transformer model with the settings specified in the model topology settings set 405. In addition, the model manager 120 applies a sparsification technique to sparsify the parameters of the transformer model to an optimal density level 415. The model manager 120 then instructs the AI ​​processor(s) 135 to implement the transformer model and train it using the number of tags specified in the settings 410. The AI ​​processor(s) 135 retrieve the requested number of tags from the training data store 130 and train the transformer model. Once the transformer model is trained, the model manager 120 may store it in the transformer model storage 125 for later use in inference.

[0046] Figure 5 A second example of determining model settings according to some embodiments is shown. Figure 5As shown, in this example, the model setting manager 115 receives (e.g., from the client device 105, a service or application operating on the computing system 110 or other computing device / system, etc.) a non-zero parameter number setting 505 for the transformer model and a tag number setting 510 for training the transformer model. Once the model setting manager 115 receives these model settings, the model setting manager 115 uses equation (9) to determine the hidden dimension size 515, the number of layers 520, and the density level 525 of the transformer model. Here, the model setting manager 115 can use a multi-objective optimization function with respect to equation (9) to determine the settings 515-525. Next, the model setting manager 115 sends the non-zero parameter number setting 505, the tag number setting 510, the hidden dimension size 515, the number of layers 520, and the density level 525 to the model manager 120. When the model manager 120 receives these settings, the model manager 120 generates and configures the transformer model with the hidden dimension size specified in the setting 515 and the number of layers specified in the setting 520. The model manager 120 then applies a sparsification technique to sparsify the parameters of the transformer model to have the number of non-zero parameters specified in setting 505 and to be at the density level specified in setting 525. The model manager 120 proceeds to instruct the AI ​​processor(s) 135 to implement the transformer model and train it using the number of labels specified in setting 510. The AI ​​processor(s) 135 retrieve the requested number of labels from the training data storage 130 and train the transformer model. After training is complete, the model manager 120 may store the transformer model in the transformer model storage 125 for later use in inference.

[0047] Figure 6 A third example of determining model settings according to some embodiments is shown. For this example, the model settings manager 115 receives (e.g., from the client device 105, a service or application operating on the computing system 110 or other computing device / system, etc.) a density level setting 605, an aspect ratio setting 610, and a number of labels used to train the transformer model setting 615 for the transformer model. In this example, the aspect ratio is defined as the size of the hidden dimension of the transformer model divided by the number of layers in the transformer model (i.e., H / n layer). When the model settings manager 115 receives these model settings, the model settings manager 115 uses equation (9) to determine the number of parameters 625, the hidden dimension size 630, and the number of layers 635 of the transformer model. For this example, the model settings manager 115 can apply the multi-objective optimization function to equation (9) to determine the settings 625-635. The model settings manager 115 sends the density level settings 605, the aspect ratio settings 610, the number of markers settings 615, the number of parameters 625, the hidden dimension size 630, and the number of layers 635 to the model manager 120. After receiving these settings, the model manager 120 generates and configures the transformer model with the hidden dimension size specified in the settings 630, the number of layers specified in the settings 635, and the aspect ratio specified in the settings 610. The model manager 120 then applies a sparsification technique to sparsify the parameters of the transformer model to have a non-zero number of parameters specified in the settings 625 and at the density level specified in the settings 605. Next, the model manager 120 instructs the AI ​​processor(s) 135 to implement the transformer model and train it using the number of tags specified in the settings 615. The AI ​​processor(s) 135 retrieve the requested number of tags from the training data store 130 and train the transformer model. Once the transformer model is trained, the model manager 120 may store it in the transformer model store 125 for later use in inference.

[0048] Figure 7A fourth example of determining model settings according to some embodiments is shown. In this example, the model settings manager 115 (e.g., from a client device 105, a service or application operating on a computing system 110 or other computing device / system, etc.) receives a model topology settings set 705 and a density level setting 710 for a transformer model. In some embodiments, the model topology settings set 705 may include the depth of the transformer model (e.g., the number of layers in the transformer model) and the hidden dimension size of the transformer model. Once the model settings manager 115 receives these model settings, the model settings manager 115 uses equation (9) to determine the number of tags 715 used to train the transformer model. Next, the model settings manager 115 sends the model topology settings set 705, the density level settings 710, and the number of tags 715 to the model manager 120. When the model manager 120 receives these settings, the model manager 120 generates and configures a transformer model with the settings specified in the model topology settings set 705. The model manager 120 then applies a sparse technique to sparse the parameters of the transformer model to the density level specified in the settings 710. The model manager 120 proceeds to instruct the AI ​​processor(s) 135 to implement the transformer model and train it using the number of tags specified in settings 715. The AI ​​processor(s) 135 retrieve the requested number of tags from the training data store 130 and train the transformer model. After training is complete, the model manager 120 may store the transformer model in the transformer model storage 125 for later use in inference.

[0049] As mentioned above, Figure 2 A graph 200 conceptually depicting the language model loss as a function of the sparsity of the model parameters is shown. In addition, equation (9) provided above is a function of multiple variables (e.g., the hidden dimension size of the transformer model, the number of layers in the transformer model, the density level of the parameters in the transformer model, the number of tokens used to train the transformer model, etc.). Thus, a multi-objective optimization function can be used to calculate Figure 1 Sparse Pareto frontier 210 is shown. The multi-objective optimization function can maximize efficiency gains with respect to multiple variables.

[0050] Reference above Figure 4-Figure 7 The described example operations utilize a sparsification technique to sparsify the transformer model.One of ordinary skill in the art will appreciate that any number of different sparsification techniques (eg, dynamic amplitude clipping techniques) may be used to sparsify the transformer model.

[0051] Figure 8 A process 800 for providing model customization of a transformer is shown according to some embodiments. In some embodiments, computing system 110 performs process 800. Process 800 begins by receiving a first set of settings for a transformer model at 810. Figure 1 and Figure 4 As an example, the model settings manager 115 may receive (eg, from the client device 105 ) a first set of settings (eg, a set of model topology settings 405 and a number of marker settings 410 for a transformer model).

[0052] Next, based on the first set of settings, process 800 determines a second set of settings for the converter model at 820. Figure 1 and Figure 4 As an example, the model settings manager 115 determines a second set of settings (eg, the optimal density level 415 ) based on the first set of settings.

[0053] Finally, process 800 configures and trains the transformer model using the first set of settings and the second set of settings at 830. Figure 1 and Figure 4 As an example, the model manager 120 may generate and configure a transformer model having the settings specified in the model topology settings set 405. In addition, the model manager 120 may apply a sparsification technique to sparsify the parameters of the transformer model to an optimal density level 415. The model manager 120 then instructs the AI ​​processor(s) 135 to implement the transformer model and train it using the number of labels specified in the settings 410.

[0054] The techniques described above can be implemented in various computer systems configured to process neural networks. Fig. 9 A simplified block diagram of an example computer system 900 that can be used to implement the techniques described in the foregoing disclosure is depicted. For example, the computer system 900 can be used to implement the client device 105 and the computing system 110. Fig. 9 As shown, computer system 900 includes one or more processors 902 that communicate with a plurality of peripheral devices via a bus subsystem 904. These peripheral devices may include a storage subsystem 906 (e.g., including a memory subsystem 908 and a file storage subsystem 910) and a network interface subsystem 916. Some computer systems may also include a user interface input device 912 and / or a user interface output device 914.

[0055] The bus subsystem 904 may provide a mechanism for the various components and subsystems of the computer system 900 to communicate with each other as intended. Although the bus subsystem 904 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple busses.

[0056] The network interface subsystem 916 may be used as an interface for transmitting data between the computer system 900 and other computer systems or networks. Embodiments of the network interface subsystem 916 may include, for example, Ethernet, Wi-Fi, and / or cellular adapters, modems (telephone, satellite, cable, ISDN (Integrated Services Digital Network), etc.), digital subscriber line (DSL) units, etc.

[0057] Storage subsystem 906 includes memory subsystem 908 and file / disk storage subsystem 910. Subsystems 908 and 910, as well as other memories described herein, are examples of non-transitory computer-readable storage media that may store executable program code and / or data that provide the functionality of embodiments of the present disclosure.

[0058] The memory subsystem 908 includes a plurality of memories, including a main random access memory (RAM) 918 for storing instructions and data during program execution and a read-only memory (ROM) 920 for storing fixed instructions. The file storage subsystem 910 may provide permanent (e.g., non-volatile) storage for program and data files, and may include a magnetic or solid-state hard drive, an optical drive and associated removable media (e.g., CD-ROM, DVD, Blu-ray, etc.), a removable flash-based drive or card, and / or other types of storage media known in the art.

[0059] It should be understood that computer system 900 is illustrative and that many other configurations are possible having more or fewer components than system 900.

[0060] Fig.10 A neural network processing system according to some embodiments is shown. In various embodiments, a neural network according to the present disclosure may be implemented and trained in a hardware environment including one or more neural network (NN) processors. For example, a neural network processor may refer to various graphics processing units (GPUs) (e.g., GPUs for processing neural networks produced by FPGAs for processing neural networks produced by Intel FPGAs), or various application specific integrated circuits (ASICs), or neural network processors including hardware architectures optimized for neural network computing. In this example environment, one or more servers 1002 (which may include the above Fig. 9 The architecture shown in FIG. 1000 may be coupled to multiple controllers 1010 ( 1 ) - 1010 (M) via a communication network 1001 (eg, a switch, a router, etc.). The controllers 1010 ( 1 ) - 1010 (M) may also include the above Fig. 9 The architecture shown. Each controller 1010(1)-1010(M) can be coupled to one or more NN processors, such as processors 1011(1)-1011(N) and 1012(1)-1012(N). In some embodiments, the NN processors 1011(1)-1011(N) and 1012(1)-1012(N) can be used to implement (multiple) AI processors 135. NN processors 1011(1)-1011(N) and 1012(1)-1012(N) can include various configurations of functional processing blocks and memories optimized for neural network processing (such as training or inference). NN processors are optimized for neural network computations. Server 1002 can configure controller 1010 using a NN model and input data for the model, for example, the input data for the model can be loaded and executed in parallel by NN processors 1011(1)-1011(N) and 1012(1)-1012(N). For example, the model may include the above-mentioned layers and associated weights. The NN processor may load the model and apply the input to produce the output result. For example, the NN processor may also implement the training algorithm described herein.

[0061] Further example embodiments

[0062] In various embodiments, the present disclosure includes systems, methods, and apparatus for providing model customization of a transformer to improve efficiency. The techniques described herein may be embodied in a non-transitory machine-readable medium storing a program executable by a computer system, the program including a set of instructions for performing the techniques described herein. In some embodiments, a system includes a set of processing units and a non-transitory machine-readable medium storing instructions that, when executed by at least one processing unit in the set of processing units, cause the at least one processing unit to perform the above-described techniques. In some embodiments, the non-transitory machine-readable medium may be, for example, a memory, which may be coupled to one or more controllers or one or more artificial intelligence processors, for example.

[0063] The following techniques may be embodied alone or in various combinations, and may further be embodied together with other techniques described herein.

[0064] For example, in one embodiment, the present disclosure includes a method comprising: receiving a first set of settings for a transformer model; based on the first set of settings, determining a second set of settings for the transformer model; and configuring and training the transformer model using the first set of settings and the second set of settings, wherein configuring and training the transformer model using the first set of settings and the second set of settings allows the transformer model to be trained using a reduced amount of computing resources.

[0065] In one embodiment, the first set of settings comprises a set of settings associated with a topology of the converter model.

[0066] In one embodiment, the set of settings includes the number of layers of the transformer model.

[0067] In one embodiment, the set of settings includes the size of the latent dimension of the transformer model.

[0068] In one embodiment, the first set of settings also includes the number of tags used to train the transformer model.

[0069] In one embodiment, the second set of settings includes density values ​​for a plurality of parameters in the transformer model.

[0070] In one embodiment, configuring and training the transformer model using the first set of settings and the second set of settings includes applying a sparsification technique to a plurality of parameters of the transformer model.

[0071] In one embodiment, the first set of settings also includes density values ​​for a plurality of parameters in the converter model.

[0072] In one embodiment, the second set of settings includes the number of tags used to train the transformer model.

[0073] In one embodiment, the first set of settings includes the number of non-zero parameters in the Transformer model, the number of labels used to train the Transformer model, and the size of the hidden dimension of the Transformer model.

[0074] In one embodiment, the second set of settings also includes the number of layers of the transformer model.

[0075] In one embodiment, the second set of settings also includes density values ​​for a plurality of parameters in the transformer model.

[0076] In one embodiment, configuring and training the transformer model using the first set of settings and the second set of settings includes applying a sparsification technique to a plurality of parameters of the transformer model.

[0077] In one embodiment, the first set of settings includes density values ​​of multiple parameters in the transformer model, a ratio of a hidden dimension size of the transformer model to the number of layers of the transformer model, and a number of tags used to train the transformer model.

[0078] In one embodiment, the second set of settings includes the number of parameters in the converter model.

[0079] In one embodiment, the converter model is a first converter model. The disclosure further determines a first loss value for the first converter model and determines a second loss value for the second converter model. Determining the second set of settings for the converter model is based on a ratio of the first loss value to the second loss value.

[0080] The above description illustrates various embodiments of the present disclosure and examples of how various aspects of specific embodiments may be implemented. The above examples should not be considered to be the only embodiments and are presented to illustrate the flexibility and advantages of specific embodiments defined by the appended claims. Based on the above disclosure and the appended claims, other arrangements, embodiments, implementations and equivalents may be adopted without departing from the scope of the present disclosure defined by the claims.

Claims

1. A machine-readable medium storing a program executable by at least one processing unit of a device, the program comprising a set of instructions for: receiving a first set of settings for a transformer model; determining a second set of settings for the converter model based on the first set of settings; and The transformer model is configured and trained using the first set of settings and the second set of settings, wherein configuring and training the transformer model using the first set of settings and the second set of settings allows the transformer model to be trained using a reduced amount of computational resources. 2 . The machine-readable medium of claim 1 , wherein the first set of settings comprises a set of settings associated with a topology of the transformer model. 3 . The machine-readable medium of claim 2 , wherein the set of settings comprises a number of layers of the transformer model. 4 . The machine-readable medium of claim 2 , wherein the set of settings includes a size of a latent dimension of the transformer model. 5 . The machine-readable medium of claim 2 , wherein the first set of settings further comprises a number of tags used to train the transformer model. 6 . The machine-readable medium of claim 5 , wherein the second set of settings includes density values ​​for a plurality of parameters in the transformer model. 7 . The machine-readable medium of claim 6 , wherein configuring and training the transformer model using the first set of settings and the second set of settings comprises applying a sparsification technique to a plurality of parameters of the transformer model.

8. The machine-readable medium of claim 2, wherein the first set of settings further comprises density values ​​for a plurality of parameters in the transformer model.

9. The machine-readable medium of claim 8, wherein the second set of settings comprises a number of tags used to train the transformer model.

10. The machine-readable medium of claim 1, wherein the first set of settings comprises a number of non-zero parameters in the transformer model, a number of labels used to train the transformer model, and a size of a hidden dimension of the transformer model.

11. The machine-readable medium of claim 10, wherein the second set of settings further comprises a number of layers of the transformer model.

12. The machine-readable medium of claim 10, wherein the second set of settings further comprises density values ​​for a plurality of parameters in the transformer model.

13. The machine-readable medium of claim 12, wherein configuring and training the transformer model using the first set of settings and the second set of settings comprises applying a sparsification technique to a plurality of parameters of the transformer model.

14. A system comprising: A collection of processing units; as well as A machine-readable medium storing instructions that, when executed by at least one processing unit in the set of processing units, cause the at least one processing unit to: receiving a first set of settings for a transformer model; determining a second set of settings for the converter model based on the first set of settings; as well as The transformer model is configured and trained using the first set of settings and the second set of settings, wherein configuring and training the transformer model using the first set of settings and the second set of settings allows the transformer model to be trained using a reduced amount of computational resources.

15. A method comprising: receiving a first set of settings for a transformer model; determining a second set of settings for the converter model based on the first set of settings; as well as The transformer model is configured and trained using the first set of settings and the second set of settings, wherein configuring and training the transformer model using the first set of settings and the second set of settings allows the transformer model to be trained using a reduced amount of computational resources.

Images