Method and system for training large-scale language model

Abstract

A computer-implemented method for model training is provided, the method being performed by a processing system. The method comprises the following steps: determining a first weight set based on a first matrix associated with a source model; determining a second weight set based on the first weight set; forming a second matrix associated with a target model based on the first weight set and the second weight set; initializing the target model based on the second matrix; and training the target model.

Description

Technical Field

[0001] The present disclosure generally relates to machine learning techniques, and more specifically, to pre-training of neural network models. Appendix A contains a document entitled "bert2BERT: Towards Reusable Pretrained Language Models" published by ChengChen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, XiaoChen, Zhiyuan Liu, Qun Liu (see aclanthology.org / 2022.acl-long.151), the entire contents of which are incorporated herein by reference. Appendix A cites exemplary experimental settings and results by applying the techniques in the present disclosure and comparison with the prior art. Background Art

[0002] A pre-trained language model (PLM) is a neural network model, such as bidirectional encoder representations from transformers (BERT) and generative pre-training transformer (GPT). PLMs have achieved great success in the field of natural language processing (NLP). Recently, there is a trend to train very large models to explore the upper limit of PLMs. For example, large-scale PLMs such as the third generation GPT (GPT-3) have about 175 billion (175B) parameters, PanGu-α has about 200B parameters, and the switch transformer has about 1571B parameters. All of these large-scale PLMs have been shown to be promising tools for language understanding and generation.

[0003] However, large-scale PLMs are pre-trained independently from scratch without leveraging the knowledge of smaller, trained PLMs. The pre-training process of large-scale PLMs is computationally expensive and has a large carbon footprint. 6The estimated cost of training GPT-3PLM is $4.6 million per graphics processing unit (“GPU”) hour, thus consuming a large amount of computing resources.

[0004] There is a need to develop solutions that reduce the training costs associated with large-scale PLM, thereby saving computational costs and computing resources. Summary of the invention

[0005] In an exemplary embodiment, the present disclosure provides a method for model training, the method being performed by a processing system. The method comprises: a) determining a first weight set based on a first matrix associated with a source model; b) determining a second weight set based on the first weight set; c) forming a second matrix associated with a target model based on the first weight set and the second weight set; d) initializing the target model based on the second matrix; and e) training the target model.

[0006] In another exemplary embodiment, the first matrix includes weights associated with connections between nodes in the current layer and nodes in the upper layer in the source model. The determining of the first set of weights based on the first matrix associated with the source model further includes: sampling weights associated with the nodes in the current layer among the weights in the first matrix; determining the first set of weights based on the sampled weights associated with one of the nodes in the current layer; forming a first intermediate matrix based on the first set of weights and the first matrix. The determining of the second set of weights based on the first set of weights is based on the intermediate matrix.

[0007] In another exemplary embodiment, the determining the second set of weights based on the first set of weights further comprises: sampling the weights associated with the nodes in the upper layer from the weights in the first intermediate matrix; and determining the second set of weights based on the sampled weights associated with one of the nodes in the upper layer. The forming the second matrix associated with the target model based on the first set of weights and the second set of weights further comprises: forming the second matrix based on the first intermediate matrix and the second set of weights.

[0008] In another exemplary embodiment, the current layer is included in a multi-head attention (MHA) module in a converter network, and the nodes in the current layer are neurons of multiple attention heads.

[0009] In another exemplary embodiment, a third matrix includes weights associated with connections between the nodes in the upper layer and the nodes in the third layer in the source model. The third layer is a layer below the upper layer. The method further includes: sampling weights associated with the nodes in the upper layer among the weights in the third matrix; determining a third set of weights based on the sampled weights associated with one of the nodes in the upper layer; and forming a second intermediate matrix based on the third set of weights and the third matrix.

[0010] In another exemplary embodiment, the determining the second set of weights based on the first set of weights further comprises: sampling the weights associated with the nodes in the third layer from the weights in the second intermediate matrix; and determining the second set of weights based on the sampled weights associated with one of the nodes in the third layer. The forming the second matrix associated with the target model based on the first set of weights and the second set of weights further comprises: forming the second matrix based on the first intermediate matrix and the second set of weights.

[0011] In another exemplary embodiment, the method further includes: f) generating multiple copies of the second matrix by copying the second matrix multiple times. The initializing the target model based on the second matrix further includes: initializing the target model using the multiple copies of the second matrix.

[0012] In another exemplary embodiment, the method further comprises: obtaining the multiple copies of the second matrix of the target model conforming to the target dimension by performing multiple iterations of a), b), c) and f).

[0013] In another exemplary embodiment, the first matrix is ​​associated with one of the plurality of modules in the source model. The method further comprises: forming a second matrix associated with each of the other modules in the plurality of models in the source model by performing a plurality of iterations from a) to c).

[0014] In another exemplary embodiment, the trained target model is used as the second source model to initialize the second target model.

[0015] In another exemplary embodiment, the training of the target model further comprises: determining a plurality of sub-models based on the target model; updating a plurality of layers in the target model by training the plurality of sub-models; and training the target model to update the plurality of layers in the target model. The target model comprises the plurality of layers, and each sub-model is used to update a subset of the layers in the plurality of layers in the target model.

[0016] In another exemplary embodiment, updating the multiple layers in the target model by training the multiple sub-models also includes: sampling the multiple sub-models; training the sampled sub-models through a training data set; and updating a corresponding subset of the layers in the multiple layers in the target model.

[0017] In another exemplary embodiment, each of the plurality of sub-models includes all or part of the plurality of layers in the target model. The subset of layers in the corresponding sub-model is part or all of the layers in the corresponding sub-model. The training of the sampled sub-model by the training data set further includes: calculating the training loss by all the layers of the layers in the corresponding sub-model based on the training data set; and updating the subset of the layers in the corresponding sub-model based on the training loss.

[0018] In another exemplary embodiment, the present disclosure provides a system for model training. The system includes one or more processors and a non-transitory computer-readable medium storing computer-executable instructions. When the computer-executable instructions are executed by the one or more processors, the one or more processors facilitate: a) determining a first set of weights based on a first matrix associated with a source model; b) determining a second set of weights based on the first set of weights; c) forming a second matrix associated with a target model based on the first set of weights and the second set of weights; d) initializing the target model based on the second matrix; e) training the target model.

[0019] In another exemplary embodiment, the first matrix includes weights associated with connections between nodes in the current layer and nodes in the upper layer in the source model. The determining of the first set of weights based on the first matrix associated with the source model further includes: sampling weights associated with the nodes in the current layer among the weights in the first matrix; determining the first set of weights based on the sampled weights associated with one of the nodes in the current layer; forming a first intermediate matrix based on the first set of weights and the first matrix. The determining of the second set of weights based on the first set of weights is based on the intermediate matrix.

[0020] In another exemplary embodiment, the determining the second set of weights based on the first set of weights further comprises: sampling the weights associated with the nodes in the upper layer from the weights in the first intermediate matrix; and determining the second set of weights based on the sampled weights associated with one of the nodes in the upper layer. The forming the second matrix associated with the target model based on the first set of weights and the second set of weights further comprises: forming the second matrix based on the first intermediate matrix and the second set of weights.

[0021] In another exemplary embodiment, a third matrix includes weights associated with connections between the nodes in the upper layer and the nodes in the third layer in the source model. The third layer is a layer below the upper layer. The one or more processors further facilitate: sampling, among the weights in the third matrix, weights associated with the nodes in the upper layer; determining a third set of weights based on the sampled weights associated with one of the nodes in the upper layer; and forming a second intermediate matrix based on the third set of weights and the third matrix.

[0022] In another exemplary embodiment, the determining the second set of weights based on the first set of weights further comprises: sampling the weights associated with the nodes in the third layer from the weights in the second intermediate matrix; and determining the second set of weights based on the sampled weights associated with one of the nodes in the third layer. The forming the second matrix associated with the target model based on the first set of weights and the second set of weights further comprises: forming the second matrix based on the first intermediate matrix and the second set of weights.

[0023] In another exemplary embodiment, the training of the target model further comprises: determining a plurality of sub-models based on the target model; updating a plurality of layers in the target model by training the plurality of sub-models; and training the target model to update the plurality of layers in the target model. The target model comprises the plurality of layers, and each sub-model is used to update a subset of the layers in the plurality of layers in the target model.

[0024] In another exemplary embodiment, the present disclosure provides a non-transitory computer-readable medium storing processor-executable instructions for model training. When the computer-executable instructions are executed by one or more processors, the one or more processors facilitate: a) determining a first set of weights based on a first matrix associated with a source model; b) determining a second set of weights based on the first set of weights; c) forming a second matrix associated with a target model based on the first set of weights and the second set of weights; d) initializing the target model based on the second matrix; e) training the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The system and method for data processing are described in detail below with reference to the accompanying drawings, in which:

[0026] Figure 1A An exemplary network environment according to one or more examples of the present disclosure is shown.

[0027] Figure 1BAn exemplary computer system according to one or more examples of the present disclosure is shown.

[0028] Figure 2A An exemplary BERT model architecture according to one or more examples of the present disclosure is shown.

[0029] Figure 2B An exemplary process of training a target model based on a trained source model according to one or more examples of the present disclosure is shown.

[0030] Figure 3A is an exemplary process of expanding a first parameter matrix associated with a source model to generate a second parameter matrix associated with a target model.

[0031] Figure 3B is an exemplary process of expanding a first parameter matrix associated with a source model to generate a second parameter matrix associated with a target model.

[0032] Figure 4 It is an exemplary process of training a target model based on a source model according to one or more examples of the present disclosure.

[0033] Figure 5 It is a flowchart for expanding various functional modules in the BERT model through FPI according to one or more examples of the present disclosure.

[0034] Figure 6 According to one or more examples of the present disclosure, Figure 5 Flowchart showing the expansion of each functional module in the BERT model.

[0035] Figure 7 An exemplary process of training a target model by implementing two-stage training according to one or more examples of the present disclosure. DETAILED DESCRIPTION

[0036] The embodiments of the present disclosure provide an efficient training method that can be applied to various scenarios to train large-scale PLM to perform various tasks, such as search engines, dialogue systems, advertising, and machine translation applications. The method of the present disclosure can solve the problem that the pre-training process of large-scale PLM is computationally expensive and generates a large carbon footprint by significantly reducing the computational cost of the training process. Therefore, the method of the present disclosure can provide great value for a wide range of machine learning applications (e.g., NLP).

[0037] In one embodiment, a method is provided for initializing a target model by knowledge learned from a trained smaller model. In the present disclosure, the size of the model is associated with the number of weights in the layer of the corresponding model. The method can be used to expand the weight matrix learned from the smaller model in width and / or depth so that the expanded weight matrix can provide enough weights to initialize the target model. In one example, the method can expand the knowledge of each layer in the smaller model based on the knowledge of the current layer. In another example, the method can expand the knowledge of each layer in the smaller model based on the knowledge of multiple layers. For example, the multiple layers may include a current layer and an upper layer with advanced knowledge (e.g., a layer after the current layer).

[0038] In another embodiment, another method for training a target model is provided. The method can be used to train the target model in multiple stages. First, the method can be used to generate multiple sub-models based on the target model and train the multiple sub-models, thereby reducing the computational complexity of training each sub-model. In some cases, the method can utilize the power of parallel computing to train the sub-models in parallel. Second, the method can utilize the weights learned from the sub-models to train the target model.

[0039] Previous work has explored techniques for efficiently pre-training neural network models. Some work proposes to accelerate pre-training through progressive learning, which is due to the fact that different layers have some similar knowledge (e.g., attention patterns). This approach first pre-trains a smaller model with fewer converter layers, and then iteratively expands the model by stacking already trained layers, so that a larger model with extended depth can be obtained. However, this approach is limited to the depth-wise expansion of the model. Some other work proposes to "back distill" the knowledge in the smaller model into the large model. This approach is also called knowledge inheritance, that is, using the knowledge of the smaller model to update some parameters / weights in the large model. Some other work focuses on data efficiency and records rarely used words during pre-training so that the model can understand the rarely used words that appear next time through learning. In another work, the Enhanced Light Efficiency Cophasing Telescope Resolution Actuator (ELECTRA) model performs a replacement token detection task to predict whether each token in the input is replaced, thereby improving pre-training efficiency.

[0040] The present disclosure provides a technology that can efficiently train a neural network model and improve cost-effectiveness. It can be understood by those skilled in the art that the technology described in the present disclosure can be performed alone or in combination with existing technologies to improve the training process.

[0041] Figure 1A An exemplary network environment 100 according to one or more examples of the present disclosure is shown. Machine learning techniques implementing the framework disclosed herein may be performed in the exemplary network environment 100. A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices 120, servers 130, and / or other device types.

[0042] The components of the network environment can communicate with each other through one or more networks 110, which can be wired, wireless, or both. For example, the network 110 can include one or more wide area networks (Wide Area Network, "WAN"), one or more local area networks (Local Area Network, "LAN"), one or more public networks (such as the Internet) and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, access points, or other components can provide wireless connections.

[0043] Compatible network environments may include one or more peer-to-peer network environments (in which case, the network environment may not include a server) and one or more client-server network environments (in which case, the network environment may include one or more servers). In a peer-to-peer network environment, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0044] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers (which may include one or more core network servers and / or edge servers). The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or one or more applications may include web-based service software or applications, respectively. In an embodiment, one or more of a plurality of client devices may use web-based service software or applications (e.g., accessing service software and / or applications through one or more application programming interfaces ("APIs")). The framework layer may be, but is not limited to, a free and open source software web application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0045] A cloud-based network environment can provide cloud computing and / or cloud storage that can perform any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these various functions can be distributed across multiple locations of a central server or core server (e.g., one or more data centers that may be distributed across a state, a region, a country, the world, etc.). A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0046] One or more client devices 120 may include Figure 1B The client device 120 may be implemented as a personal computer (“PC”), a laptop, a mobile device, a smart phone, a tablet, a virtual reality headset, a video player, a video camera, a vehicle, a virtual machine, a drone, a robot, a handheld communication device, a vehicle-mounted computer system, an embedded system controller, a workstation, an edge device, any combination of the devices described, or any other suitable device.

[0047] Figure 1B 1 is a block diagram of an exemplary computer system 150 for implementing various functions according to one or more embodiments of the present disclosure. In some examples, the computer system 150 may be configured as follows: Figure 1A The network environment 100 is shown as being implemented in a client device 120 or a server 130. One or more computing systems 150, one or more client devices 120, one or more servers 130, or a combination thereof may form a processing system that performs the processes in the present disclosure.

[0048] like Figure 1BAs shown, the computer system 150 may include one or more processors 160, a communication interface 170, a memory 180, and a display 190. The one or more processors 160 may be used to perform operations according to instructions stored in the memory 180. The one or more processors 160 may include any appropriate type of general or special microprocessor (e.g., CPU or GPU, respectively), a digital signal processor, a microcontroller, etc. The memory 180 may be used to store computer-readable instructions, which, when executed by one or more processors 160, may cause one or more processors 160 to perform various operations disclosed herein. The memory 180 may be any non-transitory type of mass storage, such as volatile or non-volatile, magnetic, semiconductor-based, tape-based, optical, removable, non-removable or other types of storage devices or tangible computer-readable media, including but not limited to read-only memory (read-only memory, "ROM"), flash memory, dynamic random access memory (random-access memory, "RAM") and / or static RAM. The various processes / flowcharts described in the present disclosure as mathematical operations may be implemented by instructions stored in the memory 180 when executed by one or more processors 160.

[0049] Communication interface 170 can be used to communicate between computer system 150 and other devices or systems (e.g., Figure 1A The communication interface 170 may be used to transmit information between the client device 120 and / or the server 130 shown in the figure. For example, the communication interface 170 may include an integrated services digital network ("ISDN") card, a cable modem, a satellite modem, or a modem to provide a data communication connection. For another example, the communication interface 170 may include a local area network ("LAN") card to provide a data communication connection to a compatible LAN. For another example, the communication interface 170 may include a high-speed network adapter, such as a fiber optic network adapter, a 10G Ethernet adapter, etc. A wireless link may also be implemented through the communication interface 170. In this implementation, the communication interface 170 may send and receive electrical signals, electromagnetic signals, or optical signals carrying digital data streams representing various types of information through the network. The network may typically include a cellular communication network, a wireless local area network ("WLAN"), a wide area network ("WAN"), etc.

[0050] The communication interface 170 may also include various I / O devices, such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, a camera, a biosensor, etc. A user may input data to the computer system 150 (eg, a terminal device) through the communication interface 170 .

[0051] The display 190 may be integrated as part of the computer system 150, or provided as a separate device communicatively coupled to the computer system 150. The display 190 may include a display device such as a liquid crystal display (LCD), a light emitting diode display (LED), a plasma display, or any other type of display, and provide a graphical user interface (GUI) for user input and data depiction presented on the display. In some embodiments, the display 190 may be integrated as part of the communication interface 170.

[0052] The application of these methods can be extended to any suitable type of deep neural network (DNN) model. A DNN model includes multiple layers of interconnected nodes (e.g., perceptrons, neurons, etc.). These nodes can be trained with a large amount of input data to quickly solve complex problems with high accuracy. The first layer in a DNN model that receives input to the DNN model is called the input layer. The last layer in a DNN model that produces the output of the DNN model is called the output layer. Any layer between the input layer and the output layer of the DNN model is called a hidden layer. Parameters / weights associated with the DNN model can be stored in the memory 180 of the processing system in the form of a data structure.

[0053] The goal is to speed up and improve the cost-effectiveness of pre-training the target model by leveraging the knowledge of the pre-trained source model. The dimensionality of the target / source model can be represented by two variables L and D. L is the number of layers (e.g., transformer layers) in the corresponding model, and D is the width of the model, which represents the hidden size, i.e., the number of features in the hidden states of the network. Therefore, the target model can be represented by Indicates that the source model can be used Indicates that the dimension of the source model is smaller than that of the target model, that is, L s ≤L t , and D s ≤D t This goal can be achieved by doing the following: (1) Based on the source model Initialize the target model with knowledge (2) Multi-stage pre-training process, or a combination thereof.

[0054] The following describes an exemplary framework and illustrates an exemplary process for training a DNN model by implementing the techniques disclosed in the present disclosure. It should be noted that the framework and process are for illustrative purposes only and are not intended to limit the present disclosure. Those skilled in the art will appreciate that the framework and process can be extended to any suitable neural network model in any suitable machine learning application.

[0055] In this example, the processing system can be used to train a BERT model to process text content. The input to the BERT model can be a sequence of tokens that represent elements in the input text content. For example, a token can be an instance of a sequence of characters (e.g., associated with a word) in a sentence that are grouped together for processing as a useful semantic unit.

[0056] Figure 2A An exemplary BERT model architecture 200 is shown. Both the source model and the target model can have Figure 2A The source model / target model may be stored in a memory 180 in a processing system, which may include a processor such as a processor. Figure 1B One or more computer systems 150 shown. One or more computer systems 150 in the processing system can be implemented as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof in the network environment 100 shown. One or more processors 160 in the processing system can execute instructions stored in the memory 180 to perform operations on the source model / target model stored in the memory 180. The BERT model may include an embedding layer 210, multiple converter layers 220, and a classifier layer 240. The embedding layer 210 converts each token into a fixed-length vector of a defined size. The elements in the vector indicate the correlation between the corresponding token and other tokens in the batch, so that the knowledge of the vector in the batch is limited by the size of the vector. That is, the generated vector represents the corresponding token after dimensionality reduction. The vector associated with the token can form a matrix called the embedding matrix W E The weight matrix of , where E represents the embedding layer. The embedding matrix W E The elements in are called weights / parameters and are used to measure the degree of relevance between tokens. The vector associated with a token can correspond to the embedding matrix W E The rows / columns in .

[0057] A normalization layer may be provided after the embedding layer 210 to generate a hidden state based on the output of the embedding layer. Upon initialization, the normalization layer may generate an initial value of the hidden state, denoted by H0. The hidden state may be iteratively processed by the transformer layer 220 as follows:

[0058] H l=Transformer l (H l-1 ),l∈[1,L](Equation 1)

[0059] Wherein, L represents the number of converter layers 220. Figure 2A As shown, each transformer layer may include a multi-head attention (MHA) module 222, a feed-forward network (FFN) 232, and one or more other modules (e.g., 224 and 234) to perform summation and / or normalization to generate an aggregated output of the previous module / layer.

[0060] The MHA module 222 may include multiple parallel attention heads (i.e., attention mechanisms). The BERT model may use the MHA module 222 to learn relationships between tokens. For example, each token may focus on different aspects of other tokens through multiple parallel attention heads in the MHA module 222 in each converter layer. Therefore, the BERT model may capture extensive relationships between tokens through multiple converter layers 220. By increasing the number of attention heads in the MHA module 222 in each converter layer, the knowledge of the BERT model may be expanded. The attention heads implemented in the BERT model may be of various types, including but not limited to self-attention heads. Self-attention (head), also known as intra-attention, is an attention mechanism that links different positions of a single sequence (e.g., words / tokens in a sentence) in order to compute representations of the same sequence. Self-attention has been shown to be very useful in machine reading, abstract summarization, or image description generation. To illustrate in this example, the BERT model includes a self-attention head in the MHA module 222.

[0061] The hidden state H from the previous layer can be l-1 Each of the self-attention heads in the MHA module 222 is fed into the i-th attention head with three parameter vectors (Q i ,K i ,V i ) indicates that, among which, Q i represents the query, K i represents the key, V i The query, key, and value can be taken as the input hidden state H by applying the following formula l-1 The linear projection is calculated:

[0062]

[0063] Where l represents the lth transformer layer, and is a matrix of learnable weights associated with the lth transformer layer and the ith attention head. The context-aware vector can be obtained by computing the scaled dot product of the query and key in the ith attention head, which can be used to compute the final output of the ith attention head using the following equation:

[0064]

[0065] in, represents the parameter matrix associated with the lth transformer layer and the ith attention head, including the learnable weights. k is the header dimension of the query and key, is the scaling factor. The softmax() function rescales the product Then, the aggregation result of the MHA module 222 can be obtained by summing all the heads in the MHA module 222 in the l-th transformer layer by applying the following equation:

[0066]

[0067] Where a is the number of self-attention heads in the MHA module 222.

[0068] like Figure 2A As shown, module 224 may also aggregate the output of MHA module 222 with the input of MHA module 222 by applying the following equation:

[0069]

[0070] The calculated result as shown in Equation 5 can be is fed into the FFN 232. The FFN 232 may include one or more layers of learnable weights that may be trained to perform other processes on the results from the MHA module 222, such as residual connections and layer normalization processes. For example, the FFN module 232 may include two linear layers and a Gaussian error linear unit (Gaussian error linear unit, "GeLU") activation function. The process performed by the FFN 232 may be represented by the following equation:

[0071]

[0072] in, and is the weight matrix including the learnable weights, and is the bias vector. Module 234 can also aggregate the output of FFN 232 with the output of MHA module 222, as expressed by the following equation:

[0073]

[0074] Both the MHA module 222 and the FFN 232 may implement layer normalization (“LN”) for stabilizing the dynamics of the hidden states in the transformer layer 220. Formally, the LN process may be represented by the following equation:

[0075]

[0076] Among them, ⊙ represents element-by-element multiplication, μ H and σ H are the statistics of the hidden state H, and are the mean and variance of H, respectively. In some examples, the transformer layer 220 in the BERT model can form multiple layers of an encoder or decoder in any suitable application. The classifier module 240 can receive the hidden state from the transformer layer 220 and then generate a classification for the token.

[0077] Figure 2B 2 shows an exemplary process 250 for training a target model based on a trained source model 255 according to one or more examples of the present disclosure. The process 250 may be performed as follows: Figure 1B The processing system shown includes one or more computer systems 150, which may be embodied as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof in the network environment 100 shown. One or more processors 160 in the processing system can execute instructions stored in the memory 180 to perform the process 250. The process 250 can be performed alone or in combination with other processes in the present disclosure. It will be appreciated by those skilled in the art that the process 250 can be performed in any suitable environment, and the blocks in the process 250 can be performed in any suitable order.

[0078] The target model may be trained on a task that is the same as or similar to the task of the trained source model 255, so that the knowledge of the trained source model 255 can be used to accelerate the learning process of the target model. In some cases, the target model may be trained on a training data set that is the same as or similar to the training data set of the trained source model 255. Figure 2B As shown, the source model 255 includes L sEncoder layers. Block 260 indicates the first step of transferring knowledge from the source model 255 to the target model for initialization. For example, the dimension of the source model can be expanded by generating additional nodes in each module / layer, generating additional layers in the source model, or a combination thereof. Thus, an initialized target model 265 can be obtained, which includes L combined with wider embedding layers and classifier layers. t Block 260 indicates a second step of training the initialized target model 265 to obtain a trained target model 275. In some variations, the width-wise expansion may be decomposed into an expansion of parameter matrices and / or vectors.

[0079] Source Model 255 may be associated with one or more parameter matrices, which may include weights associated with interconnections between nodes in adjacent layers in the source model. The parameter matrix associated with the source model may be represented as Similarly, the target model can be associated with one or more parameter matrices that can include weights associated with the interconnections between nodes in adjacent layers in the target model. The parameter matrix associated with the source model can be represented as The parameter matrix W associated with the source model can be expanded by applying width-wise expansion and / or depth-wise expansion to generate the parameter matrix U of the target model. Width-wise expansion can be used to generate a "wider" model, while depth-wise expansion can be used to stack wider models to increase the depth of the generated model. In some examples, width-wise expansion can include intra-dimensional expansion, extra-dimensional expansion, or a combination thereof. Intra-dimensional expansion is associated with an arrow connection from a node in the current layer to a node in the next layer. Extra-dimensional expansion is associated with an arrow connection from a node in the current layer to a node in the next layer. That is, W(i,j) represents a parameter element in the parameter matrix W, where i and j refer to the i-th intra-dimensional index and the j-th extra-dimensional index, respectively. In intra-dimensional expansion, an index mapping function g can be applied to the parameter matrix W. in , to generate one or more parameters in the parameter matrix U. For example, g in (i) Using the gth ... in (i) The internal parameters of the dimension (i) generate the internal parameters of the parameter matrix U in the i-th dimension. out (j) Using the gth ... out (j)-th dimensional extra-dimensional parameter generates the j-th dimensional extra-dimensional parameter of the parameter matrix U.

[0080] In one example, the processing system may use the index mapping function g in and g outThe function preserving initialization (FPI) is implemented by a set of . FPI aims to ensure that the initialized target model has the same functionality as the source model, that is, given the same input, the initialized target model produces the same output as the source model. Formally, the mapping function g in and g out It can be defined as follows:

[0081]

[0082] Where f(·) is a function that performs uniform sampling, the superscript "w" denotes the parameter matrix W associated with the source model, the superscript "u" denotes the parameter matrix U associated with the target model, and the subscripts "in" and "out" denote the in-dimension and out-dimension, respectively. In this way, a matrix with index and The additional weights of , which are associated with the nodes added to the source model. The weight expansion of the parameter matrix can be expressed as:

[0083] U=EXPN(W;g in ,g out )(Equation 8)

[0084] The weight expansion defined by Equation 8 may include the intra-dimensional expansion associated with Equations 9a and 9b and the extra-dimensional expansion associated with Equation 9c, and the weight expansion may be expressed by the following equation:

[0085]

[0086] in, is the indicator function, Yes in The value of g in (·) in (i) count. Can be used to rescale the original parameters (e.g., weights in parameter matrix W) to preserve the function preserving properties of the expanded parameter matrix (e.g., parameter matrix U). represents the intermediate parameter matrix, which is the result of applying one or more intra-dimensional expansions to the parameter matrix W.

[0087] Figure 3A is an exemplary process 300 for expanding a first parameter matrix W associated with a source model to generate a second parameter matrix U associated with a target model. The process 300 may be performed as follows: Figure 1B The processing system shown includes one or more computer systems 150, which may be embodied as Figure 1AOne or more client devices 120, one or more servers 130, or a combination thereof in the network environment 100 shown. One or more processors 160 in the processing system can execute instructions stored in the memory 180 to perform the process 300. The process 300 can be performed alone or in combination with other processes in the present disclosure. It will be appreciated by those skilled in the art that the process 300 can be performed in any suitable environment, and the blocks in the process 300 can be performed in any suitable order.

[0088] In this example, the source model has a simplified architecture, including six nodes (e.g., neurons) in three layers, namely, an input layer, a hidden layer, and an output layer. As shown in box 310, the input layer includes two nodes and takes {x1, x2} as input. The hidden layer includes two nodes and generates hidden states {h1, h2}. The output layer includes two nodes and produces {y1, y2} as output. The weighted connections associated with the nodes in the source model can be represented by a first parameter matrix W. The weights associated with the weighted connections in the source model are shown in box 310, and these weights can be elements in the first parameter matrix W. Specifically, the matrix (W l ) 312 may represent a weighted connection from two nodes in the input layer to two nodes in the hidden layer. l The elements in 312 can be indexed by column and row index Make a query.

[0089] The processing system can first use Equations 9a and 9b to transform the matrix W l 312 Apply the intra-dimensional expansion to obtain the corresponding intermediate matrix Apply the index mapping function g shown in Equation 7a in , we can calculate the matrix W l The left column of 312 is sampled to generate the intermediate matrix The other columns in the middle matrix This calculation step is equivalent to adding the node associated with the input x1 to the source model in box 310 to obtain the intermediate model in box 320. As shown in box 320, the added nodes and the added weighted connections in the intermediate model are drawn with dashed circles and dashed arrows, respectively, while only the updated or added weights are shown. , the indices associated with the added nodes are drawn with dashed grid lines.

[0090] The processing system can then use Equation 9c to calculate the intermediate matrix Apply the out-of-dimension expansion to obtain the matrix U l 332. Apply the index mapping function g shown in Equation 7b out, we can use the intermediate matrix The bottom row in is sampled to generate the matrix U l 332. This calculation step is equivalent to adding the node associated with the hidden state h2 in box 320 to the intermediate model to obtain the target model in box 330. As shown in box 330, the added nodes and the added weighted connections in the target model are drawn with dashed circles and dashed arrows, respectively, while only the updated or added weights are shown. l In block 332, the indices associated with the added nodes are plotted with dashed grid lines. In block 330, the weighted connections between the added nodes and the nodes in the output layer are also updated, in the matrix U l 332. In this way, the source model in box 310 can be expanded by applying FPI to obtain the target model in box 330. The resulting target model in box 330 can receive {x1, x2} as input to the three nodes in the input layer and produce {y1, y2} as output, thereby maintaining the functional properties of the source model in box 310.

[0091] In another example, the processing system may use the index mapping function g in and g out AKI uses not only the parameters in the current layer but also the parameters in the upper layer (the layer after the current layer) to expand the source model, thereby avoiding redundancy in the extended model and improving the convergence speed. For example, the processing system can use the current layer (W l ) and the next layer (W l+1 ) to extend the current layer in the target model. In other words, AKI can be defined as:

[0092]

[0093] Similarly, the weight expansion defined by equation 10 can include intra-dimensional expansion and extra-dimensional expansion. For FPI, the intra-dimensional expansion is similar to equations 7a and 7b, and the modified notation is:

[0094]

[0095] Out-of-dimensional expansion can be expressed by the following equation:

[0096]

[0097] In this way, the expansion matrix and Stacking is performed to construct the final matrix U l .

[0098] Figure 3B is an exemplary process 350 for expanding a first parameter matrix W associated with a source model to generate a second parameter matrix U associated with a target model. The process 350 may be performed as follows: Figure 1B The processing system shown includes one or more computer systems 150, which may be embodied as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof in the network environment 100 shown. Process 350 can be performed alone or in combination with other processes in the present disclosure. It will be appreciated by those skilled in the art that process 350 can be performed in any suitable environment, and the blocks in process 350 can be performed in any suitable order.

[0099] In this example, the source model shown in block 360 is Figure 3A The source model in the block 310 is the same as shown. Similarly, the matrix (W l ) 362 can represent the weighted connection from two nodes in the input layer to two nodes in the hidden layer. In addition, the matrix (W l+1 )364 may represent a weighted connection from two nodes in the hidden layer to two nodes in the output layer.

[0100] The processing system can first use equations 11a and 11b to transform the matrix W l 362 and matrix W l+1 364 Apply the intra-dimensional expansion to obtain the corresponding intermediate matrices and Apply the index mapping function g shown in Equation 7a in , we can calculate the matrix W l The left column of 362 is sampled to generate the intermediate matrix The other columns in the matrix W l+ 1 The right column of 364 is sampled to generate the intermediate matrix The other columns in the intermediate matrix are updated. The sampling columns in the corresponding intermediate matrix are also updated. This calculation step is equivalent to adding the node associated with the input x1 and the node associated with the hidden state h2 to the source model in box 360 to obtain the intermediate model in box 370. As shown in box 370, the added nodes and the added weighted connections in the intermediate model are drawn with dashed circles and dashed arrows, respectively, while only the updated or added weights are shown. and , the indices associated with the added nodes are drawn with dashed grid lines.

[0101] The processing system can then use Equation 12 to calculate the intermediate matrix and Apply the out-of-dimension expansion to obtain the matrix U l 382. Index mapping function g used in AKI out The intermediate matrix can be The rows in the matrix are sampled, which can provide a The knowledge provided is higher-level knowledge. In this example, the bottom row is sampled to generate the matrix U l 382. This calculation step is equivalent to adding other weighted connections in box 380 to connect the nodes added in the hidden layer with the nodes in the input layer to obtain the target model. As shown in box 380, the added weighted connections in the target model are drawn with dashed arrows, while only the added weights are shown. l In 382, ​​the indices associated with the added weighted connections are drawn with dashed grid lines. In this way, the source model in box 360 can be expanded by applying AKI to obtain the target model in box 380. Figure 3A The target model in the block 330 shown is different. When the target model in the resulting block 380 receives {x1, x2} as input to the three nodes in the input layer, the target model in the block 380 generates a different set of hidden states {h1, h2, h ′ 2} and produces {y1,y2 ′ In other words, the target model in block 380 does not maintain the functional properties of the source model in block 360.

[0102] Experiments show that adjacent converter layers have similar functions, ensuring that the knowledge contained in the parameters (e.g., weights) in the current layer may not be "destroyed" because the parameters are coupled with parameters from adjacent layers. That is, the new current layer does not produce outputs that deviate significantly from the original current layer. Therefore, AKI can provide efficient knowledge transfer from the source model to the target model. In addition, AKI can provide other benefits. First, the knowledge from adjacent layers can break the symmetry caused by FPI in the target model, thereby increasing the convergence speed of the target model. For example, FPI may cause repeated attention patterns in the same layer, which is redundant and is called symmetry. Secondly, when the upper layer is model expanded through AKI, the corresponding upper layer information provides knowledge similar to but more advanced (i.e., higher-level) than the knowledge provided by the current layer, thereby guiding the target model to converge faster.

[0103] FPI can ensure that the initialized target model has almost the same behavior as the source model, thus providing a good starting point for the later optimization of the target model. On the other hand, AKI does not follow the function preservation principle like FPI, but can still provide a good starting point for later optimization, which is also supported by empirical results. In addition, AKI can speed up convergence and achieve higher training efficiency. The processing system can implement FPI and AKI alone or in combination to initialize the target model based on the source model. In addition, KPI and / or AKI can be combined with other techniques to further improve the training process.

[0104] Figure 4 4 is an exemplary process 400 for training a target model based on a source model according to one or more examples of the present disclosure. The process 400 may be performed as follows: Figure 1B The processing system shown includes one or more computer systems 150, which may be embodied as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof in the network environment 100 shown. One or more processors 160 in the processing system can execute instructions stored in the memory 180 to perform the process 400. The process 400 can be performed alone or in combination with other processes in the present disclosure. It will be appreciated by those skilled in the art that the process 400 can be performed in any suitable environment, and the blocks in the process 400 can be performed in any suitable order.

[0105] The source model may include multiple layers of many connected nodes. The connection of nodes in a pair of adjacent layers may be represented by a first matrix, wherein each element in the first matrix may be associated with an arrow connection between two nodes in the pair of adjacent layers.

[0106] At box 410, the processing system determines a first set of weights based on a first matrix associated with the source model. The processing system can apply intra-dimensional expansion in the step of determining the first set of weights, for example, by applying Equation 7a. The first set of weights can be associated with one or more first nodes added to the first layer of the source model. Each added first node can be connected to a node in an adjacent upper layer (e.g., a layer below the first layer), wherein the first weight associated with the corresponding added first node represents a weighted connection between the first nodes. The processing system can determine the first weight associated with the first node added in the first layer of the model based on the weighted connection of another node in the same layer. For example, in Figure 3A In the example, the processing system may determine the matrix based on the weights associated with the existing nodes associated with the input "x1" shown in block 320. The processing system may calculate the first weight by applying equations 9a and 9b. In some variations, the processing system may calculate the first weights of the first nodes added to different first layers in the source model in parallel. For example, the processing system may apply equations 11a and 11b to calculate the first weights associated with different layers in the source model.

[0107] Return to reference Figure 4 , at box 420, the processing system determines a second set of weights based on the first set of weights. The processing system may apply out-of-dimensional expansion in the step of determining the second set of weights. The second set of weights may be associated with one or more second nodes added in an adjacent upper layer of the current layer in the source model. Each added second node may be connected to a node in the first layer via a weighted connection based on a second weight associated with the corresponding added second node. In one example, the processing system may determine a second weight associated with a second node added in the second layer of the model based on a weighted connection between another node in the adjacent upper layer and a node in the first layer. Figure 3A As shown, the processing system may determine the matrix U based on the weights associated with the existing nodes associated with the hidden state "h2" shown in block 330. l The other rows in 332 are the weights of the added second node associated with the hidden state "h2".

[0108] In another example, the processing system may calculate the first weights of the first nodes of different first layers added to the model in parallel. Figure 3B As shown, the processing system may determine the matrix based on the weights associated with the existing nodes associated with the input "x1" shown in block 370. At the same time, the processing system can determine the matrix based on the weights associated with the existing nodes associated with the hidden state "h2" shown in box 370. The other columns of the weights of the added first node associated with the hidden state "h2" in the block 380. In this example, layer l can be defined as the current layer. Accordingly, layer (l+1) is the adjacent upper layer of the current layer, and when the current layer is extended out of dimension, the node added in the hidden layer is regarded as the second node. In addition, layer (l+2) is the adjacent upper layer of layer (l+1). The processing system can be based on the adjacent upper layer of layer (l+1) shown in block 380 that is associated with the output "y2 ′ "The weights associated with the existing nodes determine the matrix U l 382 and the hidden state "h ′ 2” and associated with the other rows of weights for the added second nodes. In this way, the processing system can determine a second set of weights.

[0109] Return to see Figure 4 At block 430, the processing system forms a second matrix associated with the target model based on the first set of weights and the second set of weights. In some examples, the processing system may first obtain a widened model based on the source model by applying blocks 410 to 430 of process 400, and then iteratively stack the widened model to obtain the target model.

[0110] The processing system may repeat some or all of blocks 410 to 430 for multiple iterations to expand the second matrix to the target dimension of the target model, e.g., from the source model Extension to target model

[0111] At block 440 , the processing system initializes the target model based on the second weight matrix.

[0112] At block 450, the processing system trains the target model. In some variations, the processing system may initialize a second target model by repeating process 400 using the trained target model as a second source model.

[0113] The process of the present disclosure can be performed to expand various functional modules in the neural network model. The following example shows that Figure 2A The above FPI / AKI is implemented in the BERT model of the shown architecture to extend different modules.

[0114] Figure 5 FIG. 5 is a flowchart 500 for extending various functional modules in a BERT model through FPI according to one or more examples of the present disclosure. The flowchart 500 may be as follows: Figure 1B The processing system shown includes one or more computer systems 150, which may be embodied as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof are shown in the network environment 100. One or more processors 160 in the processing system may execute instructions stored in the memory 180 to perform the flowchart 500.

[0115] In this example, the model on the left is the source model, the model in the middle is the intermediate model, and the model on the right is the target model. The BERT model may include multiple functional modules, such as the networks shown in boxes 510, 520, and 530. Each functional module may include multiple layers of multiple connected neurons (i.e., nodes). For example, from bottom to top, box 510 of the source model includes three embedding neurons in the lowest layer, two neurons in each of the two hidden layers, and two neurons of two different attention heads in the highest layer. Box 520 of the source model includes two neurons of two different attention heads in the lowest layer, two neurons in each of the two hidden layers, and two FFN neurons in the highest layer. Box 530 of the source model includes two FFN neurons in the lowest layer, two neurons in each of the two hidden layers, and three embedding neurons in the highest layer. The intermediate model and the target model may have similar architectures to the source model, but with different dimensions. The weights associated with the arrow connections between neurons in adjacent layers are shown in FIG. Figure 5 In addition, the weights associated with the arrow connections between neurons in a pair of adjacent layers can be associated with a parameter / weight matrix, e.g., W EMB , W LN wait.

[0116] like Figure 5 As shown, from the source model to the intermediate model, the processing system can expand the hidden layer in blocks 510, 520, and 530. Thus, the processing system can add other neurons 512 and 514 in block 510, other neurons 522 and 524 in block 520, and other neurons 532 and 534 in block 530. From the intermediate model to the target model, the processing system can expand the neuron layer of the attention head in block 540, and can expand the FFN neuron layer in block 550. Similarly, the processing system can add other neurons for other attention heads in block 540, and add other FFN neurons in block 550. Figure 2A As shown in the exemplary BERT model architecture 200 in FIG. 1 , the BERT model may include multiple converter layers 220, wherein each converter layer may include an MHA module 222 and a FFN 232. In this example, box 540 may be associated with the MHA module 222, box 550 may be associated with the FFN 232, and both boxes 540 and 550 may be included in one converter layer. Figure 5 The embodiments may be applied to other converter layers in the plurality of converter layers, which are not shown in this figure for simplicity.

[0117] The updated weights in the intermediate / target model can be determined by applying the above-mentioned in-dimensional and / or out-dimensional expansion to the corresponding parameter / weight matrix. In this example, the processing system can determine the updated weights in the intermediate / target model by applying the following equation and based on the original embedding matrix W E To calculate the extended embedding matrix U E :

[0118]

[0119] Among them, only out-of-dimensional expansion is applied.

[0120] The processing system can calculate the head extension by applying the following equation:

[0121]

[0122] Thus increasing the number of attention heads. By applying head-wise expansion, the existing head group parameters can be reused to construct a new matrix. For example, the i-th head group in the l-th layer may include the parameter / weight matrix as described in Equations 2a to 2c and Equation 3 Accordingly, the matrix The out-of-dimensional expansion of can be expressed by the following equation:

[0123]

[0124] Among them, j is the index of the attention head, a s|t represents the corresponding number of attention heads in the source model / target model. The extension of the MHA module can apply three constraints, expressed as: The first two constraints can be used to keep the hidden layer dimensions consistent, while the third constraint can be used for seamless residual connections.

[0125] For the FFN module, the processing system can calculate the parameter matrix W using the following equation 1|2 Extensions:

[0126]

[0127] Similar to the MHA module, the extension of the FFN module can apply three constraints, expressed as:

[0128] For layer normalization, the processing system can calculate the parameter matrices W in different modules by applying only the out-of-dimensional expansion LN For example, the expansion of layer normalization in the FFN module can be obtained by equation Calculation. Equation 6 above defines the process of layer normalization, where the mean μ and variance σ are calculated based on the hidden representation in the matrix H. Therefore, the parameter matrix W LNThe expansion of inevitably leads to a gap, which prevents the target model from strictly following the function preservation principle. However, empirical results show that the gap is very small and hardly affects the initialization and convergence of the target model. In fact, the initialized target model can achieve almost the same loss as the source model, thus successfully preserving the knowledge of the source model.

[0129] Figure 6 According to one or more examples of the present disclosure, Figure 5 600 for expanding the functional modules in the BERT model shown in FIG. 600 can be as follows: Figure 1B The processing system shown includes one or more computer systems 150, which may be embodied as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof are shown in the network environment 100. One or more processors 160 in the processing system may execute instructions stored in the memory 180 to perform the flowchart 600.

[0130] In this example, other converter layers for extending the current transformer layer are proposed. The current transformer layer is the l-th converter layer, and the other converter layers are the layers above the current transformer layer, i.e., the (l+1)-th converter layer. The l-th converter layer may include a block 630 associated with the MHA module 222 and a block 640 associated with the FFN 232. Similarly, the (l+1)-th converter layer may include a block 610 associated with the MHA module 222 and a block 620 associated with the FFN 232.

[0131] like Figure 6 As shown, the middle model in the middle shows the source model (i.e. Figure 5 The extended hidden layers in blocks 510, 520 and 530 in the l-th transformer layer in the left model in the source model. Similar extensions can be applied to the (l+1)-th transformer layer in the source model, resulting in Figure 6 The (l+1)th transformer layer shown on the left is also included in the intermediate model. From the intermediate model to the target model (i.e. Figure 6In the example of FIG. 6 , the processing system may expand the neuron layer of the attention head in box 630 and may expand the FFN neuron layer in box 640 based on the right model in the intermediate model. Therefore, the processing system may add other neurons for other attention heads in box 630 and add other FFN neurons in box 640. Other arrow connections in the l-th converter layer in the target model may be determined based on existing arrow connections in the (l+1)-th converter layer in the intermediate model. For example, arrow connections associated with other neurons of other attention heads in box 630 in the target model may be determined based on arrow connections associated with left / right neurons of corresponding attention heads in box 610 in the l-th converter layer in the intermediate model. Arrow connections associated with other FFN neurons in box 640 in the target model may be determined based on arrow connections associated with left / right FFN neurons in box 620 in the l-th converter layer in the intermediate model.

[0132] The processing system may apply the same dimensional out-expansion (i.e., Equation 13) to the embedded matrix as described in flowchart 500. The processing system may calculate the expanded matrices for the MHA module and the FFN module by applying equations 11a, 11b, and 12. The constraints of the mapping function may follow the settings in flowchart 500, which are associated with the FPI.

[0133] In some embodiments, the processing system may perform depth-wise expansion to increase the depth of the expanded model to reach the target depth of the target model. Various techniques may be used. For example, the processing system may iteratively stack certain layers in the model that are expanded by width-wise expansion. In other words, depth-wise expansion may be performed after width-wise expansion by copying certain widened layers to meet the target depth of the target model. The bottom layer in the model may be used for copying. Table 1 shows an exemplary algorithm for implementing depth-wise expansion, which may be executed by one or more processors 160 in the processing system by executing instructions stored in memory 180.

[0134] Table 1: Exemplary algorithm for target model initialization

[0135]

[0136]

[0137] In some embodiments, a two-stage training strategy can be implemented to further accelerate the training process of the target model. Figure 7 700 is an exemplary process for training a target model by implementing two-stage training according to one or more examples of the present disclosure. The process 700 may be performed as follows: Figure 1BThe processing system shown includes one or more computer systems 150, which may be embodied as Figure 1A One or more client devices 120, one or more servers 130, or a combination thereof in the network environment 100 shown. One or more processors 160 in the processing system can execute instructions stored in the memory 180 to perform the process 700. The process 700 can be performed alone or in combination with other processes in the present disclosure. It will be appreciated by those skilled in the art that the process 700 can be performed in any suitable environment and in any suitable order.

[0138] At block 710, the processing system initializes the target model. The processing system may determine initial weights (or parameters) of the target model using the source model by performing any of the processes of the present disclosure.

[0139] At block 720, the processing system trains a plurality of sub-models based on the target model to update a plurality of weights in the target model. The processing system may first generate a plurality of sub-models based on the target model. Each of the plurality of sub-models may include a subset of layers in the target model. During training, all or part of the layers in the subset of layers of the corresponding sub-model may be updated. The plurality of sub-models may be trained to update different layers in the target model.

[0140] In some cases, the processing system may train multiple sub-models in a random manner so that different layers in the target model can be randomly updated, thereby achieving full coverage of the target model at a low cost. For example, in an exemplary BERT model with nine converter layers, three sub-models can be constructed using different numbers of converter layers, and a classification layer located above the converter layer can be shared. The first sub-model may include the bottom three converter layers of the nine converter layers, the second sub-model may include the bottom six converter layers of the nine converter layers, and the third sub-model may include nine converter layers. In each optimization (i.e., training) step, the processing system may randomly sample sub-models from the three sub-models. When sampling the first sub-model, the processing system may update the three converter layers and the classification layer in the first sub-model. When sampling the second sub-model, the processing system may use all six converter layers to calculate the training loss, and then based on the calculated training loss, the top three converter layers and the classification layer in the second sub-model may be updated. When sampling the third sub-model, the processing system may use all nine converter layers to calculate the training loss, and then based on the calculated training loss, the top three converter layers and the classification layer in the third sub-model may be updated.

[0141] At block 730 , the processing system trains the target model to update a plurality of weights of the target model.

[0142] Table 2 shows an exemplary algorithm for implementing two-stage training, which may be performed by one or more processors 160 in the processing system by executing instructions stored in the memory 180 .

[0143] Table 2: Example algorithm for two-stage pre-training

[0144]

[0145]

[0146] Additional details and advantages related to exemplary embodiments of the present disclosure are discussed in Appendix A.

[0147] It should be noted that the techniques described herein may be embodied in executable instructions stored in a computer-readable medium for use by a processor-based instruction execution machine, system, device or apparatus or in combination with a processor-based instruction execution machine, system, device or apparatus. It will be appreciated by those skilled in the art that for some embodiments, various types of computer-readable media may be included to store data. As used herein, "computer-readable media" includes one or more of any suitable media for storing executable instructions of a computer program, so that an instruction execution machine, system, device or apparatus can read (or obtain) instructions from the computer-readable medium and execute these instructions to perform the described embodiments. Suitable storage formats include one or more of the following: electronic format, magnetic format, optical format, and electromagnetic format. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random-access memory (RAM); read-only memory (ROM); erasable programmable read only memory (EPROM); flash memory devices; and optical storage devices including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

[0148] It should be understood that the arrangement of the components shown in the drawings is for illustrative purposes, and other arrangements are also possible. For example, one or more of the elements described herein may be implemented in whole or in part as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. In addition, some or all of these other elements may be combined, some elements may be completely omitted, and other components may be added while still achieving the functions described herein. Therefore, the subject matter described herein may be embodied in many different variations, and it is contemplated that all of these variations are within the scope of the claims.

[0149] In order to facilitate understanding of the subject matter described herein, many aspects are described according to the order of actions. It will be appreciated by those skilled in the art that various actions can be performed by specialized circuits, by program instructions executed by one or more processors, or by a combination of the two. The description of any sequence of actions herein is not intended to imply that a specific sequence described for executing the sequence must be followed. Unless otherwise specified herein or clearly contradictory to the context, all methods / processes described herein can be performed in any suitable order.

[0150] Unless otherwise specified herein or clearly contradictory to the context, the use of the terms "one", "an" and "the" and similar references in the description of the subject matter (especially in the attached claims) should be interpreted as covering the singular and the plural. Unless otherwise specified herein or clearly contradictory to the context, the use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be interpreted as meaning to select one item from the listed items (A or B) or any combination of two or more items from the listed items (A and B). In addition, the above description is for illustrative purposes only and not for limiting purposes, because the scope of protection is defined by the claims described below and any equivalents thereof. Unless otherwise required, the use of any and all examples or exemplary language (such as "for example") provided herein is intended only to better illustrate the subject matter and does not limit the scope of the subject matter. The use of the term "based on" and other similar phrases indicating conditions that lead to a result in the claims and written description is not intended to exclude any other conditions that lead to the result. Any language in the specification should not be interpreted as indicating that any non-claimed element is essential to the implementation of the claimed invention.

Claims

1. A computer-implemented method for model training, the method being performed by a processing system and comprising: a) determining a first set of weights based on a first matrix associated with a source model; b) determining a second set of weights based on the first set of weights; c) forming a second matrix associated with a target model based on the first set of weights and the second set of weights; d) initializing the target model based on the second matrix; e) training the target model.

2. The method according to claim 1, wherein: The first matrix includes weights associated with connections between nodes in a current layer and nodes in a previous layer in the source model; The determining of the first set of weights based on the first matrix associated with the source model further comprises: Among the weights in the first matrix, sampling weights associated with the nodes in the current layer; determining the first set of weights based on the sampling weight associated with one of the nodes in the current layer; A first intermediate matrix is ​​formed based on the first weight set and the first matrix, wherein: The determining of a second set of weights based on the first set of weights is based on the intermediate matrix.

3. The method according to claim 2, wherein: The determining a second weight set based on the first weight set further comprises: sampling, among the weights in the first intermediate matrix, weights associated with the nodes in the upper layer; determining the second set of weights based on the sampling weight associated with one of the nodes in the upper layer; The forming a second matrix associated with the target model based on the first weight set and the second weight set further comprises: The second matrix is ​​formed based on the first intermediate matrix and the second set of weights.

4. The method according to claim 2, wherein: The current layer is included in a multi-head attention MHA module in a converter network, and the nodes in the current layer are neurons of multiple attention heads.

5. The method according to claim 2, wherein: The third matrix includes weights associated with connections between the nodes in the upper layer and nodes in a third layer in the source model, wherein the third layer is a layer below the upper layer; the method further comprising: Among the weights in the third matrix, sampling weights associated with the nodes in the upper layer; determining a third set of weights based on the sampling weight associated with one of the nodes in the upper layer; A second intermediate matrix is ​​formed based on the third set of weights and the third matrix.

6. The method according to claim 5, wherein: The determining a second weight set based on the first weight set further comprises: Among the weights in the second intermediate matrix, sampling the weights associated with the nodes in the third layer; determining the second set of weights based on the sampling weight associated with one of the nodes in the third layer; The forming a second matrix associated with the target model based on the first weight set and the second weight set further comprises: The second matrix is ​​formed based on the first intermediate matrix and the second set of weights.

7. The method according to claim 1, further comprising: f) generating a plurality of copies of the second matrix by replicating the second matrix multiple times; Wherein, initializing the target model based on the second matrix further includes: The target model is initialized by the multiple copies of the second matrix.

8. The method according to claim 7, further comprising: By performing multiple iterations of a), b), c) and f), the copy of the second matrix of the target model conforming to the target dimensions is obtained.

9. The method according to claim 1, wherein: The first matrix is ​​associated with one of the modules in the source model; the method further comprising: By performing a plurality of iterations from a) to c), a second matrix associated with each module in the other modules in the plurality of models in the source model is formed.

10. The method according to claim 1, wherein: The trained target model is used as a second source model to initialize a second target model.

11. The method according to claim 1, wherein: The training of the target model further comprises: Determining a plurality of sub-models based on the target model, wherein the target model includes a plurality of layers, and each sub-model is used to update a subset of layers in the plurality of layers in the target model; Updating the multiple layers in the target model by training the multiple sub-models; The target model is trained to update the plurality of layers in the target model.

12. The method according to claim 11, wherein: The updating of the multiple layers in the target model by training the multiple sub-models further comprises: Sampling the multiple sub-models; Training the sampling sub-model using a training data set; A respective subset of layers in the plurality of layers in the target model is updated.

13. The method according to claim 12, wherein: Each sub-model in the plurality of sub-models includes all or part of the plurality of layers in the target model, wherein: The subset of layers in the corresponding sub-model is part or all of the layers in the corresponding sub-model; The training of the sampling sub-model by the training data set further comprises: Based on the training data set, calculating the training loss through all layers of the layers in the corresponding sub-model; The subset of the layers in the corresponding sub-model is updated based on the training loss.

14. A system for model training, comprising: one or more processors; A non-transitory computer-readable medium storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to facilitate: a) determining a first set of weights based on a first matrix associated with a source model; b) determining a second set of weights based on the first set of weights; c) forming a second matrix associated with a target model based on the first set of weights and the second set of weights; d) initializing the target model based on the second matrix; e) training the target model.

15. The system of claim 14, wherein: The first matrix includes weights associated with connections between nodes in a current layer and nodes in a previous layer in the source model; The determining of the first set of weights based on the first matrix associated with the source model further comprises: Among the weights in the first matrix, sampling weights associated with the nodes in the current layer; determining the first set of weights based on the sampling weight associated with one of the nodes in the current layer; A first intermediate matrix is ​​formed based on the first weight set and the first matrix, wherein: The determining of a second set of weights based on the first set of weights is based on the intermediate matrix.

16. The system of claim 15, wherein: The determining a second weight set based on the first weight set further comprises: sampling, among the weights in the first intermediate matrix, weights associated with the nodes in the upper layer; determining the second set of weights based on the sampling weight associated with one of the nodes in the upper layer; The forming a second matrix associated with the target model based on the first weight set and the second weight set further comprises: The second matrix is ​​formed based on the first intermediate matrix and the second set of weights.

17. The system of claim 15, wherein: a third matrix including weights associated with connections between the nodes in the upper layer and nodes in a third layer in the source model, wherein the third layer is a layer below the upper layer; and the one or more processors further facilitate: Among the weights in the third matrix, sampling weights associated with the nodes in the upper layer; determining a third set of weights based on the sampling weight associated with one of the nodes in the upper layer; A second intermediate matrix is ​​formed based on the third set of weights and the third matrix.

18. The system of claim 17, wherein: The determining a second weight set based on the first weight set further comprises: Among the weights in the second intermediate matrix, sampling the weights associated with the nodes in the third layer; determining the second set of weights based on the sampling weight associated with one of the nodes in the third layer; The forming a second matrix associated with the target model based on the first weight set and the second weight set further comprises: The second matrix is ​​formed based on the first intermediate matrix and the second set of weights.

19. The system of claim 1, wherein: The training of the target model further comprises: Determining a plurality of sub-models based on the target model, wherein the target model includes a plurality of layers, and each sub-model is used to update a subset of layers in the plurality of layers in the target model; Updating the multiple layers in the target model by training the multiple sub-models; The target model is trained to update the plurality of layers in the target model.

20. A non-transitory computer-readable medium storing computer-executable instructions for model training, which, when executed by one or more processors, cause the one or more processors to facilitate: a) determining a first set of weights based on a first matrix associated with a source model; b) determining a second set of weights based on the first set of weights; c) forming a second matrix associated with a target model based on the first set of weights and the second set of weights; d) initializing the target model based on the second matrix; e) training the target model.

Images