Parameter efficient training method and device, and storage medium

Abstract

This application relates to the field of neural network technology, and in particular to a method, apparatus, and storage medium for efficient parameter training. The method includes: acquiring a pre-trained language model, the language model including first parameters corresponding to each of m neurons; training based on the language model and a training corpus to obtain second parameters corresponding to each of the m neurons; when a few-shot learning task is acquired, determining n neurons corresponding to the few-shot learning task; and initializing and training the parameters of the few-shot learning task based on the second parameters corresponding to each of the n neurons. This application's embodiment determines the corresponding neurons for a downstream few-shot learning task, and for each neuron, pre-trained parameters are available to activate that neuron. Therefore, using these pre-trained parameters for the few-shot learning task can significantly improve training efficiency and the ability to perform few-shot or zero-shot training on that task.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a method, apparatus and storage medium for efficient parameter training. Background Technology

[0002] Pre-training of a language model refers to training the corresponding language model using a massive amount of language sequence data, so that the language model has the ability to predict the probability of a certain language unit appearing at a certain position.

[0003] In related technologies, a parameter-efficient module is added to a pre-trained language model. When fine-tuning on downstream tasks, only the parameters of the parameter-efficient module are adjusted, while the parameters of the pre-trained language model remain unchanged. The parameter-efficient module contains only a small fraction of the parameters in the pre-trained language model. Fine-tuning these parameters can achieve the same effect as full-parameter fine-tuning. This decouples the downstream task parameters from the language model. Compared to tuning the entire language model, parameter efficiency can significantly reduce the number of parameters that need to be trained, while ensuring that task-related parameters remain a small number. This avoids the need to adapt a large language model for each task, allowing the large language model to be shared across multiple tasks.

[0004] There are currently various methods and techniques for adding parameters to efficient modules. The main difference lies in which part of the language model the parameters are added to. For example, methods that add a small number of parameters to the input layer, such as prefix tuning, are illustrated.

[0005] However, current parameter-efficient methods typically add parameters that are randomly initialized, which slows down the convergence speed of model training and results in poor model performance with few samples. Summary of the Invention

[0006] In view of this, a parameter-efficient training method, apparatus, and storage medium are proposed. In this embodiment, for a downstream few-shot learning task, n neurons corresponding to the few-shot learning task are determined. For each neuron, parameters that can activate that neuron have been pre-trained. Therefore, for the few-shot learning task, using these pre-trained parameters can significantly improve the training efficiency and the ability to perform few-shot or zero-shot training on that task.

[0007] In a first aspect, embodiments of this application provide a parameter-efficient training method, the method comprising:

[0008] Obtain a pre-trained language model, wherein the language model includes the first parameters corresponding to each of m neurons, where m is a positive integer;

[0009] The language model and training corpus are used to train the m neurons to obtain the second parameters corresponding to each of the m neurons. The second parameters are different from the first parameters.

[0010] When a few-shot learning task is obtained, n neurons corresponding to the few-shot learning task are determined, where n is a positive integer;

[0011] The few-shot learning task is initialized and trained according to the second parameters corresponding to each of the n neurons.

[0012] In this implementation, a pre-trained language model is obtained, which includes the first parameters corresponding to each of m neurons, where m is a positive integer. The language model is then trained using the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons, which are different from the first parameters. When a few-shot learning task is obtained, n neurons corresponding to the few-shot learning task are determined, where n is a positive integer. The few-shot learning task is initialized and its parameters are trained based on the second parameters corresponding to each of the n neurons. By determining the n neurons corresponding to a downstream few-shot learning task, and considering that each neuron has pre-trained second parameters that can activate it, using these pre-trained second parameters for the few-shot learning task can significantly improve the training efficiency and the ability to perform few-shot or zero-shot training on that task.

[0013] In one possible implementation, the training corpus includes training corpora corresponding to each of the m neurons. Before training based on the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons, the method further includes:

[0014] For each neuron, based on multiple first unlabeled corpora, the language model output is invoked to obtain multiple first activation values ​​corresponding to the neuron;

[0015] The training corpus corresponding to the neuron is determined based on the plurality of first activation values ​​corresponding to the neuron.

[0016] In this implementation, for each neuron, based on multiple first unlabeled corpora, the language model output is called to obtain multiple first activation values ​​corresponding to the neuron; based on the multiple first activation values ​​corresponding to the neuron, the training corpus corresponding to the neuron is determined; for example, the model's ability to perceive relevant features is activated through binary classification or multi-class classification tasks, so that a large amount of training corpus for subsequent training can be constructed for each neuron.

[0017] In another possible implementation, the step of training based on the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons includes:

[0018] For each neuron, the second parameter corresponding to the neuron is randomly initialized to obtain the initial value of the parameter;

[0019] Based on the language model and the training corpus corresponding to the neuron, the initial values ​​of the parameters are pre-trained to obtain the second parameters corresponding to the neuron.

[0020] In this implementation, for each neuron, the second parameter corresponding to the neuron is randomly initialized to obtain the initial parameter value; based on the language model and the training corpus corresponding to the neuron, the initial parameter value is pre-trained to obtain the second parameter corresponding to the neuron; unsupervised pre-training of the neuron with a small number of parameters is adopted, and the pre-training method is simple and convenient.

[0021] In another possible implementation, determining the n neurons corresponding to the few-shot learning task when a few-shot learning task is obtained includes:

[0022] When the few-shot learning task is obtained, the language model output is called to obtain the second activation value corresponding to each of the m neurons according to the few-shot learning task. The second activation value indicates the correlation between the neuron and the few-shot learning task.

[0023] Based on the second activation value corresponding to each of the m neurons, determine the n neurons corresponding to the few-shot learning task from the m neurons.

[0024] In this implementation, when a few-shot learning task is obtained, the language model output is called to obtain the second activation values ​​corresponding to each of the m neurons, which indicate the correlation between the neurons and the few-shot learning task. Based on the second activation values ​​corresponding to each of the m neurons, n neurons corresponding to the few-shot learning task are determined from the m neurons. Thus, the second parameters corresponding to the appropriate neurons are selected as the initialization of the downstream task, the few-shot learning task.

[0025] In another possible implementation, the initialization and parameter training of the few-shot learning task based on the second parameters corresponding to each of the n neurons includes:

[0026] The second parameters corresponding to each of the n neurons are combined to obtain the target parameter values;

[0027] The target value of the parameter is used as the initialization parameter to train the parameters for the few-shot learning task.

[0028] In this implementation, the second parameters corresponding to each of the n neurons are combined to obtain the target parameter value. The target parameter value is used as the initialization parameter to train the parameters for the few-shot learning task. Based on this, the few-shot learning can achieve faster convergence and higher model performance.

[0029] Secondly, embodiments of this application provide a parameter-efficient training device, the device comprising:

[0030] An acquisition unit is used to acquire a pre-trained language model, wherein the language model includes the first parameters corresponding to each of m neurons, where m is a positive integer.

[0031] The first training unit is used to train according to the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons, wherein the second parameters are different from the first parameters;

[0032] The determining unit is used to determine the n neurons corresponding to the few-shot learning task when a few-shot learning task is obtained, where n is a positive integer;

[0033] The second training unit is used to initialize and train the parameters of the few-shot learning task according to the second parameters corresponding to each of the n neurons.

[0034] In one possible implementation, the training corpus includes training corpora corresponding to each of the m neurons, and the apparatus further includes: a training corpus determination unit, configured to:

[0035] For each neuron, based on multiple first unlabeled corpora, the language model output is invoked to obtain multiple first activation values ​​corresponding to the neuron;

[0036] The training corpus corresponding to the neuron is determined based on the plurality of first activation values ​​corresponding to the neuron.

[0037] In another possible implementation, the first training unit is further configured to:

[0038] For each neuron, the second parameter corresponding to the neuron is randomly initialized to obtain the initial value of the parameter;

[0039] Based on the language model and the training corpus corresponding to the neuron, the initial values ​​of the parameters are pre-trained to obtain the second parameters corresponding to the neuron.

[0040] In another possible implementation, the determining unit is further configured to:

[0041] When the few-shot learning task is obtained, the language model output is called to obtain the second activation value corresponding to each of the m neurons according to the few-shot learning task. The second activation value indicates the correlation between the neuron and the few-shot learning task.

[0042] Based on the second activation value corresponding to each of the m neurons, determine the n neurons corresponding to the few-shot learning task from the m neurons.

[0043] In another possible implementation, the second training unit is further used for:

[0044] The second parameters corresponding to each of the n neurons are combined to obtain the target parameter values;

[0045] The target value of the parameter is used as the initialization parameter to train the parameters for the few-shot learning task.

[0046] Thirdly, embodiments of this application provide a parameter-efficient training device, the device comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to implement the method provided by the first aspect or any possible implementation of the first aspect when executing the instructions.

[0050] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method provided by the first aspect or any possible implementation of the first aspect.

[0051] Fifthly, embodiments of this application provide a computer program product comprising computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device performs the method provided by the first aspect or any possible implementation thereof. Attached Figure Description

[0052] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0053] Figure 1 A system architecture diagram provided by an exemplary embodiment of this application is shown.

[0054] Figure 2 A schematic diagram of the structure of a computing device provided in an exemplary embodiment of this application is shown.

[0055] Figure 3 A flowchart of an efficient parameter training method provided by an exemplary embodiment of this application is shown.

[0056] Figure 4 This illustration shows a schematic diagram of the activation value distribution of a neuron provided in an exemplary embodiment of this application.

[0057] Figure 5 A flowchart of an efficient parameter training method provided by another exemplary embodiment of this application is shown.

[0058] Figure 6 A block diagram of an efficient parameter training apparatus provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0059] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0060] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0061] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0062] First, some terms used in the embodiments of this application will be introduced.

[0063] 1. Language Model (LM): A mathematical model that describes the relationships between characters in a natural language or programming language.

[0064] 2. Pre-training: The initial training stage of the model using general big data.

[0065] 3. Corpus: Also known as free text, a corpus can consist of words, phrases, sentences, fragments, articles, and any combination thereof.

[0066] 4. Training sample: Data used for model training.

[0067] 5. Attention mechanism: A device for analyzing the importance of different parts within a sequence or between different parts of different sequences.

[0068] 6. Transformer: A sequence modeling neural network based entirely on attention mechanisms.

[0069] 7. Decoder: A device that converts input information into information that is easy for humans to understand.

[0070] 8. Natural language: A language that evolves naturally with culture.

[0071] 9. Machine learning: The ability of a computer to perform tasks by using data to guide it through certain algorithms.

[0072] 10. Deep learning: A type of machine learning based on deep neural networks.

[0073] 11. Downstream task: In contrast to pre-training, downstream task refers to the task corresponding to the application scenario where the pre-trained model is expected to be applied, such as question answering task, classification task, etc.

[0074] 12. Fine-tuning: refers to further training the model on the basis of the pre-trained model using data from a specific downstream task, so that the model can achieve the best performance on the corresponding downstream task.

[0075] 13. Few-shot learning: also known as few-shot learning, refers to the practice of using only a small number of downstream task samples for model training during fine-tuning, typically only a few or a few dozen samples.

[0076] Few-time learning is a novel approach proposed in the field of deep learning in recent years. It aims to explore how models can quickly adapt to new tasks. It posits that the target classification exists in a low-dimensional dense space as a degenerate prototype. By restricting the training data to a small number of samples, a mapping to this embedding space is learned.

[0077] 14. Few-parameter learning: also known as parameter-efficient learning, refers to adding a small number of parameters to certain network structures of a pre-trained model during fine-tuning. By adjusting these few parameters, large-scale parameter adjustments can be avoided.

[0078] A language model refers to the modeling of language sequences, such as natural language or programming languages. Currently, most language models use neural networks, which predict the probability of occurrence of language sequences. For example, a generative language model predicts the probability of a particular character, word, or other language unit appearing at each position in the sequence. Generative Pre-Training (GPT) language models, for instance, use a decoder network composed of multiple Transformer neural network computational units. The decoder predicts the possible words for the next position based on the preceding context, iterating until the entire sequence is predicted, i.e., decoding is complete. Pre-training of a language model involves training the corresponding language model using a massive corpus of language sequences, enabling the model to predict the probability of a particular language unit appearing at a specific position.

[0079] After pre-training, a pre-trained language model is obtained. This pre-trained language model is then fine-tuned on various natural language processing tasks to obtain downstream task models. Compared to training a model from scratch based on downstream task data, fine-tuning the pre-trained language model can utilize a large amount of language knowledge learned during pre-training, thus achieving a significant performance improvement.

[0080] As the parameter size of pre-trained language models continues to increase, traditional fine-tuning techniques, while achieving good results on downstream tasks, incur significant computational and storage costs due to the need to update all model parameters. To better utilize ultra-large-scale pre-trained language models, parameter fine-tuning has become an important research direction. When adapting to downstream tasks, parameter fine-tuning fixes the original pre-trained language model and adds a small number of additional parameters during training, thus significantly reducing the cost of fine-tuning.

[0081] Few-shot learning aims to learn from a very small number of downstream task samples, exploring a learning pattern similar to that of humans, and is an important direction for the development of deep learning.

[0082] In related technologies, researchers have analyzed the gap between parameter fine-tuning and full parameter fine-tuning in few-shot learning and found that the initialization of the newly added few parameters is a significant influencing factor. A good few-parameter initialization can significantly improve the few-shot learning performance of parameter fine-tuning, even surpassing that of full parameter fine-tuning. To obtain a good few-parameter initialization, a pre-training process needs to be introduced for the few parameters, so that the parameters are at a better initial point for learning.

[0083] One approach is to categorize by downstream task type. For each task type, a set of parameters trained with relevant data can be specified as the initialization parameters for new tasks of that type (such as question answering, information extraction, etc.). This can effectively improve the learning effect with few samples and the convergence speed of model training.

[0084] However, current low-parameter pre-training methods rely on manually constructed rules, which can only generalize to a small number of downstream task data types, and also require a large number of human experts to design the rules.

[0085] In related technologies, correlation analysis between neurons and downstream tasks has revealed that among the tens of thousands of neurons in a pre-trained model (usually located in the Transformer structure), downstream tasks have specific related neurons, such as skill neurons, which are very sparse. These neurons largely determine the accuracy of the downstream tasks.

[0086] At the same time, for different downstream tasks, similar tasks will activate the same batch of neurons, and the skill neurons of a certain task will also have similar performance. Therefore, selecting the best one can represent a type of task and the corresponding skill neurons of this type of task.

[0087] In high-efficiency parameter scenarios, the skill neurons are determined by the pre-trained model itself, and fine-tuning of new parameters does not affect the selection and distribution of skill neurons. Therefore, in this embodiment, for a certain downstream task, the relevant neurons for that task are determined, and for each neuron, a set of parameters that can activate these relevant neurons has been pre-trained using a low-parameter method. For that task or similar tasks, using these pre-trained parameters can significantly improve the training efficiency and the ability to perform tasks with few or zero samples.

[0088] Please refer to Figure 1 This illustrates a system architecture diagram provided in an exemplary embodiment of this application. Figure 1 As shown, the system includes a data collection device 11, a server device 12, and a client device 13, which are connected to each other through a communication network.

[0089] The data collection device 11 can be a terminal or a server. It is used to acquire the data required for this embodiment through various means (such as manual input, online searching, etc.) and transmit the acquired data to the server device 12. The server device 12 is used to execute the parameter-efficient training method provided in this embodiment based on the data transmitted by the data collection device 11, thereby obtaining the trained model result and deploying it on the client device 13. The client device 13 can be a terminal or a server. It is used to execute downstream tasks based on the trained model result, such as question-answering, classification, or dialogue generation tasks.

[0090] Among them, server device 12 can also be a network cloud server, and client device 13 can also be a cloud service device.

[0091] The application scenario of this application is a few-shot learning scenario for downstream tasks based on a large model. The entire training process can be set up on the cloud side. After training is completed, the service for the corresponding downstream tasks can be provided on the cloud side or the edge side. The downstream tasks can be downstream tasks of natural language processing, such as information extraction tasks, classification tasks, or text generation tasks. This application does not limit this.

[0092] The embodiments of this application can be implemented through program code and instructions, wherein data and instructions can be stored on various types of memory, such as read-only memory (ROM), or other types of static storage devices capable of storing static information and instructions, random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions. It can also be electrically erasable programmable read-only memory (EEPROM), disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by the application's search device. The execution of instructions or functional modules can be implemented by a processor or controller, such as a central processing unit (CPU) or a digital signal processor (DSP).

[0093] It should be noted that the data collection device 11, server device 12, and client device 13 described above can be integrated into a single computing device. In other words, the data collection device 11 and client device 13 are optional, and the tasks of the data collection device 11 and client device 13 can be directly performed by the server device 12.

[0094] For ease of explanation, the following description uses a computing device as the execution subject of the method provided in the embodiments of this application as an example. Please refer to... Figure 2 The diagram illustrates a structural schematic of a computing device provided in an exemplary embodiment of this application.

[0095] The computing device can be a terminal or a server. Terminals include mobile terminals or fixed terminals, such as mobile phones, tablets, laptops, and desktop computers. A server can be a single server, a server cluster consisting of several servers, or a cloud computing service center.

[0096] like Figure 2 As shown, the computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will understand that... Figure 2 The structure shown does not constitute a limitation on the computing device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0097] The processor 10 is the control center of the computing device. It connects various parts of the computing device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 20, and by calling data stored in the memory 20, it performs various functions of the computing device and processes data, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a graphics processing unit (GPU).

[0098] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system 21, the acquisition unit 22, the first training unit 23, the determination unit 24, the second training unit 25, and application programs required for at least one function. The data storage area may store data created according to the use of the computing device. The memory 20 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), EEPROM, Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Accordingly, the memory 20 may also include a memory controller to provide the processor 10 with access to the memory 20.

[0099] Specifically, processor 10 performs the following functions by running acquisition unit 22: acquiring a pre-trained language model, which includes the first parameters corresponding to each of m neurons, where m is a positive integer; processor 10 performs the following functions by running first training unit 23: training based on the language model and training corpus to obtain the second parameters corresponding to each of the m neurons, where the second parameters are different from the first parameters; processor 10 performs the following functions by running determination unit 24: when a few-shot learning task is acquired, determining the n neurons corresponding to the few-shot learning task, where n is a positive integer; processor 10 performs the following functions by running second training unit 25: initializing and training the parameters of the few-shot learning task based on the second parameters corresponding to each of the n neurons.

[0100] The parameter-efficient training method provided in this application improves the performance of parameter fine-tuning in scenarios with few learning iterations by providing good few-parameter initialization through few-parameter pre-training and designing a few-parameter initialization retrieval method to select appropriate initialization for specific tasks. Several exemplary embodiments are described below to illustrate the parameter-efficient training method provided in this application.

[0101] Please refer to Figure 3 This document illustrates a flowchart of an efficient parameter training method provided in an exemplary embodiment of this application. This embodiment uses the method in a computing device as an example for illustration. The method includes the following steps.

[0102] Step 301: Obtain the pre-trained language model. The language model includes the first parameters corresponding to each of the m neurons, where m is a positive integer.

[0103] The computing device acquires a pre-trained language model, which includes the first parameters of each of m neurons, where m is a positive integer. Optionally, the language model is a neural network used for natural language processing. The pre-trained language model is the original language model, which includes the first parameters but does not include the second parameters.

[0104] The first parameter is the original parameter of the language model. Optionally, the first parameter includes the weight values ​​corresponding to each of the original variables of the language model.

[0105] Step 302: Train the language model and training corpus to obtain the second parameters corresponding to each of the m neurons. The second parameters are different from the first parameters.

[0106] The computing device trains the language model using a training corpus to obtain the second parameters for each of the m neurons. These second parameters differ from the first parameters. The trained language model then includes both the first and second parameters.

[0107] The second parameter is a newly added parameter of the language model. Optionally, the second parameter includes the weight value corresponding to at least one newly added variable of the language model.

[0108] The training corpus includes training corpora corresponding to each of the m neurons. Optionally, for each neuron, multiple first activation values ​​are obtained by calling the language model output based on multiple first unlabeled corpora; the training corpus corresponding to the neuron is determined based on the multiple first activation values.

[0109] Optionally, for each neuron, the second parameter corresponding to the neuron is randomly initialized to obtain the initial parameter value; based on the language model and the training corpus corresponding to the neuron, the initial parameter value is pre-trained to obtain the second parameter corresponding to the neuron.

[0110] Optionally, a large number of parameter training tasks and the required data are constructed on large-scale text. In order to train a small number of pre-trained parameters that can activate any specific neuron in the model, the training corpus for activating that neuron is determined for all or some neurons in the model. The parameter training task and data construction use the activation values ​​of the language model on the data as the original label information, and then the data is filtered and the task is constructed.

[0111] Optionally, training can be performed for a specific parameter training task to obtain the corresponding parameters, which are then used for downstream task initialization. This step is a standard parameter training process, optimizing only the newly added parameters of the language model, i.e., the second parameter, while keeping the original parameters of the language model, i.e., the first parameter, unchanged.

[0112] Numerous parameter pre-training tasks with unique properties are constructed based on large-scale unsupervised text corpora. Compared to language model training methods, activation-based parameter pre-training tasks can more effectively stimulate the corresponding capabilities of language models.

[0113] Since each neuron in a language model possesses unique language capabilities, the purpose of parameter pre-training is to enable the parameters to better stimulate these unique capabilities. If a downstream task is particularly relevant to a particular neuron, using the pre-trained parameters corresponding to that neuron as initialization can yield better results in downstream few-shot learning tasks.

[0114] Depending on the input data, the neurons in the language model will produce different activation values. In an illustrative example, such as... Figure 4 As shown, the distribution of activation values ​​of neurons can be approximated by a normal distribution. The magnitude of the activation value indicates the magnitude of the target feature in the input sample.

[0115] Optionally, in this embodiment, the activation value of a neuron is defined as follows: Text data is first represented as a sequence of word vectors. Then, the neural network calculates the relationships between the word vectors and generates new word vectors. Before generating new word vectors, each word vector is typically processed using a non-linear function, such as the Linear Rectification Function (ReLU), for feature selection. A specific dimension of the word vector is defined as a neuron. After passing through the non-linear function, the maximum value of all word vectors in the sequence within that dimension is the activation value of the neuron. Based on this definition, this embodiment calculates the activation value of the text data for all neurons.

[0116] Therefore, the unlabeled training corpus can be labeled based on the activation values ​​of neurons to generate binary classification training tasks or even multi-class classification training tasks.

[0117] For a selected language model, prepare unsupervised sample corpora (e.g., sentences, paragraphs, and other text fragments, without annotation). Input each sample corpus into the language model and observe the activation values ​​of neurons. Based on whether the activation value is greater than the mean, construct a binary classification supervision signal. If the activation value of a neuron is greater than the mean of its activation value distribution, the sample corpus is determined as a positive sample for pre-training the parametric model that activates that neuron; if the activation value of a neuron is less than or equal to the mean of its activation value distribution, the sample corpus is determined as a negative sample. Alternatively, one sample corpus can be used as a query sample, and multiple randomly sampled sample corpora can be used as selection samples, requiring the model to select the sample whose activation value is closest to the query sample. By activating the model's perception ability of relevant features through binary or multi-class classification tasks, a large amount of training corpus can be constructed for each neuron to pre-train the parametric model that activates that neuron.

[0118] For each neuron, a pre-training task and data with desired parameters can be constructed, and then a corresponding second parameter can be trained for each neuron. These second parameters can be considered as switches for different neurons, and by controlling the switches, the specific capabilities contained in the model can be further activated.

[0119] Step 303: When a few-shot learning task is obtained, determine the n neurons corresponding to the few-shot learning task, where n is a positive integer.

[0120] When a computing device acquires a few-shot learning task, it determines the n neurons corresponding to the few-shot learning task, where n is a positive integer. Specifically, n is a positive integer less than m. For example, n is 10.

[0121] Optionally, when a few-shot learning task is obtained, the language model output is called to obtain the second activation value corresponding to each of the m neurons, based on the few-shot learning task. The second activation value indicates the correlation between the neuron and the few-shot learning task. Based on the second activation value corresponding to each of the m neurons, n neurons corresponding to the few-shot learning task are determined from the m neurons.

[0122] Step 304: Initialize and train the parameters for the few-shot learning task based on the second parameters corresponding to each of the n neurons.

[0123] The computing device initializes and trains parameters for a few-shot learning task based on the second parameters corresponding to each of the n neurons.

[0124] Optionally, the computing device combines the second parameters corresponding to each of the n neurons to obtain the target parameter value; and uses the target parameter value as the initialization parameter to train the parameters for a few-shot learning task.

[0125] Optionally, for downstream few-shot learning tasks, parameter initialization is performed by selecting and combining parameters. This step requires selecting n neurons corresponding to the downstream task and then initializing the parameters of each neuron obtained from pre-training. This step needs to be tailored to the capabilities required by the downstream task, selecting and combining existing parameters to provide a good initialization. The combination can be averaging, pooling, or more complex combination methods. Illustratively, the computing device determines the mean of the second parameters corresponding to each of the n neurons as the target parameter value.

[0126] Optionally, after initialization, parameter training can be performed using few-shot learning based on the initialization of the neuron's parameters.

[0127] Initializing the second parameter for a downstream few-shot learning task will result in a huge search space. For example, for a language model with tens of thousands of neurons, it is necessary to select one or several suitable second parameters from tens of thousands of second parameters.

[0128] Optionally, based on the activation values ​​of neurons in the language model on downstream few-shot learning task data, n neurons with task-related capabilities are initially identified, thus pruning the search space. Illustratively, the corpus of the downstream few-shot learning task is input into the original language model (i.e., the language model including the first parameter but excluding the second parameter), and the second activation values ​​of all neurons or a certain model substructure are calculated, such as a feedforward neural network (FFN) layer. The computing device sorts multiple neurons according to their second activation values ​​from largest to smallest, and the first n neurons after sorting are determined as the n neurons. For these n neurons, pre-trained second parameters that can activate the n neurons are selected and combined to finally obtain a suitable parameter set initialization, i.e., the parameter target values, for the downstream few-shot learning task. Since few-shot learning has low training overhead, this step can quickly validate the neurons in the initial framework, train using their corresponding parameter initializations, test the results on the validation set, and select the parameter initializations with higher accuracy for combination.

[0129] In summary, the embodiments of this application determine n neurons corresponding to a certain downstream few-shot learning task. For each neuron, parameters that can activate the corresponding neuron have been pre-trained. Therefore, for the few-shot learning task, using these pre-trained parameters can significantly improve the training efficiency and the ability to perform few-shot or zero-shot tasks.

[0130] Please refer to Figure 5This document illustrates a flowchart of an efficient parameter training method provided in another exemplary embodiment of this application, which is described in this embodiment using the method in a computing device. The method includes the following steps.

[0131] Step 501: Obtain the pre-trained language model, which includes the first parameters corresponding to each of the m neurons.

[0132] The computing device acquires a pre-trained language model as the base model. This pre-trained language model can be a pure encoding-based pre-trained language model with a Transformer architecture, such as Bidirectional Encoder Representation from Transformers (BERT); it can also be an encoder-decoder pre-trained language model, such as the T5 model; or a pure decoder pre-trained model, such as a Generative Pre-trained Transformer (GPT) model. Subsequently, a small number of parameters are added to this base model for adaptation to downstream tasks, which is the parameter fine-tuning method.

[0133] Step 502: Obtain the activation value distribution parameters for each neuron.

[0134] Optionally, for each of the multiple unlabeled corpora, the computing device inputs the unlabeled corpus (such as a text segment, sentence paragraph, etc.) into the language model and outputs the activation value of each neuron. Multiple activation values ​​corresponding to each neuron can be obtained from multiple unlabeled corpora input.

[0135] The activation value distribution parameter of a neuron indicates the multiple first activation values ​​corresponding to that neuron. For example, the activation value distribution parameter of a neuron is the mean of the multiple first activation values ​​corresponding to that neuron.

[0136] Step 503: Construct the training corpus for each neuron.

[0137] In one possible implementation, the activation value distribution parameter is the mean of multiple first activation values ​​corresponding to the neuron; the computing device constructs training corpora for each neuron, including: for each of the multiple second unlabeled corpora, calling the language model output to obtain the third activation value corresponding to the neuron; and constructing a binary classification supervision signal based on whether the third activation value is greater than the mean. Illustratively, when the third activation value is greater than the mean, the second unlabeled corpus is determined as a positive sample in the training corpus; when the third activation value is less than or equal to the mean, the second unlabeled corpus is determined as a negative sample in the training corpus.

[0138] In another possible implementation, the computing device constructs training corpora for each neuron, including: determining one sample from multiple unlabeled second corpora as the query sample, determining multiple randomly sampled samples as selection samples, and requiring the model to select the sample whose activation value is closest to the query sample, thereby obtaining training corpora for each neuron. By activating the model's perception ability for relevant features through binary or multi-class classification tasks, a large amount of pre-trained training corpora can be constructed for each neuron to activate that neuron. This application does not limit the method of constructing training corpora in its embodiments.

[0139] Since deep learning is a typical data-driven approach, in order to construct a large amount of unlabeled corpus, multiple unlabeled text data are obtained from the network as unlabeled corpora (multiple first unlabeled corpora and multiple second unlabeled corpora), including unlabeled corpora from sources such as encyclopedias, news, and forums. Optionally, at least one corpus in the multiple first unlabeled corpora and the multiple second unlabeled corpora may be the same, or the multiple first unlabeled corpora and the multiple second unlabeled corpora may have no overlap.

[0140] Step 504: Train the language model and training corpus to obtain the second parameters corresponding to each of the m neurons.

[0141] The computing device trains the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons. These second parameters can be regarded as the switches of different neurons, and by controlling the switches, the specific capabilities contained in the model can be further activated.

[0142] Optionally, the second parameter is initialized randomly; a neuron is selected, and the corresponding training corpus (such as pre-trained binary classification data) is obtained; based on the training corpus corresponding to the neuron, the second parameter is optimized using stochastic gradient descent, and the second parameter corresponding to the neuron is trained.

[0143] Step 505: When a few-shot learning task is obtained, obtain the second parameters of each of the n neurons corresponding to the few-shot learning task.

[0144] When a downstream few-shot learning task is obtained, the computing device determines the n neurons corresponding to the few-shot learning task and obtains the second parameters corresponding to each of the n neurons obtained from the above training.

[0145] Step 506: Initialize and train the parameters for the few-shot learning task based on the second parameters corresponding to each of the n neurons.

[0146] Optionally, the computing device combines the second parameters corresponding to each of the n neurons to obtain target parameter values; these target values ​​are then used as initialization parameters for fine-tuning, completing the training of the few-shot learning task. The combination can be averaging, pooling, or more complex methods.

[0147] Optionally, the computing device performs parameter fine-tuning to complete the training of the few-shot learning task, including: training with the few-shot learning task data, initializing the few parameters using the average result of the parameters in the previous step, optimizing the second parameter based on the stochastic gradient descent method, and evaluating the model performance of the trained language model on the validation set.

[0148] In an illustrative example, for downstream tasks, training was performed using both the parameter-efficient learning method provided by related technologies and the parameter-efficient learning method provided in the embodiments of this application. The scores of the two methods on different tasks are shown in Table 1. The different tasks are: sentence classification tasks, such as The Stanford Sentiment Treebank (SST-2); question answering tasks, such as Boolean Questions (BoolQ); and text recognition tasks, such as Recognizing Textual Entailment (RTE) and CB tasks.

[0149] Table 1

[0150] method SST-2 BoolQ RTE CB Related technologies 92.0 62.1 55.5 54.0 Examples of this application 92.1 76.8 59.3 63.1

[0151] In summary, the embodiments of this application are applicable to all methods employing efficient parameter learning, aiming to activate neurons in existing pre-trained models. Corresponding unsupervised parameter initialization pre-training methods were designed to obtain parameter initialization values ​​for all neurons. For downstream tasks, selecting corresponding neurons in a simple manner for a few rounds of efficient parameter learning initialization for fine-tuning training can achieve faster convergence speed and better learning results.

[0152] This application proposes a novel parameter initialization method for efficient parameter learning. Current efficient parameter initialization methods involve manually designing and classifying a large number of downstream task templates, unifying the downstream tasks into several patterns, using these patterns to pre-train parameters on a large number of downstream tasks to obtain parameter initializations for each type, and then manually selecting the corresponding initialization based on the type of downstream task. This scheme employs an unsupervised approach to pre-train the parameters of neurons and selects appropriate parameters corresponding to the neurons as the initialization for downstream task parameters.

[0153] The embodiments of this application can pre-train some or all neurons simultaneously. The pre-training method is simple and convenient; it does not depend on the specific type of downstream task and does not require manually designed patterns to classify them into certain task categories; compared with existing parameter-efficient pre-training, it has better results.

[0154] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0155] Please refer to Figure 6 This diagram illustrates a block diagram of an efficient parameter training apparatus provided in an exemplary embodiment of this application. The apparatus can be implemented as all or part of a computing device through software, hardware, or a combination of both. The apparatus may include: an acquisition unit 610, a first training unit 620, a determination unit 630, and a second training unit 640.

[0156] Acquisition unit 610 is used to acquire the pre-trained language model, which includes the first parameters of each of m neurons, where m is a positive integer.

[0157] The first training unit 620 is used to train based on the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons. The second parameters are different from the first parameters.

[0158] The determination unit 630 is used to determine the n neurons corresponding to the few-shot learning task when a few-shot learning task is obtained, where n is a positive integer.

[0159] The second training unit 640 is used to initialize and train parameters for a few-shot learning task based on the second parameters corresponding to each of the n neurons.

[0160] In one possible implementation, the training corpus includes training corpora corresponding to each of the m neurons, and the device further includes: a training corpus determination unit 630, used for:

[0161] For each neuron, based on multiple first unlabeled corpora, the language model output is called to obtain multiple first activation values ​​corresponding to the neuron;

[0162] The training corpus corresponding to a neuron is determined based on the multiple first activation values ​​corresponding to that neuron.

[0163] In another possible implementation, the first training unit 620 is also used for:

[0164] For each neuron, the second parameter corresponding to the neuron is randomly initialized to obtain the initial value of the parameter;

[0165] Based on the language model and the training corpus corresponding to the neurons, the initial values ​​of the parameters are pre-trained to obtain the second parameters corresponding to the neurons.

[0166] In another possible implementation, determining unit 630 is also used for:

[0167] When a few-shot learning task is obtained, the language model output is called to obtain the second activation value corresponding to each of the m neurons, based on the few-shot learning task. The second activation value indicates the correlation between the neuron and the few-shot learning task.

[0168] Based on the second activation values ​​of each of the m neurons, determine the n neurons corresponding to the few-shot learning task from the m neurons.

[0169] In another possible implementation, the second training unit 640 is also used for:

[0170] The target parameter value is obtained by combining the second parameters corresponding to each of the n neurons;

[0171] The target parameter value is used as the initialization parameter to train the parameters for a few-shot learning task.

[0172] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0173] Embodiments of this application provide a parameter-efficient training apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the methods executed by the computing device in the various embodiments described above when executing the instructions.

[0174] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the methods executed by a computing device in the various embodiments described above.

[0175] Embodiments of this application provide a computer program product comprising computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in an electronic device, the processor in the electronic device performs the methods executed by the computing device in the various embodiments described above.

[0176] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0177] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0178] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0179] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0180] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0181] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0183] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0184] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0185] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A parameter-efficient training method, characterized in that, The method includes: Obtain a pre-trained language model, wherein the language model includes the first parameters corresponding to each of m neurons, where m is a positive integer; The language model and training corpus are used to train the m neurons to obtain the second parameters corresponding to each of the m neurons. The second parameters are different from the first parameters. When a few-shot learning task is obtained, n neurons corresponding to the few-shot learning task are determined, where n is a positive integer; The few-shot learning task is initialized and trained according to the second parameters corresponding to each of the n neurons. The initialization and parameter training of the few-shot learning task based on the second parameters corresponding to each of the n neurons includes: The second parameters corresponding to each of the n neurons are combined to obtain the target parameter values; The target value of the parameter is used as the initialization parameter to train the parameters for the few-shot learning task.

2. The method according to claim 1, characterized in that, The training corpus includes training corpora corresponding to each of the m neurons. Before training based on the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons, the method further includes: For each neuron, based on multiple first unlabeled corpora, the language model output is invoked to obtain multiple first activation values ​​corresponding to the neuron; The training corpus corresponding to the neuron is determined based on the plurality of first activation values ​​corresponding to the neuron.

3. The method according to claim 2, characterized in that, The step of training based on the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons includes: For each neuron, the second parameter corresponding to the neuron is randomly initialized to obtain the initial value of the parameter; Based on the language model and the training corpus corresponding to the neuron, the initial values ​​of the parameters are pre-trained to obtain the second parameters corresponding to the neuron.

4. The method according to any one of claims 1 to 3, characterized in that, When a few-shot learning task is obtained, determining the n neurons corresponding to the few-shot learning task includes: When the few-shot learning task is obtained, the language model output is called to obtain the second activation value corresponding to each of the m neurons according to the few-shot learning task. The second activation value indicates the correlation between the neuron and the few-shot learning task. Based on the second activation value corresponding to each of the m neurons, determine the n neurons corresponding to the few-shot learning task from the m neurons.

5. A parameter-efficient training device, characterized in that, The device includes: An acquisition unit is used to acquire a pre-trained language model, wherein the language model includes the first parameters corresponding to each of m neurons, where m is a positive integer. The first training unit is used to train according to the language model and the training corpus to obtain the second parameters corresponding to each of the m neurons, wherein the second parameters are different from the first parameters; The determining unit is used to determine the n neurons corresponding to the few-shot learning task when a few-shot learning task is obtained, where n is a positive integer; The second training unit is used to initialize and train the parameters of the few-shot learning task according to the second parameters corresponding to each of the n neurons. The second training unit is also used for: The second parameters corresponding to each of the n neurons are combined to obtain the target parameter values; The target value of the parameter is used as the initialization parameter to train the parameters for the few-shot learning task.

6. A parameter-efficient training device, characterized in that, The device includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-4 when executing the instructions.

7. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-4.

8. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, characterized in that, When the computer-readable code is run in an electronic device, the processor in the electronic device performs the method according to any one of claims 1-4.

Images