A neural network training method and device

Abstract

This application discloses a neural network training method and apparatus in the field of artificial intelligence, which is used to reduce the size of the search space by performing a homogeneous search under the constraints of resource constraints, and efficiently search for a model that meets the requirements. The method includes: obtaining a super pre-trained language model; obtaining resource constraints and a hyperparameter search space, wherein the resource constraints indicate constraints on the resources occupied by the model, and the hyperparameter search space indicates the range of the model's hyperparameters; searching the hyperparameter search space for the structure of a target model under the constraints of the resource constraints, wherein the hyperparameters of at least two network layers of the target model are the same; and training the target model to obtain a trained target model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a neural network training method and device. Background Art

[0002] Currently, large-scale pre-trained language models have become the dominant model in the field of neural network training, significantly surpassing previous models in almost all natural language understanding tasks. These models are based on the advanced Transformer architecture and are trained through self-supervised pre-training tasks on large corpora (e.g., 10GB to 100GB). However, due to the huge computing resources required and slow inference time of these models, deploying them in production environments remains a technical challenge.

[0003] Typically, model search methods can be used to build pre-trained language models or neural translation models. These methods change the Transformer structure and introduce components with different structures. As a result, the search space and training costs are huge, and the efficiency of the final model is low. Summary of the Invention

[0004] The present application provides a neural network training method and apparatus for efficiently searching for a model that meets the requirements by performing homogeneous search under the constraints of resource constraints, reducing the size of the search space.

[0005] In view of this, in a first aspect, the present application provides a neural network training method, comprising: obtaining a super pre-trained language model; obtaining resource constraints and a hyperparameter search space, wherein the resource constraints indicate constraints on the resources occupied by the model, and the hyperparameter search space indicates the range of the model's hyperparameters; searching for the structure of a target model from the hyperparameter search space under the constraints of the resource constraints, wherein the hyperparameters of at least two network layers of the target model are the same; and training the target model to obtain a trained target model.

[0006] Therefore, in the embodiments of the present application, the hyperparameters of at least two network layers of the searched sub-model are the same, so the hyperparameter search space required to be constructed is smaller, and the efficiency of searching the sub-model is also higher, so that the target model that meets the requirements can be efficiently searched. In addition, the target model obtained by the search has a homogeneous structure, that is, the hyperparameters of at least two network layers are the same, and the structure is simple, so the model can be deployed in more scenarios. In other words, the method provided by the present application can be applied to a variety of scenarios and has strong generalization capabilities.

[0007] In one possible implementation, the hyperparameters of each network layer of the target model are identical. Therefore, in the implementation of the present application, each network layer of the target model ultimately has the same hyperparameters, resulting in a homogeneous model. This reduces the required hyperparameter search space and allows for efficient search for models that meet resource constraints.

[0008] In one possible implementation, the aforementioned acquisition of a super pre-trained language model may include: first constructing a neural network to obtain an untrained initial model; dividing the training set into multiple subsets; collecting sub-models from the initial model to obtain multiple first sub-models; and using multiple subsets to train the multiple first sub-models in parallel to obtain a super pre-trained language model.

[0009] Therefore, in the embodiments of the present application, the sub-models in the initial model can be trained in parallel using the training set, thereby improving the training efficiency of the initial model and more efficiently obtaining a super pre-trained language model.

[0010] In one possible implementation, the aforementioned use of multiple subsets to train multiple first sub-models in parallel to obtain a super pre-trained language model may include: distributing the multiple subsets and the multiple first sub-models to multiple computing nodes to calculate the gradients of the multiple first sub-models in the multiple computing nodes; updating the parameters of the initial model according to the gradients of the multiple first sub-models to obtain the super pre-trained language model.

[0011] Therefore, in the implementation manner of the present application, multiple subsets and corresponding sub-models can be distributed to multiple computing nodes, so that multiple computing nodes can calculate the gradients of each sub-model in parallel, thereby improving the training efficiency of the initial model and efficiently obtaining the trained model.

[0012] Optionally, the aforementioned computing nodes may include nodes with processing capabilities such as servers, personal computers, processors, processes or threads.

[0013] In one possible implementation, the aforementioned updating of the parameters of the initial model according to the gradients of multiple sub-models to obtain the super pre-trained language model may include: fusing the gradients of multiple first sub-models to obtain a fused gradient; and updating the parameters of the initial model according to the fused gradient to obtain the super pre-trained language model.

[0014] In the embodiments of the present application, after the gradients of multiple sub-models are calculated separately, the gradients of the multiple sub-models can be fused, and the initial model can be updated based on the fused gradients. Therefore, the gradients of each sub-model can be efficiently calculated in parallel, and then the complete initial model can be updated based on the fused gradients, thereby efficiently completing the training of the initial model and obtaining a model with better output effects.

[0015] Optionally, the method of fusing the ladders of multiple first sub-models may specifically include weighted fusion, summation, or averaging.

[0016] In one possible embodiment, the aforementioned search for the target model structure from the hyperparameter search space under the resource constraint condition may include: under the resource constraint condition, searching for at least one model structure from the hyperparameter search space, wherein the hyperparameters of at least two network layers of the model corresponding to each model structure in the at least one model structure are the same; collecting at least one second sub-model corresponding to the at least one model structure from the super pre-trained language model; and screening the model structure whose output results meet the preset conditions from the at least one model structure based on the at least one second sub-model to obtain the target model structure. It can be understood that first, at least one model structure that meets the resource constraint condition is searched from the hyperparameter search space, and the structural parameters of different model structures may be different, such as different depth or width, and the output effects of the models corresponding to different model structures may also be different. Therefore, a model structure with better output effect can be further screened from the at least one model to serve as the model structure of the target model.

[0017] In the implementation manner of the present application, multiple sub-models can be searched under the constraints of resource constraints, and a sub-model with better output effect can be screened out from the multiple sub-models as the target model, so that a target model that meets the resource constraints and has better output effect can be obtained.

[0018] In one possible embodiment, the aforementioned acquisition of at least one second sub-model corresponding to at least one model structure from the super pre-trained language model may include: acquiring a sub-model from the initial model to obtain at least one third sub-model; and acquiring at least one module from each network layer of each third sub-model in the at least one third sub-model to obtain at least one second sub-model.

[0019] In the implementation mode of the present application, in order to further reduce the complexity of the model, after the sub-model is collected from the initial model, one or more modules can be collected from the network layer of the sub-model to obtain a sub-model with a smaller model structure, thereby further improving the training efficiency. Usually, when the hyperparameter range set in the hyperparameter search space is within a pre-set range, the structure of each network layer in the sub-model is similar. Therefore, one or more modules are extracted from each network layer of the sub-model. While reducing the model structure, an output effect close to that of the original model can also be achieved, so that the model can be deployed in more devices and the generalization ability of the model is improved.

[0020] In a possible implementation, the aforementioned searching for at least one model structure from the hyperparameter search space under the constraints of resource constraints may include: searching for multiple model structures from the hyperparameter search space; obtaining multiple running results of models corresponding to the multiple model structures running in hardware; and screening out the model structure whose running results meet the resource constraints from the multiple model structures to obtain at least one model structure.

[0021] In the implementation manner of the present application, whether the model meets the resource constraint conditions can be tested by running the sub-model in hardware, thereby obtaining a model that meets the resource constraint conditions.

[0022] In a possible embodiment, the aforementioned method of screening out a model structure whose output results meet preset conditions from at least one model structure based on at least one second sub-model to obtain the structure of the target model may include: evaluating the output results of at least one second sub-model using a test set to obtain an evaluation result of each second sub-model; screening out a model structure that meets the preset conditions from at least one model structure based on the evaluation result of each second sub-model; and using the model structure that meets the preset conditions as the structure of the target model.

[0023] In the implementation manner of the present application, the output effect of the searched sub-model can be evaluated to obtain a target model with better output effect.

[0024] In one possible implementation, the aforementioned obtaining of the target model based on the structure of the model that meets the preset conditions may include: constructing the target model based on the structure of the model that meets the preset conditions; or, collecting the corresponding sub-model from the super pre-trained language model to obtain the target model.

[0025] In the implementation manner of the present application, after determining the structure of the target model, the target model can be constructed, or a target model with better output effect can be collected from the super pre-trained language model, which can adapt to various scenarios and obtain the target model more flexibly.

[0026] In one possible implementation, each network layer of the super pre-trained language model includes a multi-head attention network, which includes multiple attention modules, each attention module has parameters of multiple dimensions, and the multi-head attention network is used to calculate the correlation between multiple input vectors and obtain an output vector including contextual semantics based on the correlation.

[0027] The aforementioned collection of at least one second sub-model corresponding to at least one model structure from the super pre-trained language model can specifically include: collecting at least one attention module from multiple attention modules in the multi-head attention network of each network layer of the super pre-trained language model, and collecting parameters of at least one dimension from parameters of multiple dimensions to obtain multiple second sub-models; or, collecting at least one parameter from each dimension in each attention module in the multi-head attention network of each network layer of the super pre-trained language model to obtain multiple second sub-models.

[0028] Therefore, in the embodiment of the present application, when collecting sub-models, partial parameters can be collected from each head in the multi-head attention network, or all parameters of some heads can be collected, so that the parameters of each head can be independent of each other, resulting in a more independent sub-model.

[0029] In a possible implementation, the aforementioned acquisition of resource constraints may include: receiving user input data; and acquiring resource constraints from the user input data, where the resource constraints include constraints related to the model's latency or memory usage.

[0030] Therefore, in the implementation manner of the present application, resource constraints can be determined based on user input data, so that a model that meets the resource requirements of the user and has better output effects can be obtained later, thereby improving user experience.

[0031] In a possible implementation, the aforementioned searching for at least one model structure from the hyperparameter search space under the constraint of resource constraints may include: sampling the structure of the sub-model according to the structural range included in the hyperparameter search space under the constraint of resource constraints to obtain the at least one model structure, the structural range includes at least one of a depth range or a width range, the depth range includes the range of the depth of the collected sub-model, and the width range includes the range of the width of the collected sub-model.

[0032] Therefore, in an embodiment of the present application, the hyperparameter search space may include a depth range or a width range, so that the depth or width and other structures of the model can be determined based on the hyperparameters included in the hyperparameter search space.

[0033] In a second aspect, embodiments of the present application provide a neural network training device that implements the neural network training method of the first aspect. This functionality can be implemented through hardware or through hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functionality.

[0034] In a third aspect, embodiments of the present application provide a neural network training device comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute processing-related functions of any of the neural network training methods described in the first aspect. Optionally, the neural network training device may be a chip.

[0035] In a fourth aspect, an embodiment of the present application provides a neural network training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the first aspect or any optional embodiment of the first aspect.

[0036] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0037] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect or any optional embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of an artificial intelligence framework used in this application;

[0039] Figure 2 A schematic diagram of the structure of the network layer of the neural network provided in this application;

[0040] Figure 3 A schematic diagram of the system architecture provided for this application;

[0041] Figure 4 Another system architecture diagram provided for this application;

[0042] Figure 5 A flowchart of a neural network training method provided in this application;

[0043] Figure 6 A flowchart of another neural network training method provided in this application;

[0044] Figure 7 A schematic diagram of a method for collecting sub-models provided in this application;

[0045] Figure 8 A schematic diagram of another method for collecting sub-models provided in this application;

[0046] Figure 9 A flowchart of another neural network training method provided in this application;

[0047] Figure 10 A schematic diagram of the pre-training process provided in this application;

[0048] Figure 11 A flowchart of the search phase provided for this application;

[0049] Figure 12 A schematic diagram of the structure of a neural network training device provided in this application;

[0050] Figure 13 A schematic diagram of the structure of another neural network training device provided in this application;

[0051] Figure 14 A schematic diagram of the structure of a chip provided in this application. DETAILED DESCRIPTION

[0052] The following will describe the technical solutions in the embodiments of this application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The following diagram illustrates a structural diagram of the AI ​​framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0054] (1) Infrastructure

[0055] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. Communication with the outside world occurs through sensors; computing power is provided by intelligent chips, such as central processing units (CPUs), neural-network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or hardware acceleration chips like field programmable gate arrays (FPGAs). The foundational platform includes distributed computing frameworks and networks, along with other related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0056] (2) Data

[0057] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0058] (3) Data processing

[0059] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0060] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0061] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0062] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0063] (4) General ability

[0064] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0065] (5) Smart products and industry applications

[0066] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0067] The embodiments of the present application involve applications related to neural networks and neural network training (natural language processing, NLP). In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of neural networks that may be involved in the embodiments of the present application are first introduced below.

[0068] Corpus: Also known as free text, it can be words, phrases, sentences, fragments, articles, or any combination thereof. For example, "The weather is really nice today" is a piece of corpus.

[0069] Loss function: Also known as the cost function, this metric compares the difference between a machine learning model's predicted output for a sample and the sample's true value (also called the supervised value). In other words, it measures the difference between the predicted output of a machine learning model and the true value of the sample. During neural network training, the goal is to keep the output as close as possible to the desired predicted value. This is done by comparing the network's predictions with the desired target values. The weight vectors of each layer are then updated based on the difference between the two. (Of course, before the first update, there is typically an initialization process, which pre-configures the parameters of each layer in a deep neural network.) For example, if the network's prediction is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the deep neural network predicts the desired target value or a value very close to it. Therefore, it is necessary to predefine how to compare the difference between the predicted and target values. This is the loss function, or objective function, which is a key equation used to measure the difference between the predicted and target values. For example, a higher loss function output (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss. The loss function can usually include mean square error, cross entropy, logarithm, exponential loss function, etc. For example, the mean square error can be used as the loss function, which is defined as The specific loss function can be selected according to the actual application scenario.

[0070] Gradient: The vector of derivatives of the loss function with respect to the parameters.

[0071] Stochastic gradient: The number of samples in machine learning is very large, so the loss function is calculated each time based on randomly sampled data, and the corresponding gradient is called stochastic gradient.

[0072] Back propagation (BP): An algorithm that calculates the gradient of model parameters based on a loss function and updates the model parameters. Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, making the reconstruction error loss of the neural network model smaller and smaller. Specifically, forward propagation of the input signal to the output generates error loss. The error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The back propagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0073] Neural machine translation: Neural machine translation is a typical task in neural network training. Given a source language sentence, it outputs its corresponding target language sentence. In common neural machine translation models, words in both the source and target language sentences are encoded into vector representations. The connections between words and sentences are calculated in this vector space to perform the translation task.

[0074] A pre-trained language model (PLM) is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction. A pre-trained language model is based on a multi-layer Transformer architecture and is obtained by pre-training on large-scale, unlabeled corpora. Pre-trained language models possess powerful semantic understanding capabilities, far surpassing many other models in downstream tasks. PLM training consists of two phases: pre-training and fine-tuning. In the pre-training phase, the model is trained on a large amount of unsupervised text for language modeling tasks, learning word representations. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase and trained in a relatively small number of steps on downstream tasks such as text classification or sequence labeling. This allows the semantic information learned from pre-training to be successfully transferred to the downstream task.

[0075] Super Pretrained Language Model (SuperPLM): Compared with PLM, SuperPLM has a larger model structure and can be obtained through a single training. Sub-models of arbitrary structures can be extracted from it, and the output effect of the sub-model can be close to the output effect of the PLM obtained through independent training, which greatly saves training costs.

[0076] Transformer structure: Generally, PLM can be built based on the Transformer structure. For example, PLM can be obtained by stacking multiple Transformer structures. A Transformer structure can also be called a Transformer layer. Compared with other neural networks, such as recurrent neural networks or convolutional neural networks, the Transformer layer relies on its attention mechanism to capture the long-term dependencies between words. For example, the structure of the Transformer layer can be as follows: Figure 2As shown in the figure, the Transformer layer consists of a multi-head attention network and a feedforward network module. The multi-head attention network calculates the correlation between words to obtain corresponding weights, resulting in contextual word representations. This is the core component of the Transformer structure. The feedforward network further transforms the obtained representations to produce the final output of the Transformer layer. In addition to these two important components, a residual layer (ADD) and linear normalization (Norm) are also stacked on top of these two components to optimize the Transformer layer's output.

[0077] Knowledge distillation: Knowledge distillation is a key technology for miniaturizing deep learning models and meeting the requirements for terminal device deployment. Compared to compression techniques such as quantization and sparsification, it does not require specific hardware support to achieve the goal of compressing models. Knowledge distillation adopts a teacher-student model learning strategy. The teacher model has strong semantic representation capabilities and large model parameters, which generally cannot meet deployment requirements; the student model has fewer parameters and can be deployed directly. By designing an effective knowledge distillation algorithm, the student model learns to imitate the behavior of the teacher model and effectively transfers knowledge, ultimately achieving the same performance as the teacher model.

[0078] Epoch: defines the number of times the learning algorithm works on the entire training set. An epoch can be considered to represent the entire dataset, that is, the number of times the neural network is trained using the entire training set.

[0079] Batch: is closely related to the definition of epoch. An epoch involves training the neural network using the entire dataset, and a batch represents one of the batches of data in an epoch. Specifically, batch_size*batchs=epoch. It can be understood that each epoch is divided into one or more batches, and each batch can use part of the data in the training set to train the neural network.

[0080] The neural network training method provided in the embodiments of the present application can be executed on a server or on a terminal device. The terminal device can be a mobile phone with image processing capabilities, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smart watch, a wearable device (WD), or an autonomous vehicle, etc., and the embodiments of the present application are not limited thereto.

[0081] See attached Figure 3 , an embodiment of the present application provides a system architecture 200. The system architecture includes a database 230 and a client device 240. A data acquisition device 260 is used to collect data and store it in the database 230. A training module 202 generates a target model / rule 201 based on the data maintained in the database 230. The following will describe in more detail how the training module 202 obtains the target model / rule 201 based on the data. The target model / rule 201 is the target model mentioned in the following embodiments of the present application. For details, please refer to the relevant description in the following embodiments.

[0082] The calculation module may include a training module 202. The target model / rule obtained by the training module 202 may be applied to different systems or devices. Figure 3 In the embodiment, the execution device 210 is configured with a transceiver 212, which can be a wireless transceiver, an optical transceiver or a wired interface (such as an I / O interface), etc., to interact with external devices. The "user" can input data to the transceiver 212 through the client device 240. For example, the client device 240 can send a target task to the execution device 210, request the execution device to train the neural network, and send a database for training to the execution device 210.

[0083] The execution device 210 can call data, code, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0084] The calculation module 211 processes the input data using the target model / rule 201. Specifically, the calculation module 211 is used to:

[0085] Finally, the transceiver 212 returns the constructed or trained neural network to the client device 240 so that the neural network can be deployed in the client device 240 or other devices.

[0086] More deeply, the training module 202 can obtain corresponding target models / rules 201 based on different data for different tasks to provide users with better results.

[0087] In the attached Figure 3 In the case shown in , the data input to execution device 210 can be determined based on user input data. For example, the user can operate in the interface provided by transceiver 212. In another case, client device 240 can automatically input data to transceiver 212 and obtain results. If automatic data input by client device 240 requires user authorization, the user can set the corresponding permissions on client device 240. The user can view the results output by execution device 210 on client device 240, which can be presented in a specific form such as display, sound, action, etc. Client device 240 can also serve as a data collection terminal to store the collected data associated with the target task in database 230.

[0088] The training or updating process mentioned in this application can be performed by the training module 202. It is understandable that the training process of the neural network is to learn the way to control the spatial transformation, more specifically, to learn the weight matrix. The purpose of training the neural network is to make the output of the neural network as close to the expected value as possible. Therefore, the weight vector of each layer of the neural network in the neural network can be updated according to the difference between the predicted value and the expected value of the current network (of course, the weight vector can usually be initialized before the first update, that is, the parameters of each layer in the deep neural network are pre-configured). For example, if the predicted value of the network is too high, the value of the weight in the weight matrix is ​​adjusted to reduce the predicted value. After continuous adjustment, the value output by the neural network is close to or equal to the expected value. Specifically, the difference between the predicted value and the expected value of the neural network can be measured by a loss function or an objective function. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. The training of the neural network can be understood as the process of minimizing the loss as much as possible. The process of updating the weight of the starting network and training the serial network in the following embodiments of this application can refer to this process and will not be repeated below.

[0089] like Figure 3 As shown, the target model / rule 201 is obtained through training according to the training module 202. In the embodiment of the present application, the target model / rule 201 may be the target model in the present application.

[0090] During the training phase, database 230 can be used to store sample sets for training. Execution device 210 generates a target model / rule 201 for processing the samples and iteratively trains target model / rule 201 using the sample sets in the database, resulting in a mature target model / rule 201, which is specifically represented by a neural network. The neural network generated by execution device 210 can be applied to various systems or devices.

[0091] During the inference phase, the execution device 210 can access data, code, etc. from the data storage system 250, or store data, instructions, etc. in the data storage system 250. The data storage system 250 can be located within the execution device 210, or it can be an external memory relative to the execution device 210. The computing module 211 can process the samples obtained by the execution device 210 through a neural network to obtain a prediction result. The specific form of the prediction result is related to the function of the neural network.

[0092] It should be noted that the attached Figure 3 This is only an exemplary schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 3 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other scenarios, the data storage system 250 can also be placed in the execution device 210.

[0093] The target model / rule 201 trained by the training module 202 can be applied to different systems or devices, such as mobile phones, tablet computers, laptops, augmented reality (AR) / virtual reality (VR), vehicle terminals, etc., and can also be servers or cloud devices.

[0094] The target model / rule 201 may include the super pre-trained language model or target model in the embodiment of the present application. Specifically, the model provided in the embodiment of the present application may include CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN) and other neural networks.

[0095] See attached Figure 4The present embodiment also provides a system architecture 300. The execution device 210 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, and load balancers. The execution device 210 can be deployed at a single physical site or distributed across multiple physical sites. The execution device 210 can use data in the data storage system 250 or call program code in the data storage system 250 to implement the steps of the neural training method described below in the present application.

[0096] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0097] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes but is not limited to: a fifth-generation mobile communication technology (5th-Generation, 5G) system, a long-term evolution (long term evolution, LTE) system, a global system for mobile communication (global system for mobile communication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (near field communication, NFC) and any one or more combinations thereof. The wired network can include a fiber optic communication network or a network composed of coaxial cables, etc.

[0098] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, the local device 301 may provide local data or feedback calculation results to the execution device 210. The local device may also be referred to as a computing device.

[0099] It should be noted that all functions of the execution device 210 can also be implemented by local devices. For example, the local device 301 implements the functions of the execution device 210 and provides services to its own users, or provides services to the users of the local device 302.

[0100] In some common PLM-oriented pre-training processes, a teacher model can be used to perform knowledge distillation on the student model, resulting in a more compact student model. However, pre-training often uses traditional Transformer model hyperparameter settings, resulting in low pre-training efficiency. Alternatively, a PLM can be constructed using architectural search. During this construction process, the model's Transformer structure is often modified, introducing heterogeneous modules or components. The structure of each network layer may vary. Consequently, the search space and training cost are both large, and the resulting model may lack regularity, making deployment difficult and generalizing poorly.

[0101] Therefore, the present application provides a neural network training method, which efficiently searches for a model that meets the requirements through a homogeneous search method. The method provided by the present application is introduced in detail below.

[0102] See Figure 5 , a flowchart of a neural network training method provided in this application is as follows.

[0103] 501. Obtain a super pre-trained language model.

[0104] Among them, the super pre-trained language model (SuperPLM for short) can be obtained by pre-training the initial model. The SuperPLM can be understood as a model with a larger structure, which can include multiple trained sub-models.

[0105] In one possible implementation, a SuperPLM can be obtained by pre-training an initial model. The pre-training process may include: constructing the initial model, either randomly or by searching for structural parameters within a pre-set range; dividing the training set into multiple subsets, and extracting multiple sub-models from the initial model, referred to herein as multiple first sub-models for ease of distinction. The multiple subsets are then used to train the multiple first sub-models in parallel to obtain the SuperPLM.

[0106] Optionally, when using the multiple subsets to train the multiple first sub-models in parallel, the multiple subsets and the corresponding first sub-models can be distributed to multiple computing nodes, so that the gradient of each first sub-model is calculated in the multiple computing nodes, and the parameters of the initial model to be trained are updated according to the gradient of each first sub-model, thereby obtaining SuperPLM.

[0107] Specifically, a computing node can be a server, a personal computer (PC), a processor, a process or a thread, etc., which has processing capabilities. It can be adjusted according to the actual application scenario, and this application does not limit this.

[0108] It should be noted that in the following implementation manner of this application, the computing node is used as a thread for illustrative explanation. The threads mentioned below can also be replaced by servers, PCs, processors or processes, etc., which will not be repeated below.

[0109] For example, after obtaining multiple subsets and multiple sub-models, an operating system API can be called to allocate the subsets and sub-models, for example, by allocating processing resources through the operating system's process scheduling function. Multiple subsets and corresponding sub-models can be randomly allocated to multiple threads, or evenly allocated to multiple threads. Alternatively, the allocation of multiple subsets and corresponding sub-models to multiple threads can be determined based on information such as the number of data blocks and threads, the thread busyness, and so on. Each subset and corresponding sub-model can then be allocated to multiple threads by calling the API.

[0110] More specifically, after the multiple threads calculate the gradients corresponding to the multiple first sub-models, they can fuse the gradients of the multiple first sub-models to obtain a fused gradient, which is then used to update the initial model to obtain the SuperPLM. The fusion of the gradients of the multiple first sub-models can be achieved by averaging or weighted summing, and the appropriate fusion method can be selected based on the actual application scenario, and this application does not limit this.

[0111] Optionally, a specific method for acquiring the first sub-model from the initial model may include presetting a depth range or a width range, randomly sampling structural parameters such as depth values ​​or width values ​​from the depth range or the width range, and then acquiring a corresponding sub-model from the initial model based on the structural parameters. For example, if structural parameters with a depth value of 5 and a width value of 6 are acquired, a sub-model with a depth of 5 and a width of 6 may be acquired from the initial model.

[0112] In one possible implementation, to reduce the complexity of the collected sub-models, when the parameters of the initial model fall within a certain range, a small number of modules can be collected from each network layer in the sub-model, thereby obtaining a sub-model with a simpler structure. Specifically, sub-models can be collected from the initial model, and then some modules can be collected from each network layer in each collected sub-model to obtain the aforementioned multiple first sub-models.

[0113] In a possible embodiment, the method of collecting sub-models from the initial model may also include: each network layer of the initial model may include a multi-head attention network, the multi-head attention network includes multiple attention modules, each attention module has parameters of multiple dimensions, and the multi-head attention network is used to calculate the correlation between multiple input vectors, and obtain an output vector including contextual semantics based on the correlation; at least one attention module can be collected from the multiple attention modules in the multi-head attention network of each network layer of the initial model, and at least one dimension of parameters can be collected from the multiple dimensions of parameters in each collected attention module to obtain multiple first sub-models, thereby collecting the structure of each attention module and part of the parameters of each attention module, so that the dimension of the attention head of the sub-model is consistent with the dimension of the attention head of the initial model; or, at least one parameter is collected from each dimension of each attention module in the multi-head attention network of each network layer of the initial model to obtain multiple first sub-models, so that the number of heads of the sub-model is consistent with the number of heads of the network layer in the initial model.

[0114] Of course, in a possible implementation, the trained SuperPLM can also be extracted from other devices or from a memory. For example, the method provided by this application can be Figure 2 The execution device 210 mentioned in the above is executed. The execution device 210 can directly extract the trained SuperPLM from the database 230, or receive the SuperPLM sent by other devices, or use the training module 202 to train the initial model to obtain the SuperPLM.

[0115] 502. Obtain resource constraints and hyperparameter search space.

[0116] The resource constraint conditions include the constraint conditions on the resources occupied by the final model, that is, the range of resources occupied by the target model.

[0117] The hyperparameter search space includes the range of hyperparameters of the final model. Typically, the hyperparameter search space can include hyperparameter ranges such as depth range or width range. The depth range is the range of depth of the searched model structure, and the width range is the range of width of the searched model. Typically, the depth range can be understood as the range of the number of network layers of the model, and the width range can be understood as the number of modules or input and output channels of each network layer.

[0118] Optionally, the resource constraint condition can be obtained through data input by the user. Specifically, the user input data can be received, and then the resource constraint condition can be obtained from the user input data. The resource constraint condition can specifically include constraints related to the delay or memory usage of the model. For example, the resource constraint condition can include the maximum delay required for the forward reasoning process from input to output of the model to be built at runtime, or the maximum memory usage of the model to be built at runtime, so that a model that meets the user's needs can be searched in the subsequent structural search, thereby improving the user experience.

[0119] Typically, the SuperPLM structure can cover the model structures corresponding to the structural parameters included in the hyperparameter search space. For example, if the number of SuperPLM network layers is 100, the depth range included in the hyperparameter search space will not exceed 100. If the width of the SuperPLM network layer does not exceed 20, the maximum width range included in the hyperparameter search space will not exceed 20. Therefore, in the embodiments of the present application, a smaller hyperparameter search space can be used to achieve structural search, improving search efficiency.

[0120] 503. Search the target model structure from the hyperparameter search space under the constraints of resource constraints.

[0121] After obtaining the resource constraints and hyperparameter search space, the structure of the target model that meets the requirements can be searched from the hyperparameter search space under the constraints of the resource constraints. The hyperparameters of at least two network layers of the target model are the same, such as the width of at least two network layers is the same, the types of each component are the same, etc.

[0122] Specifically, under resource constraints, at least one model structure can be searched from the hyperparameter search space, where the hyperparameters of at least two network layers of the model corresponding to each model structure are identical, such as the width of the first network layer and the width of the second network layer being identical, or the hyperparameters of each network layer being identical. Sub-models corresponding to the at least one model structure are then collected from SuperPLM, referred to herein as second sub-models for ease of distinction. Model structures that meet preset conditions are then screened based on the output of the at least one collected second sub-model to obtain the structure of the target model.

[0123] The preset conditions may specifically include indicators such as the output accuracy of the model being greater than the preset accuracy, the average accuracy of the model being greater than the preset average accuracy, etc., which are used to measure the output effect of the model. These can be set according to the actual application scenario, and this application does not limit this.

[0124] Furthermore, when the parameters searched from the hyperparameter search space are within a pre-set range, the method for acquiring the second sub-model may include: acquiring a sub-model from the SuperPLM, which is referred to as a third sub-model for ease of distinction, and acquiring at least one module from each network layer of each of the multiple acquired third sub-models to obtain one or more second sub-models. Therefore, by acquiring a portion of the modules from each network layer, the structure of the acquired sub-model can be simplified, thereby improving search efficiency.

[0125] In one possible implementation, the aforementioned searching for at least one model structure from the hyperparameter search space under the constraints of resource constraints may specifically include: searching for multiple model structures from the hyperparameter search space, constructing multiple corresponding models based on the multiple model structures, and then obtaining multiple running results of the multiple models running in pre-selected hardware, and obtaining at least one model structure based on the model results whose running results meet the resource constraints from the multiple model structures. Therefore, in the implementation of the present application, the structure of the sub-model finally collected can be made simpler by taking some modules from each network layer of the sub-model, thereby improving the efficiency of screening and training, and thus improving the efficiency of obtaining the final model.

[0126] In one possible embodiment, each network layer in SuperPLM may include a multi-head attention network, which includes multiple attention modules, each attention module has parameters of multiple dimensions, and the multi-head attention network is used to calculate the correlation between multiple input vectors and obtain an output vector including contextual semantics based on the correlation. When collecting sub-models from SuperPLM, such as collecting the second sub-model or the third sub-model, at least one attention module can be collected from the multiple attention modules in the multi-head attention network of each network layer of SuperPLM, and at least one dimension of parameters can be collected from the multiple dimensions of parameters in each collected attention module to obtain multiple sub-models, thereby collecting the structure of each attention module and part of the parameters of each attention module, so that the dimension of the attention head of the sub-model is consistent with the dimension of the attention head of the initial model; or, at least one parameter can be collected from each dimension of each attention module in the multi-head attention network of each network layer of the initial model to obtain multiple sub-models, so that the number of heads of the sub-model is consistent with the number of heads of the network layer in the initial model.

[0127] 504. Train the target model to obtain a trained target model.

[0128] After obtaining the structure of the target model, the target model can be constructed based on the structure of the target model, or the target model can be collected from SuperPLM based on the structure, and then the target model is trained to obtain a trained target model.

[0129] Therefore, in the embodiment of the present application, after obtaining the constraints on the resources occupied by the model, the model structure can be searched in the hyperparameter search space under the constraints of the resource constraints, so that the target model that meets the requirements can be obtained efficiently and quickly, and the target model can be trained to optimize the output effect of the model. It is equivalent to giving a specific resource constraint and efficiently searching to obtain the final target model based on the resource constraint. It can be understood that because the hyperparameters of at least two network layers of the target model are the same, it is equivalent to a homogeneous search to obtain the final target model. Compared with a heterogeneous search, the hyperparameter search space required in the search method provided by the present application is smaller, and the search efficiency is higher, so that the final target model is efficiently searched. In addition, the target model finally obtained meets the resource constraint conditions, so that the target model can be adaptively deployed in the hardware, meeting the deployment requirements of different hardware, and has strong generalization capabilities.

[0130] The above describes the process of the neural network training method provided in the present application. For ease of understanding, the following further describes the process of the neural network training method provided in the present application by way of example.

[0131] The neural network training method provided in this application can be applied to a variety of scenarios, such as search scenarios, voice assistants, news recommendations, or machine translation. For ease of understanding, the following examples illustrate possible application scenarios of the neural network training method provided in this application.

[0132] Scenario 1: Search Engine for News Recommendation

[0133] Currently, pre-trained language models have shown significant advantages in the search engine recall phase, which involves finding related information that users may be interested in by collecting information they are interested in. However, due to the huge number of web pages (e.g., over 100 billion) and the need for frequent model updates, the distillation method for large models and specific tasks is obviously inefficient to deploy. The method provided in this application can build an efficient general pre-trained language model, which can efficiently obtain a target model with a simpler model structure and performance close to the original model with a larger structure.

[0134] Scenario 2: Translation Scenario

[0135] The method provided in this application can be used to obtain all or part of a language model with a simpler structure, and the trained model can be deployed on a terminal with lower computing power. For example, it can be deployed on a terminal through an application. The user can input the corpus to be translated on the terminal display interface, and the deployed model outputs the contextual semantics of each part of the corpus, thereby converting it into an output corpus in a language different from that of the corpus.

[0136] Among them, in the neural network training method provided by this application, the training process of SuperPLM and the training of the target model can be performed by the same device. The embodiment of this application exemplarily divides the method provided by this application into two parts, including a pre-training stage and a search stage, which are introduced separately below.

[0137] For example, a more detailed process of another neural network training method provided in this application can be as follows: Figure 6 As shown below,

[0138] 1. Pre-training stage

[0139] In the pre-training phase, the initial model is trained using the training set to obtain a trained SuperPLM. For example, the pre-training phase may include the following steps 601 to 604.

[0140] 601. Divide the training set into multiple subsets, and collect sub-models from the initial model to obtain multiple first sub-models.

[0141] The training set may include multiple samples, and each sample may include a piece of corpus, or a representation vector of the corpus, etc.

[0142] The initial model can be a pre-built model with a larger structure, such as the initial model can include multiple layers of Transformer layers, as mentioned above Figure 2 As shown, the initial model can be obtained by stacking multiple Transformer layers. Multiple first sub-models can be obtained by randomly sampling or sampling from the initial model in a predetermined manner. For example, the depth or width can be randomly sampled, and then the corresponding sub-models can be sampled from the initial model based on the depth or width to obtain the first sub-model.

[0143] Specifically, the Transformer layer can include at least two substructures, namely a multi-head attention network and a forward network. The multi-head attention network can be regarded as a separable structure, which contains h (h is a positive integer) independent and parallel self-attention modules, or called heads. The output of the entire multi-head attention network is obtained by summing the outputs of h self-attention modules.

[0144] For example, each attention head can be described by four matrix parameters: and The input of the multi-head attention network is the output of the previous Transformer layer

[0145] For example, the output of the multi-head attention network can be defined as:

[0146]

[0147]

[0148]

[0149]

[0150] in, They are query matrix (Queries) | key matrix (keys) | value matrix (values), is the model parameter, H is the input representation of the current layer, ATTN(.) represents the dot product attention operation, and the output of each attention head is Finally, the output of each head is summed to obtain the output MHA(H).

[0151] In addition, the self-attention module can also include residual connections and layer normalization, which can be expressed as:

[0152] H MHA=LayerNorm(H+MHA(H))

[0153] Due to the existence of residual links, the dimension needs to satisfy {d q =d k ,d o =d m}. H MHA Represents the output of the entire self-multi-head attention network.

[0154] The Transformer layer also contains the forward network module, which can be expressed as:

[0155] H FFH =max(0,H MHA W 1 +b1)W 2 +b2

[0156] in, It is a model parameter. FFN also contains residual connections and standard layers. FFH Represents the output of the entire forward network module.

[0157] When collecting sub-models from the initial model, we can first define a set of receiving hyperparameters α = {l,d m ,d q ,d k ,d v ,d f ,d o}, extracting sub-models from the initial model in both depth and width. Specifically, we can extract the next l Transformer layers in depth, and extract in width from the parameter matrix of the initial model. The multi-head attention network in each Transformer layer can include multiple heads, each with a corresponding parameter matrix. A variety of methods can be used to extract sub-models.

[0158] The following is a detailed introduction to the method of collecting sub-models from the initial model.

[0159] For example, the hyperparameter search space can be set to the structural hyperparameter set α={l,d m ,d q ,d k ,d v ,d f ,d o}, extracting sub-models from the initial model in both depth and width dimensions. Typically, the multi-head attention network in each initial Transformer layer can include multiple attention modules, each of which can be understood as a head. Each attention module has parameters in multiple dimensions. When extracting sub-modules, various extraction methods can be used. The following examples illustrate some possible extraction methods.

[0160] 1. Extract the number of heads, and the dimension of the head remains unchanged

[0161] For example, Figure 7 As shown in Figure 1, some heads are extracted from the multi-head attention network, and then the matrix parameters of each head in the part are extracted, thereby obtaining a sub-model with a reduced number of heads and an unchanged head dimension.

[0162] 2. Dimension of the extraction head

[0163] For example, Figure 8 As shown in Figure 1, all heads can be extracted from the multi-head attention network, and part of the parameters can be extracted from each head, so as to obtain a sub-model with the number of heads remaining unchanged and the head dimension reduced.

[0164] Therefore, in the embodiment of the present application, the parameters of each head in the extracted sub-network are independent of each other and there is no overlapping part, thereby avoiding the inconsistent gradients for the head obtained by optimizing different training batches due to the overlap of head parameters, and effectively realizing model optimization.

[0165] 602. Distribute the plurality of subsets and the plurality of first sub-models to a plurality of threads, so as to calculate the gradients of the plurality of first sub-models in the plurality of threads.

[0166] After obtaining multiple subsets and multiple first sub-models, the multiple subsets and multiple first sub-models are respectively assigned to multiple computing nodes, such as assigned to multiple threads or multiple processes for processing, so that the gradient of each first sub-model is calculated by multiple computing nodes.

[0167] The training of the initial model can be divided into multiple epochs. In each epoch, the training set can be divided into multiple batches. In each batch, the training set can be divided into multiple subsets. Multiple sub-models are collected from the initial model, and the gradients of each sub-model are calculated separately. Then, the gradients of each model are used to update the initial model.

[0168] It should be noted that the embodiment of the present application is only illustrative of the updating of the initial model. The initial model mentioned in the embodiment of the present application can be replaced by the model output by the previous epoch.

[0169] Therefore, in the embodiment of the present application, the pre-training efficiency of the model can be improved by parallel processing of multiple threads, and the SuperPLM can be trained efficiently.

[0170] 603. Fuse the gradients of the multiple first sub-models to obtain a fused gradient, and use the fused gradient to update the initial model.

[0171] After each thread calculates the gradients of each sub-model, one of the threads, another thread, or another process can fuse the gradients of the sub-models to obtain a fused gradient. Fusion methods include, but are not limited to, weighted fusion, summation, or averaging. The fused gradients are then used as the gradients of the initial model, which is then updated to obtain an updated initial model.

[0172] 604. Determine whether to terminate the iteration. If so, execute step 605; if not, execute step 602.

[0173] After the initial model is updated, it is determined whether to terminate the iteration. If not, the model output by the current iteration can be used as a new initial model for updating. If the iteration is terminated, the search phase can be continued.

[0174] Specifically, determining whether the iteration is terminated may include determining whether a termination condition is satisfied. If the termination condition is satisfied, the search phase may continue. The termination condition may include, but is not limited to, one or more of the following: the number of iterations reaches a predetermined number of iterations, the output accuracy of the model reaches a predetermined output accuracy, or the average accuracy of the model reaches a predetermined average accuracy.

[0175] In the pre-training phase, the training data can be divided into multiple epochs, each epoch into multiple batches, and each batch into multiple sub-batches. Each sub-batch is then assigned to a corresponding thread. Multiple structural sampling and forward and backward calculations are performed in each thread to obtain the gradient of each sub-model. The gradients calculated by each thread are then merged and the parameters of the initial model are updated to obtain the updated model output for each batch. After the iteration is terminated, the trained SuperPLM is output.

[0176] Therefore, in the embodiments of the present application, during the pre-training phase, the gradients of the sub-models can be calculated in parallel to improve the efficiency of pre-training, thereby efficiently obtaining a SuperPLM with better output. Furthermore, when collecting sub-models from the initial model, the heads of the sub-models do not overlap, thereby avoiding gradient conflicts in each batch caused by overlapping gradients corresponding to the parameters of the sub-models. This, in turn, improves the training efficiency of the initial model and allows for more accurate optimization, resulting in a SuperPLM with better output.

[0177] 2. Search Phase

[0178] In the search phase, the optimal model structure can be obtained as the model structure of the target model by searching in the hyperparameter search space. For example, the search phase can refer to the following steps 605 to 609.

[0179] 605. Obtain resource constraints and hyperparameter search space.

[0180] Among them, the resource constraint condition can be a self-generated condition, or it can be obtained based on the information carried in the data received from the user input. For example, the resource constraint condition can be a range calculated based on the computing power of the device where the target model needs to be deployed, including the range of resources occupied when running the target model, such as latency or memory usage. For another example, the user can input user requirements through the client, and the method provided in this application can be executed by the server. After the server receives the information from the client, it can extract the user requirements and generate resource constraints.

[0181] The hyperparameter search space can include hyperparameters required for model construction, such as the model's depth range or width range. The depth range refers to the range of the number of network layers in the model, and the width range refers to the range of the number of input and output channels in the network layer. Generally, the range of structural parameters included in the hyperparameter search space does not exceed the range covered by SuperPLM's structural parameters. For example, if SuperPLM has 100 network layers, the depth range included in the structural search space will also not exceed 100, thus preventing the subsequent acquisition of completed sub-models.

[0182] For example, the hyperparameter search space may be pre-set to include a set of structural parameters α = {l,d m ,d q ,d k ,d v ,d f ,d o}, the hyperparameter variables are restricted to {d q =d k ,d o =d m}, can be used to collect sub-models from SuperPLM in terms of depth and width. Usually, in some common structure search methods, the hyperparameters can be set to {d q|k|v|o =d m ,d f =4d m}, the value range of each variable is Therefore, the overall hyperparameter search space can be expressed as: Obviously, through the method provided by this application, the size of the required hyperparameter search space is significantly smaller than the commonly used structural search method, the search efficiency is higher, and the final target model can be obtained efficiently.

[0183] 606. Under the constraints of resource constraints, search for at least one model structure from the hyperparameter search space.

[0184] Among them, the resource constraint condition can be used as a constraint on the resources occupied by the collected sub-models, so as to search for a model structure whose occupied resources are within the required range from the hyperparameter search space.

[0185] Specifically, multiple model structures can be obtained from the hyperparameter search space, and then the multiple model structures are screened under the constraints of resource constraints to select at least one model structure that meets the resource constraints, or a candidate set of model structures.

[0186] In one possible implementation, sub-models may be constructed based on the multiple model structures obtained through the search, and then the multiple sub-models may be run in pre-set hardware to screen out at least one model structure corresponding to a model whose occupied resources are within the resource constraint range.

[0187] In one possible implementation, corresponding sub-models can be collected from SuperPLM based on the multiple model structures obtained through the search, and then the multiple sub-models can be run on pre-set hardware to screen out at least one model structure corresponding to a model whose occupied resources are within the resource constraint range, thereby obtaining a model that meets user needs.

[0188] It should be noted that the method of acquiring sub-models from SuperPLM can refer to the method of acquiring sub-models from the initial model in the aforementioned step 601, and the initial model can be replaced by SuperPLM, which will not be repeated here.

[0189] In one possible implementation, after obtaining at least one model structure under the constraints of resource constraints, in order to further reduce the model complexity, one or more modules can be collected from each network layer in the corresponding sub-model to obtain a new network layer, thereby obtaining at least one model structure with a simpler structure. For example, when the searched structural parameters are within a certain range, such as 1.6*d m ≤d f ≤1.9*d m ;0.7*d m ≤d q|k|v ≤1.0*d m , models with the same number of layers usually have similar structures, such as d m|f|q|k|v The values ​​are close. Therefore, the output effects of these models are similar, and modules can be extracted from each network layer to form a new network layer, whose output effect is close to the output effect of the sub-model before the module is extracted. For example, if the depth of the sub-model corresponding to the model structure is 4 and the width is 5, one of the modules can be extracted from each network layer in the 4-layer network as a new network layer, thereby obtaining a sub-model with a simpler structure and an output effect close to that of the sub-model with a complex structure. In other words, it is equivalent to obtaining a model with a simpler structure and no reduction in output effect, which is more conducive to model deployment and improves the model's generalization ability.

[0190] 607. Collect at least one second sub-model corresponding to at least one model structure from the super pre-trained language model.

[0191] Among them, the candidate set of model structures is screened under the constraints of resource constraints After that, the candidate set can be collected from the SuperPLM The sub-model corresponding to each model structure in is obtained Corresponding proxy sub-model

[0192] Among them, if the corresponding sub-model has been collected from SuperPLM in step 607, there is no need to collect the second sub-model from SuperPLM again. At least one second sub-model that meets the resource constraint conditions can be screened out from the multiple sub-models collected in step 607. Step 607 is an optional step.

[0193] 608. Obtain a target model according to the at least one second sub-model.

[0194] After obtaining at least one second sub-model, one of the models can be selected from the at least one second sub-model as the target model, or a model can be selected from the at least one second sub-model based on the at least one second sub-model. One of the model structures is selected as the target model structure.

[0195] Optionally, when After selecting one of the model structures as the target model structure, a model can be constructed based on the structure to obtain the target model, or the target model can be obtained by collecting the corresponding sub-model from SuperPLM based on the structure.

[0196] Specifically, the method of selecting a target model or target models may include: using samples in a test set as inputs for each second sub-model, then ranking at least one second sub-model based on the output of each second sub-model, and then selecting a target model from the at least one second sub-model based on the resource usage of each second sub-model. For example, a Pareto front may be calculated based on the output effect and resource usage of each second sub-model, and then a second sub-model that takes both output effect and resource usage into consideration may be selected as the target model, thereby obtaining a model that performs better in both output effect and resource usage.

[0197] 609. Train the target model to obtain a trained target model.

[0198] After the target model is screened out, in order to improve the output accuracy of the target model, the target model can be trained to obtain a trained target model.

[0199] Therefore, in the embodiment of the present application, in the pre-training stage, the method of parallel calculation of the gradient of the sub-model can be used to improve the efficiency of pre-training, thereby efficiently obtaining a SuperPLM with better output effect. In addition, when collecting sub-models from the initial model, the heads of the sub-models do not overlap, thereby avoiding the gradient conflict of each batch caused by the overlap of the gradients corresponding to the parameters of each sub-model, thereby improving the training efficiency of the initial model and more accurate optimization, and obtaining a SuperPLM with better output effect. In the search stage, a smaller parameter search space is constructed, so that the target model that meets the requirements and has better output effect can be efficiently searched under the constraints of resource constraints.

[0200] Further, see Figure 9 , the process of the neural network training method provided in this application is exemplified.

[0201] Take one of the Transformer layers in the initial model as an example, Figure 2 Similarly, the Transformer layer includes a multi-head attention network and a forward network. The multi-head attention network can include multiple heads, such as Figure 9 The W shown in v 、W k 、W kIn the pre-training phase, the initial model is trained to obtain SuperPLM. In the search phase, a proxy model can be searched from SuperPLM, and then the evaluator is used to evaluate the resource usage or output effect of the proxy model, and the search is performed based on the evaluation results. The search method may include evolutionary search, random search, or other search methods. If no model that meets the requirements is found, a new proxy model can be collected from SuperPLM and evaluated again until the optimal model α is found. opt , and then further train the searched model to obtain a target model with better output effect.

[0202] In addition, in the pre-training stage and the search stage, when collecting sub-models, some parameters, that is, parameters of some dimensions, can be collected from each head in the multi-head attention network, such as Figure 9 The lower right corner shows some of the collected parameters of one of the heads, making the parameters of each head independent of each other and reducing overlap, which can improve the accuracy of the gradients of each sub-model calculated in the pre-training phase and thus improve the training effect of the model.

[0203] For example, a more detailed process of the pre-training phase can be found in Figure 10 , as described below.

[0204] After entering the pre-training phase, data initialization 1001 is first performed.

[0205] The data that needs to be initialized during the initialization process may include but is not limited to: setting the candidate set A, the number of threads (or GPUs) N, the unlabeled dataset D (i.e., the training set), the number of training batches, the number of iterative training times E, the number of samples per training batch M, the number of samples per batch m, the initial parameters theta of the initial model, and the current iteration round t, etc.

[0206] Then determine whether the current iterative discussion t is greater than E, that is, step 1002.

[0207] If t>E, the pre-training is ended, that is, step 1008.

[0208] If t≤E, it can be determined whether the sampling of dataset D is completed, that is, step 1003. Generally, each batch of samples sampled from dataset D is different, so step 1003 is to determine whether all samples in dataset D have been used for training.

[0209] If so, the next iteration may be continued, ie, step 1002 is continued and t=t+1 is set.

[0210] If not, a batch may be sampled from the data set D and the batch may be divided into multiple sub-batches, that is, step 1004 is executed.

[0211] It is determined whether the number of sampling times m from the data set D exceeds M, that is, step 1005 is executed.

[0212] If so, N sub-models can be sampled from the initial model and distributed to N threads. The sub-models are trained independently in each of the N threads to obtain the gradients of the N sub-models. The number of iterations is set to be increased by 1, that is, m = m + 1. Then, step 1006 is executed.

[0213] In addition, the method of collecting sub-models from the initial model can refer to the above Figure 8 or Figure 9 The method shown in the figure makes the parameters of each sub-model independent of each other, which is more conducive to model optimization.

[0214] Then, a batch is collected from the dataset D and divided into multiple sub-batches until m>M. The average gradient of each sub-thread is then used to update the model parameters of the initial model, that is, step 1007 is executed.

[0215] Then, step 1003 may be continued to be executed, that is, determining whether the sampling of the data set D in this iteration has been completed.

[0216] Therefore, in the implementation manner of the present application, in the pre-training stage, the initial model can be trained by independent parallel training of multiple threads, thereby improving the efficiency of pre-training in a parallel manner.

[0217] For example, the final target model can be obtained by searching through an evolutionary algorithm. Figure 11 As shown in the figure, after SuperPLM is trained and the specific delay or memory requirements input by the user are obtained, the hyperparameter search space is set, the model structure is sampled, and the model performance test is performed on the pre-set hardware to obtain the model structure parameter set that meets the user's requirements. Then based on SuperPLM, use the planning algorithm to Search for the optimal structure α opt .

[0218] Specifically, the detailed process of the evolutionary algorithm may include: first, a random initial set of hyperparameters is generated, and the corresponding sub-model is extracted from SuperPLM and tested in hardware. The hyperparameter combination with the best test results is used as the parent generation, and then the parent generation parameters are evolved (such as adjusting some parameters) to obtain a new generation of candidate structures. As the aforementioned testing and evolution process continues, the optimal model structure α is finally obtained. opt Then based on the optimal structure α opt The target model can be obtained by extracting the corresponding sub-model from SuperPLM or directly building the model.

[0219] Therefore, in the implementation of the present application, during the search phase, an evolutionary algorithm can be used to search for a better model under the constraints of resource constraints, thereby efficiently obtaining a target model that meets user needs and has better output effects. It can be understood that the search phase provided by the present application only considers homogeneous Transformer structures. Compared with the method that allows heterogeneous Transformer structures, the hyperparameter search space of the present application is equivalent to an exponential level. Reduced to linear level This greatly reduces the difficulty of searching, allowing for efficient search results that meet user needs and deliver superior output. Furthermore, the model structure derived from this homogeneous search is more hardware and software platform-friendly, making it easier to deploy. It can be deployed on devices without changing the original program, demonstrating strong generalization capabilities.

[0220] Furthermore, compared to the commonly used SuperPLM training method, the HAT baseline method exhibits significant performance differences when compared to models trained from scratch, especially on small models. Both the homogeneous search search space and multi-head attention extraction (SME) improve results for medium and large models. When combined with efficient batch training, SuperPLM achieves nearly the same performance as models trained from scratch on small models.

[0221] The above describes in detail the process of the neural network training method provided by the present application. The following describes the device for executing the neural network training method provided by the present application.

[0222] See Figure 12 , the present application provides a structural diagram of a neural network training device, the neural network training device may include:

[0223] A pre-training module 1201 is used to obtain a super pre-trained language model;

[0224] An acquisition module 1202 is configured to acquire resource constraints and a hyperparameter search space, wherein the resource constraints indicate constraints on resources occupied by the model, and the hyperparameter search space indicates a range of hyperparameters of the model;

[0225] A search module 1203 is configured to search for a target model structure from a hyperparameter search space under resource constraints, wherein at least two network layers of the target model have the same hyperparameters;

[0226] The training module 1204 is used to train the target model to obtain a trained target model.

[0227] In one possible implementation, the hyperparameters of each network layer in the target model are the same.

[0228] In one possible implementation, the pre-training module 1201 is specifically used to: construct a neural network to obtain an initial model; divide the training set into multiple subsets; collect sub-models from the initial model to obtain multiple first sub-models; use multiple subsets to train the multiple first sub-models in parallel to obtain a super pre-trained language model.

[0229] In one possible implementation, the pre-training module 1201 is specifically used to: distribute multiple subsets and multiple first sub-models to multiple computing nodes to calculate the gradients of the multiple first sub-models in the multiple computing nodes; update the parameters of the initial model according to the gradients of the multiple first sub-models to obtain a super pre-trained language model.

[0230] In one possible implementation, the pre-training module 1201 is specifically configured to: fuse the gradients of multiple first sub-models to obtain a fused gradient; and update the parameters of the initial model according to the fused gradient to obtain a super pre-trained language model.

[0231] In one possible implementation, the search module 1203 is specifically configured to: search for at least one model structure from a hyperparameter search space under resource constraints, wherein the hyperparameters of at least two network layers of the model corresponding to the at least one model structure are the same; collect at least one second sub-model corresponding to the at least one model structure from the super pre-trained language model; and screen out a model structure whose output meets preset conditions from the at least one model structure based on the at least one second sub-model to obtain the structure of the target model.

[0232] In one possible implementation, the search module 1203 is specifically used to: collect sub-models from the initial model to obtain at least one third sub-model; collect at least one module from each network layer of each third sub-model in at least one third sub-model to obtain at least one second sub-model.

[0233] In one possible implementation, the search module 1203 is specifically used to: search and obtain multiple model structures from the hyperparameter search space; obtain multiple running results of the models corresponding to the multiple model structures running in the hardware; and filter out the model structures whose running results meet the resource constraint conditions from the multiple model structures to obtain at least one model structure.

[0234] In one possible implementation, the search module 1203 is specifically used to: evaluate the output results of at least one second sub-model using a test set to obtain an evaluation result of each second sub-model; filter out the structure of a model that meets preset conditions from at least one model structure based on the evaluation result of each second sub-model; and use the structure of the model that meets the preset conditions as the structure of the target model.

[0235] In a possible implementation, the search module 1203 is further used to: construct a target model based on the structure of a model that meets preset conditions; or, collect corresponding sub-models from the super pre-trained language model to obtain the target model.

[0236] In one possible implementation, each network layer of the super pre-trained language model includes a multi-head attention network, which includes multiple attention modules, each attention module has parameters of multiple dimensions, and the multi-head attention network is used to calculate the correlation between multiple input vectors and obtain an output vector including contextual semantics based on the correlation.

[0237] The search module 1203 is specifically used to: collect at least one attention module from multiple attention modules in the multi-head attention network of each network layer of the super pre-trained language model, and collect parameters of at least one dimension from parameters of multiple dimensions to obtain multiple second sub-models; or, collect at least one parameter from each dimension in each attention module in the multi-head attention network of each network layer of the super pre-trained language model to obtain multiple second sub-models.

[0238] In a possible implementation, the acquisition module 1202 is specifically configured to: receive user input data; and acquire resource constraints from the user input data, where the resource constraints include constraints related to the model's latency or memory usage.

[0239] See also Figure 13 , a structural schematic diagram of another neural network training device provided in this application is described as follows.

[0240] The neural network training device may include a processor 1301 and a memory 1302. The processor 1301 and the memory 1302 are interconnected via a circuit. The memory 1302 stores program instructions and data.

[0241] The memory 1302 stores the aforementioned Figure 5-Figure 11 The program instructions and data corresponding to the steps in .

[0242] Processor 1301 is used to execute the above Figure 5-Figure 11 The method steps are performed by the neural network training device shown in any embodiment.

[0243] Optionally, the neural network training device may further include a transceiver 1303 for receiving or sending data.

[0244] In an embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium stores a program for generating a vehicle speed. When the program is running on a computer, the computer executes the above-mentioned Figure 5-Figure 11 The illustrated embodiments describe steps in a method.

[0245] Optionally, the aforementioned Figure 14 The neural network training device shown in is a chip.

[0246] The embodiment of the present application also provides a neural network training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned Figure 5-Figure 11 The method steps are performed by the neural network training device shown in any embodiment.

[0247] The present application also provides a digital processing chip. The digital processing chip integrates circuits and one or more interfaces for implementing the above-mentioned processor 1301 or the functions of the processor 1301. When the digital processing chip integrates a memory, the digital processing chip can perform the method steps of any one or more of the aforementioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the neural network training device in the above-mentioned embodiment according to the program code stored in the external memory.

[0248] The present application also provides a computer program product which, when executed on a computer, causes the computer to execute the aforementioned Figure 5-Figure 11 The illustrated embodiment describes the steps performed by the neural network training device in the method.

[0249] The neural network training device provided in the embodiment of the present application can be a chip, which includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figure 5-Figure 11 The neural network training method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0250] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0251] For example, see Figure 14 , Figure 14 A schematic diagram of the structure of a chip provided in an embodiment of the present application, which can be represented as a neural network processor NPU 140. NPU 140 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1403, which is controlled by controller 1404 to extract matrix data from memory and perform multiplication operations.

[0252] In some implementations, arithmetic circuit 1403 includes multiple processing engines (PEs). In some implementations, arithmetic circuit 1403 is a two-dimensional systolic array. Arithmetic circuit 1403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1403 is a general-purpose matrix processor.

[0253] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1408.

[0254] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 via direct memory access controller (DMAC) 1405. Input data is also transferred to unified memory 1406 via DMAC.

[0255] The bus interface unit (BIU) 1410 is used for interaction between the AXI bus, the DMAC, and the instruction fetch buffer (IFB) 1409 .

[0256] The bus interface unit 1410 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0257] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or transfer weight data to the weight memory 1402 or transfer input data to the input memory 1401.

[0258] The vector calculation unit 1407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0259] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in a subsequent layer in a neural network.

[0260] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;

[0261] Unified memory 1406, input memory 1401, weight memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0262] The operations of each layer in the recurrent neural network can be performed by the operation circuit 1403 or the vector calculation unit 1407.

[0263] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors for controlling the above Figure 5-Figure 11 The method is performed by programming an integrated circuit.

[0264] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0265] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0266] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0267] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0268] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0269] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A neural network training method, characterized in that: include: Get a super pre-trained language model; Obtaining a resource constraint condition and a hyperparameter search space, wherein the resource constraint condition indicates a constraint on resources occupied by the model, and the hyperparameter search space indicates a range of hyperparameters of the model; Searching for a target model structure from the hyperparameter search space under resource constraints, wherein at least two network layers of the target model have the same hyperparameters; Training the target model to obtain a trained target model; Searching for a target model structure from the hyperparameter search space under the resource constraint includes: Under the constraints of resource constraints, at least one model structure is searched from the hyperparameter search space, wherein the hyperparameters of at least two network layers of the model corresponding to each model structure in the at least one model structure are the same; Acquire at least one second sub-model corresponding to the at least one model structure from the super pre-trained language model; According to the at least one second sub-model, a model structure whose output result meets a preset condition is screened out from the at least one model structure to obtain the structure of the target model.

2. The method according to claim 1, characterized in that The obtaining of the super pre-trained language model includes: Build a neural network and obtain an initial model; Divide the training set into multiple subsets; Collecting sub-models from the initial model to obtain a plurality of first sub-models; The multiple subsets are used to train the multiple first sub-models in parallel to obtain the super pre-trained language model.

3. The method according to claim 2, characterized in that The using the multiple subsets to train the multiple first sub-models in parallel to obtain the super pre-trained language model includes: Distributing the plurality of subsets and the plurality of first sub-models to a plurality of computing nodes to calculate gradients of the plurality of first sub-models in the plurality of computing nodes; The parameters of the initial model are updated according to the gradients of the multiple first sub-models to obtain the super pre-trained language model.

4. The method according to claim 3, characterized in that The updating of the parameters of the initial model according to the gradients of the multiple sub-models to obtain the super pre-trained language model includes: Fusing the gradients of the multiple first sub-models to obtain a fused gradient; The parameters of the initial model are updated according to the fused gradient to obtain the super pre-trained language model.

5. The method according to claim 4, characterized in that The acquiring, from the super pre-trained language model, at least one second sub-model corresponding to the at least one model structure comprises: Collecting sub-models from the initial model to obtain at least one third sub-model; At least one module is collected from each network layer of each third sub-model in the at least one third sub-model to obtain the at least one second sub-model.

6. The method according to claim 1, characterized in that The step of searching for at least one model structure from the hyperparameter search space under the constraints of resource constraints includes: Obtaining multiple running results of models corresponding to multiple model structures running in hardware, wherein the multiple models are model structures obtained by searching the hyperparameter search space; A model structure whose running result meets the resource constraint condition is screened out from the multiple model structures to obtain the at least one model structure.

7. The method according to claim 1, characterized in that The step of screening out a model structure whose output result meets a preset condition from the at least one model structure according to the at least one second sub-model to obtain the structure of the target model includes: Evaluate the output result of the at least one second sub-model using the test set to obtain an evaluation result of each second sub-model; Screening out a model structure that meets the preset condition from the at least one model structure according to the evaluation result of each second sub-model; The structure of the model that meets the preset conditions is used as the structure of the target model.

8. The method according to claim 7, characterized in that The method further comprises: Constructing the target model according to the structure of the model that meets the preset conditions; Alternatively, a corresponding sub-model is collected from the super pre-trained language model to obtain the target model.

9. The method according to any one of claims 5 to 8, characterized in that Each network layer of the super pre-trained language model includes a multi-head attention network, which includes multiple attention modules, each attention module has parameters of multiple dimensions, and the multi-head attention network is used to calculate the correlation between multiple input vectors and obtain an output vector including contextual semantics based on the correlation; The collecting at least one second sub-model corresponding to the at least one model structure from the super pre-trained language model includes: Collecting at least one attention module from the multiple attention modules in the multi-head attention network of each network layer of the super pre-trained language model, and collecting parameters of at least one dimension from the parameters of the multiple dimensions to obtain multiple second sub-models; Alternatively, at least one parameter is collected from each dimension of each attention module in the multi-head attention network of each network layer of the super pre-trained language model to obtain multiple second sub-models.

10. The method according to any one of claims 1 to 4, characterized in that The resource acquisition constraint conditions include: Receive user input data; The resource constraint condition is obtained from the user input data, where the resource constraint condition includes a constraint condition related to the delay or memory occupation of the model.

11. A neural network training device, characterized in that: include: Pre-training module, used to obtain super pre-trained language models; An acquisition module, configured to acquire resource constraints and a hyperparameter search space, wherein the resource constraints indicate constraints on resources occupied by the model, and the hyperparameter search space indicates a range of hyperparameters of the model; A search module, configured to search for a target model structure from the hyperparameter search space under resource constraints, wherein at least two network layers of the target model have the same hyperparameters; A training module, used to train the target model to obtain a trained target model; The search module is specifically used to: Under the constraints of resource constraints, at least one model structure is searched from the hyperparameter search space, wherein the hyperparameters of at least two network layers of the model corresponding to the at least one model structure are the same; Acquire at least one second sub-model corresponding to the at least one model structure from the super pre-trained language model; According to the at least one second sub-model, a model structure whose output result meets a preset condition is screened out from the at least one model structure to obtain the structure of the target model.

12. The device according to claim 11, characterized in that The pre-training module is specifically used for: Build a neural network and obtain an initial model; Divide the training set into multiple subsets; Collecting sub-models from the initial model to obtain a plurality of first sub-models; The multiple subsets are used to train the multiple first sub-models in parallel to obtain the super pre-trained language model.

13. The device according to claim 12, characterized in that The pre-training module is specifically used for: Distributing the plurality of subsets and the plurality of first sub-models to a plurality of computing nodes to calculate gradients of the plurality of first sub-models in the plurality of computing nodes; The parameters of the initial model are updated according to the gradients of the multiple first sub-models to obtain the super pre-trained language model.

14. The device according to claim 13, characterized in that The pre-training module is specifically used for: Fusing the gradients of the multiple first sub-models to obtain a fused gradient; The parameters of the initial model are updated according to the fused gradient to obtain the super pre-trained language model.

15. The device according to claim 14, characterized in that The search module is specifically used to: Collecting sub-models from the initial model to obtain at least one third sub-model; At least one module is collected from each network layer of each third sub-model in the at least one third sub-model to obtain the at least one second sub-model.

16. The device according to claim 11, characterized in that The search module is specifically used to: Obtaining multiple running results of models corresponding to multiple model structures running in hardware, wherein the multiple models are model structures obtained by searching the hyperparameter search space; A model structure whose running result meets the resource constraint condition is screened out from the multiple model structures to obtain the at least one model structure.

17. The device according to claim 11, characterized in that The search module is specifically used to: Evaluate the output result of the at least one second sub-model using the test set to obtain an evaluation result of each second sub-model; Screening out a model structure that meets the preset condition from the at least one model structure according to the evaluation result of each second sub-model; The structure of the model that meets the preset conditions is used as the structure of the target model.

18. The device according to claim 17, characterized in that The search module is further used to: Constructing the target model according to the structure of the model that meets the preset conditions; Alternatively, a corresponding sub-model is collected from the super pre-trained language model to obtain the target model.

19. The device according to any one of claims 15 to 18, characterized in that Each network layer of the super pre-trained language model includes a multi-head attention network, which includes multiple attention modules, each attention module has parameters of multiple dimensions, and the multi-head attention network is used to calculate the correlation between multiple input vectors and obtain an output vector including contextual semantics based on the correlation; The search module is specifically used to: Collecting at least one attention module from the multiple attention modules in the multi-head attention network of each network layer of the super pre-trained language model, and collecting parameters of at least one dimension from the parameters of the multiple dimensions to obtain the multiple second sub-models; Alternatively, at least one parameter is collected from each dimension of each attention module in the multi-head attention network of each network layer of the super pre-trained language model to obtain the multiple second sub-models.

20. The device according to any one of claims 11 to 14, characterized in that The acquisition module is specifically used to: Receive user input data; The resource constraint condition is obtained from the user input data, where the resource constraint condition includes a constraint condition related to the delay or memory occupation of the model.

21. A neural network training device, characterized in that: The method comprises a processor coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 10 is implemented.

22. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method according to any one of claims 1 to 10.

23. A neural network training device, characterized in that: The method comprises a processing unit and a communication interface, wherein the processing unit obtains program instructions through the communication interface, and when the program instructions are executed by the processing unit, the method according to any one of claims 1 to 10 is implemented.

24. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images