A model structure determination method and related apparatus

Abstract

A method for determining a model structure is applied to the field of artificial intelligence technology. In the method, after a plurality of candidate network structures are added to a pre-trained model to obtain a first model, a network structure search process of the first model is constrained by pre-setting a performance constraint condition, a selection probability of each candidate network structure in the first model is obtained, and it is ensured that a network structure to be added selected based on the selection probability meets the performance constraint condition. In this way, when the network structure actually needed to be added in the pre-trained model is selected based on the selection probability of each candidate network structure, a network structure with appropriate performance consumption can be selected, so that the performance resources consumed by the second model obtained by adding the network structure meet the device requirements.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and related apparatus for determining a model structure. Background Technology

[0002] Currently, in the field of artificial intelligence, fine-tuning the pre-trained language model using training samples from the NLP task is an effective method for solving natural language processing (NLP) tasks. However, as the number of parameters in pre-trained language models increases, and with many NLP tasks to be solved, full parameter fine-tuning needs to be performed for each NLP task, and a complete set of fine-tuned model parameters needs to be saved.

[0003] Therefore, efficient model parameter tuning methods (hereinafter referred to as parameter-efficient methods) have emerged. In parameter-efficient methods, for each NLP task, the parameters of the pre-trained language model are kept unchanged, and only a small number of parameters added to the pre-trained language model are fine-tuned. These parameter-efficient methods achieve similar results to full-parameter fine-tuning on NLP tasks, and when handling multiple NLP tasks, they enable multiple NLP tasks to share the pre-trained language model parameters, with each NLP task only needing to save a small number of fine-tuned newly added parameters.

[0004] However, in current parameter-efficient methods, the number of parameters that need to be fine-tuned (i.e., the number of newly added parameters) still accounts for a relatively high proportion of the entire pre-trained language model. This results in a large number of parameters in the pre-trained language model that need to be fine-tuned and saved for each NLP task, which consumes a lot of processing device performance resources. Summary of the Invention

[0005] This application provides a method and related apparatus for determining a model structure, which enables the performance resources consumed by the determined model to meet equipment requirements and reduces the performance resources consumed by the model during application.

[0006] The first aspect of this application provides a method for determining the structure of a model, applied to determining the structure of a neural network model in the field of artificial intelligence. Specifically, the method includes: firstly obtaining a first model, which is obtained by adding multiple candidate network structures to a pre-trained model. Adding candidate network structures to the pre-trained model means inserting a candidate network structure at a certain position in the pre-trained model, thereby making the candidate network structure a part of the pre-trained model.

[0007] Then, based on performance constraints, the selection probability of each candidate network structure in the first model is determined. The performance consumption increment of the first model satisfies the performance constraints, and the performance consumption increment is related to the selection probability of each candidate network structure and the performance consumption of each candidate network structure.

[0008] In simple terms, the performance constraint requires that the performance consumption increment of the first model be within a certain range, and this performance consumption increment is related to the selection probability and performance consumption of each candidate network structure. Since the performance consumption of each candidate network structure is fixed, the performance constraint is actually used to constrain the selection probability of each candidate network structure, so that the finally selected candidate network structure can meet the requirements.

[0009] Secondly, based on the selection probability of each candidate network structure, at least one candidate network structure is selected from multiple candidate network structures. Generally speaking, the higher the selection probability of a candidate network structure, the more likely it is to be selected.

[0010] Finally, at least one candidate network structure is added to the pre-trained model to obtain the second model. Specifically, after selecting at least one candidate network structure, referring to the position where each candidate network structure was added in the first model, the at least one candidate network structure is added to the same position in the pre-trained model to obtain the second model.

[0011] In this scheme, after adding multiple candidate network structures to the pre-trained model to obtain the first model, the network structure search process of the first model is constrained by pre-setting performance constraints. This allows us to obtain the selection probability of each candidate network structure in the first model and ensures that the network structure to be added based on this selection probability meets the performance constraints. In this way, when selecting the network structure actually needed to be added to the pre-trained model based on the selection probabilities of each candidate network structure, we can select a network structure with appropriate performance consumption, ensuring that the performance resources consumed by the second model obtained by adding network structures meet the device requirements.

[0012] In one possible implementation, the performance cost of each candidate network structure is related to the target parameters of each candidate network structure, which include one or more of the following parameters: the number of network structure parameters, the computational cost of the network structure, and the computation time of the network structure.

[0013] In other words, the performance consumption of a candidate network structure can be related to one parameter or multiple parameters. For example, when the performance consumption of a candidate network structure is only related to the number of network structure parameters, the larger the number of network structure parameters, the greater the performance consumption of the candidate network structure. That is, adding this candidate network structure to the pre-trained model consumes more storage resources from the processing device.

[0014] For example, when the performance consumption of a candidate network structure is related to multiple parameters (such as the number of network structure parameters, the computational cost of the network structure, and the computation time of the network structure), the performance consumption of the candidate network structure can be represented by the weighted sum of the multiple parameters corresponding to the candidate network structure.

[0015] In one possible implementation, the performance constraint mentioned above is that the performance consumption increment of the first model is not higher than a preset increment.

[0016] Since the performance cost of each candidate network structure is fixed, the higher the selection probability of each candidate network structure, the greater the performance cost increment of the first model. Therefore, given the performance cost increment of the first model is set to not exceed a preset increment as a performance constraint, it is necessary to assign an appropriate selection probability to each candidate network structure in the first model so that the performance cost increment of the first model can meet the performance constraint.

[0017] In one possible implementation, at least one candidate network structure is selected from multiple candidate network structures based on the selection probability of each candidate network structure. Specifically, this includes selecting at least one candidate network structure from multiple candidate network structures in descending order of selection probability; wherein the sum of the performance consumption of at least one candidate network structure does not exceed a preset increment.

[0018] In other words, in this scheme, after adding at least one selected candidate network structure to the pre-trained model, the increased performance consumption of the pre-trained model will not exceed the preset increment, ensuring that the model after adding the candidate network structure can meet the performance constraints.

[0019] In one possible implementation, when selecting candidate network structures, the selection can be based on the selection probability of each candidate network structure, randomly choosing from multiple candidate network structures to obtain at least one candidate network structure. Furthermore, during the selection process, it can be ensured that the sum of the performance costs of the at least one selected candidate network structure does not exceed a preset increment.

[0020] In one possible implementation, the multiple candidate network structures include multiple types of network structures, and each type of network structure includes one or more candidate network structures.

[0021] That is, multiple candidate network structures can be divided into multiple parts. Candidate network structures in the same part belong to the same type of network structure, while candidate network structures in different parts belong to different types of network structures. For example, suppose 100 candidate network structures are added to the pre-trained model. Among these 100 candidate network structures, 10 are MLP structures, 40 are modules that include addition operations, and 50 are modules that include multiplication operations.

[0022] In this approach, by combining the network structures used in various parameter-efficient methods and various possible neural network structures as multiple candidate network structures to be added to the pre-trained model, the diversity of candidate network structures can be effectively improved, thereby improving the performance of the model obtained by adding candidate network structures.

[0023] In one possible implementation, several candidate network structures of different types are deployed in the same location in the first model.

[0024] Specifically, for any type of candidate network structure, there are often certain restrictions on where this type of candidate network structure can be added to the pre-trained model. That is, a specific type of candidate network structure can only have a positive effect on the model's performance when added to specific positions in the pre-trained model. Therefore, when each type of candidate network structure has corresponding possible addition positions, the possible addition positions for some types of candidate network structures may overlap, allowing some different types of candidate network structures to be deployed in the same positions in the first model.

[0025] In one possible implementation, during the determination of the selection probability of each candidate network structure in the first model, the adjustment directions of the selection probabilities of multiple candidate network structures are competitive. The adjustment of the selection probabilities of multiple candidate network structures is guided by a common optimization objective, which is that the performance consumption increment of the first model satisfies the performance constraints. Thus, when adjusting the selection probabilities of multiple candidate network structures, their selection probabilities cannot all be adjusted in the same direction; that is, the selection probabilities of multiple candidate network structures cannot increase or decrease simultaneously. If the selection probability of some candidate network structures increases, due to the competitive relationship, the selection probabilities of other candidate network structures must necessarily decrease.

[0026] In this scheme, the adjustment directions of the selection probabilities of all candidate network structures in the first model are made competitive during the process of determining the selection probability of each candidate network structure. This allows all candidate network structures to compete and select the globally optimal set of candidate network structures, thus ensuring the performance of the final generated model.

[0027] In one possible implementation, the selection probability of each candidate network structure in the first model is determined based on performance constraints. Specifically, this includes: inputting training data into the first model and constructing a loss function based on the output of the first model, wherein the loss function is used to characterize the difference between the output of the first model and the true value of the training data; and updating the weight parameters and selection probabilities of multiple candidate network structures in the first model using gradient descent based on the loss function and performance constraints.

[0028] In one possible implementation, the method further includes: training a second model based on a training dataset to obtain a trained second model. During the training of the second model, the parameters belonging to the original structure of the pre-trained model in the second model can be kept fixed, while only the parameters of at least one newly added candidate network structure are adjusted, thereby achieving fine-tuning of some parameters.

[0029] In this scheme, after selecting the target network structure from multiple candidate network structures to determine the structure of the second model, the second model is retrained based on the training dataset. This allows for the adjustment of the parameters in the second model, further improving its performance and ensuring that the performance of the second model meets the accuracy requirements.

[0030] In one possible implementation, the second model is used to perform natural language processing tasks.

[0031] A second aspect of this application provides a device for determining a model structure, comprising:

[0032] The acquisition module is used to acquire the first model, which is obtained by adding multiple candidate network structures to the pre-trained model.

[0033] The processing module is used to determine the selection probability of each candidate network structure in the first model based on performance constraints, wherein the performance consumption increment of the first model satisfies the performance constraints, and the performance consumption increment is related to the selection probability of each candidate network structure and the performance consumption of each candidate network structure.

[0034] The processing module is also used to select at least one candidate network structure from multiple candidate network structures based on the selection probability of each candidate network structure.

[0035] The processing module is also used to add at least one candidate network structure to the pre-trained model to obtain a second model.

[0036] In one possible implementation, the performance cost of each candidate network structure is related to the target parameters of each candidate network structure, which include one or more of the following parameters: the number of network structure parameters, the computational cost of the network structure, and the computation time of the network structure.

[0037] In one possible implementation, the performance constraint is that the performance consumption increment of the first model is not higher than a preset increment.

[0038] In one possible implementation, the processing module is further configured to select at least one candidate network structure from multiple candidate network structures in descending order of selection probability.

[0039] Among them, the sum of the performance consumption of at least one candidate network structure is not higher than the preset increment.

[0040] In one possible implementation, the multiple candidate network structures include multiple types of network structures, and each type of network structure includes one or more candidate network structures.

[0041] In one possible implementation, several candidate network structures of different types are deployed in the same location in the first model.

[0042] In one possible implementation, the adjustment directions of the selection probabilities of multiple candidate network structures are competitive in the process of determining the selection probability of each candidate network structure in the first model.

[0043] In one possible implementation, the processing module is also used for:

[0044] The training data is input into the first model, and a loss function is constructed based on the output of the first model;

[0045] Based on the loss function and performance constraints, the weight parameters and selection probabilities of multiple candidate network structures in the first model are updated.

[0046] In one possible implementation, the processing module is also used to: train the second model based on the training dataset to obtain the trained second model.

[0047] In one possible implementation, the second model is used to perform natural language processing tasks.

[0048] A third aspect of this application provides a model structure determination apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any implementation thereof is implemented. For details regarding the steps in the various possible implementations of the first aspect executed by the processor, please refer to the first aspect; further details will not be repeated here.

[0049] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the first aspect described above.

[0050] The fifth aspect of this application provides a circuit system including a processing circuit configured to perform the method of any implementation of the first aspect described above.

[0051] The sixth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform any implementation of the first aspect described above.

[0052] A seventh aspect of this application provides a chip system including a processor for supporting a server or threshold value acquisition device in implementing the functions involved in any implementation of the first aspect described above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0053] The beneficial effects of the second to seventh aspects mentioned above can be referred to the introduction of the first aspect above, and will not be repeated here. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application;

[0055] Figure 2 This is a schematic diagram of another convolutional neural network structure provided in an embodiment of this application;

[0056] Figure 3 A schematic diagram of a system architecture 300 provided for an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application;

[0058] Figure 5 A flowchart illustrating a method for determining a model structure provided in an embodiment of this application;

[0059] Figure 6 A schematic diagram of a search space provided for an embodiment of this application;

[0060] Figure 7 This is a schematic diagram illustrating the selection probabilities of various candidate network structures obtained by performing a model structure search process under global constraints, as provided in an embodiment of this application.

[0061] Figure 8 This is a schematic diagram illustrating how a candidate network structure is selected based on the selection probability of each candidate network structure to construct a target model, as provided in an embodiment of this application.

[0062] Figure 9 This is a schematic diagram of a device for determining a model structure provided in an embodiment of this application;

[0063] Figure 10 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0064] Figure 11 A schematic diagram of the structure of a chip provided in an embodiment of this application;

[0065] Figure 12 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will understand that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0068] For ease of understanding, some technical terms involved in the embodiments of this application will be introduced below.

[0069] (1) Parameter fine-tuning

[0070] Based on a pre-trained model, the model is further trained using specific training samples for a specific task to slightly adjust the parameters in the pre-trained model.

[0071] (2) Full parameter fine-tuning

[0072] Full parameter tuning is a type of parameter tuning. Furthermore, full parameter tuning refers to tuning where all parameters in the pre-trained model can be adjusted.

[0073] (3) Pre-trained model

[0074] A pre-trained model is a neural network that has been trained on a large amount of data and stored. Simply put, a pre-trained model is a model created by predecessors to solve similar problems. Having a model created by predecessors means that when encountering a similar new problem, there is no need to train a new model from scratch; instead, the existing model can be used directly for simple learning to solve the new problem.

[0075] (4) Pre-trained language model

[0076] Pre-trained language models are a type of pre-trained model specifically designed for NLP tasks.

[0077] (5) Neural Networks

[0078] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0079]

[0080] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0081] (6) Deep Neural Networks

[0082] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three types based on their layer positions: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary: the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as... It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0083] (7) Convolutional Neural Network (CNN)

[0084] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or feature map. A convolutional layer refers to the layer of neural units in a CNN that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). In a convolutional layer of a CNN, a neural unit can be connected to only some neighboring neural units. A convolutional layer typically contains several feature planes, each of which can be composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations on the image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0085] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0086] Specifically, such as Figure 1 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0087] The structure consisting of the convolutional layer / pooling layer 120 and the neural network layer 130 can be the first convolutional layer and the second convolutional layer described in this application. The input layer 110 is connected to the convolutional layer / pooling layer 120, and the convolutional layer / pooling layer 120 is connected to the neural network layer 130. The output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform non-linear processing on the output of the neural network layer 130.

[0088] Convolutional / pooling layers: 120. Convolutional layers: such as... Figure 2The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0089] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0090] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0091] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0092] Pooling layers: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 1 In the example of 120, each layer 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0093] Neural Network Layer 130: After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required output information. As before, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layer 130 to generate one or a set of required class numbers of output. Therefore, neural network layer 130 may include multiple hidden layers (such as...). Figure 1 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0094] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0095] It should be noted that, as Figure 1 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 2The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0096] (8) Multi-Layer Perception (MLP)

[0097] MLP is a feedforward artificial neural network model used to map multiple input datasets to a single output dataset. Typically, an MLP consists of three neural network layers: an input layer, hidden layers, and an output layer, where there can be one or more hidden layers; furthermore, the different neural network layers in an MLP are fully connected, meaning that any neuron in one layer is connected to all neurons in the next layer.

[0098] (9) Attention Network

[0099] Attention networks are network models that utilize attention mechanisms to improve model training speed. Currently, typical attention networks include the Transformer model. Models applying attention mechanisms can assign different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence and resulting in a more accurate output.

[0100] In deep learning, attention mechanisms can be implemented using weight vectors that describe importance: when predicting or inferring an element, the weight vectors determine the correlation between that element and other elements. For example, for a pixel in an image or a word in a sentence, attention vectors can be used to quantitatively estimate the correlation between the target element and other elements, and the weighted sum of the attention vectors serves as an approximation of the target value.

[0101] The attention mechanism in deep learning simulates the human brain's attention mechanism. For example, when a human views a painting, although the eyes can see the entire picture, upon closer inspection, the eyes actually focus on only a portion of the image. At this point, the brain primarily focuses on this smaller area. In other words, when humans carefully observe an image, the brain's attention to the entire image is not uniform but rather differentiated by weight; this is the core idea of ​​the attention mechanism.

[0102] In simple terms, the human visual processing system often selectively focuses on certain parts of an image while ignoring other irrelevant information, thus aiding the brain's perception. Similarly, in the attention mechanism of deep learning, in problems involving language, speech, or vision, certain parts of the input may be more relevant than others. Therefore, through the attention mechanism in attention models, different processing can be applied to different parts of the input data, allowing the attention model to dynamically focus only on task-relevant data.

[0103] (10) Residual Network (ResNet)

[0104] Residual networks are an improved type of convolutional neural network characterized by their ease of optimization and the ability to improve accuracy by increasing depth considerably. The residual blocks within a residual network utilize skip connections, mitigating the vanishing gradient problem inherent in increasing depth in deep neural networks.

[0105] (11) Loss Function

[0106] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0107] (12) Backpropagation algorithm

[0108] Neural networks can use the back propagation (BP) algorithm to correct the initial parameters of the model during training, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss. By backpropagating this error loss information, the initial model parameters are updated, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0109] (13) Gradient descent

[0110] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate a model with the minimum bias. To find a local minimum of a function using gradient descent, iterative searches must be performed at points a specified step distance away in the opposite direction of the gradient (or approximate gradient) at the current point on the function. Gradient descent is one of the most frequently used methods for solving model parameters in machine learning algorithms, i.e., unconstrained optimization problems.

[0111] Specifically, when finding the minimum value of the loss function, we can use gradient descent to iteratively solve the problem step by step, obtaining the minimized loss function and model parameter values. Conversely, if we need to find the maximum value of the loss function, we need to use gradient ascent iteratively.

[0112] (14) Neural structure search

[0113] Neural architecture search is an automated method for designing neural networks, widely used in machine learning. It can automatically design high-performance network structures based on a sample set, sometimes even rivaling human experts in certain tasks, and may even discover previously unproposed network structures. This effectively reduces the cost of using and implementing neural networks. The principle of neural architecture search is to find the optimal neural network structure from a given set of candidate neural network structures called the search space using a certain strategy. The quality of a neural network structure can be measured using metrics such as accuracy and computational speed, a process known as performance evaluation.

[0114] (15) Natural Language Processing (NLP)

[0115] NLP is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. Natural Language Processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistics, but also important differences. Natural Language Processing does not simply study natural language in general; rather, it focuses on developing computer systems, especially the software systems, that can effectively achieve natural language communication. Therefore, it is a part of computer science.

[0116] Generally speaking, natural language processing is mainly applied to machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, and character recognition.

[0117] Currently, in the field of artificial intelligence, when solving NLP tasks based on pre-trained language models, fine-tuning the pre-trained language model using the training samples of the NLP task is an effective method. However, as the number of parameters in pre-trained language models increases, and with many NLP tasks to be solved, full parameter fine-tuning needs to be performed for each NLP task, and a complete copy of the fine-tuned model parameters needs to be saved.

[0118] To address the challenges of large parameter counts and frequent fine-tuning required for full parameter tuning, parameter-efficient methods have emerged. These methods, for each NLP task, maintain the parameters of the pre-trained language model unchanged and only fine-tune a small number of parameters added to the pre-trained model.

[0119] For example, the specific execution process of a current parameter-efficient method is described below.

[0120] First, in the pre-trained language model based on transformers, the computation between model parameters and neurons in the self-attention module of each layer can be simplified to the following form:

[0121] h = Wx + b

[0122] Where x represents the input neural unit, h represents the output neural unit, and W and b are the model's weight parameters.

[0123] Then, parameter-efficient modules are added to the self-attention modules of each layer of the pre-trained language model. The method for adding parameter-efficient modules is as follows:

[0124] h = Wx + b + BAx

[0125] That is, in addition to the computation between model parameters and neural units, computation between the parameters of the parameter-efficient module and neural units is added. Here, x represents the input neural unit, h represents the output neural unit, W and b are the parameters of the pre-trained language model, and B and A are the parameters of the parameter-efficient module.

[0126] Finally, when fine-tuning the pre-trained language model on downstream tasks, only the parameters of the parameter-efficient modules are fine-tuned (i.e., only parameters B and A are fine-tuned), while the parameters of the pre-trained language model remain unchanged (i.e., parameters W and b remain unchanged).

[0127] However, in current parameter-efficient methods, the number of parameters requiring fine-tuning (i.e., newly added parameters) still accounts for a relatively high proportion of the total pre-trained language model. For example, in the parameter-efficient method described above, the number of parameters requiring fine-tuning accounts for approximately 24% of the total parameters in the pre-trained language model. This results in a large number of parameters in the pre-trained language model that need to be fine-tuned and saved for various NLP tasks, consuming significant processing resources.

[0128] In view of this, embodiments of this application provide a method for determining a model structure. After adding multiple candidate network structures to a pre-trained model to obtain a first model, the network structure search process of the first model is constrained by pre-setting performance constraints. This allows the selection probability of each candidate network structure in the first model to be obtained, ensuring that the network structure to be added based on the selection probability meets the performance constraints. In this way, when selecting the network structure actually needed to be added to the pre-trained model based on the selection probabilities of each candidate network structure, a network structure with appropriate performance consumption can be selected, ensuring that the performance resources consumed by the second model obtained by adding the network structure meet the device requirements.

[0129] For ease of understanding, the system architecture applied to the model structure determination method provided in the embodiments of this application will be introduced below.

[0130] Please see Figure 3 This application provides a schematic diagram of a system architecture 300. For example... Figure 3 As shown, in this system architecture 300, the execution device 210 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The execution device 210 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 210 can use data from the data storage system 250 or call program code from the data storage system 250 to implement the model structure determination method provided in this application embodiment, thereby obtaining the model.

[0131] Users can interact with execution device 210 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0132] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0133] In one implementation, the execution device 210 is used to implement the model structure determination method provided in the embodiments of this application, and sends the obtained model to the local device 301 and the local device 302 through a communication network, so that the local device 301 and the local device 302 can deploy and run the model.

[0134] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, the local device 301 may provide local data or feedback calculation results to the execution device 210, or execute the method for determining the model structure provided in the embodiments of this application.

[0135] It should be noted that all the functions of execution device 210 can also be implemented by a local device. For example, local device 301 implements the functions of execution device 210 and provides services to its own users, or provides services to users of local device 302.

[0136] In summary, the method for determining the model structure provided in this application embodiment can be applied to electronic devices, such as the execution device 210, local device 301, or local device 302 described above. Exemplarily, the electronic device may be a server, a wireless electronic device in industrial control, a smartphone, a personal computer (PC), a laptop computer, a tablet computer, or other similar devices. For ease of understanding, the method provided in this application embodiment will be described below using the application of the method to a server as an example.

[0137] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application. Figure 4As shown, electronic device 101 includes processor 103, which is coupled to system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to system bus 105. System bus 105 is coupled to input / output (I / O) bus via bus bridge 111. I / O interface 115 is coupled to the I / O bus. I / O interface 115 communicates with various I / O devices, such as input device 117 (e.g., touchscreen), external storage 121 (e.g., hard disk, floppy disk, optical disk, or USB flash drive), multimedia interface, etc. A transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and moving digital video images), and an external USB port 125. Optionally, the interface connected to I / O interface 115 can be a USB interface.

[0138] The processor 103 can be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an ASIC.

[0139] Electronic device 101 can communicate with software deployment server 149 via network interface 129. Exemplarily, network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0140] Hard disk drive interface 131 is coupled to system bus 105. Hardware driver interface is connected to hard disk drive 133. Internal memory 135 is coupled to system bus 105. Data running in internal memory 135 may include operating system (OS) 137, applications 143, and schedules of electronic device 101.

[0141] An operating system consists of the Shell 139 and the kernel 141. The Shell 139 is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waiting for user input, interpreting user input for the operating system, and processing various operating system outputs.

[0142] Kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Kernel 141 interacts directly with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0143] For example, in the case where the electronic device 101 is a smartphone, the application 143 includes instant messaging related programs. In one embodiment, when the application 143 needs to be executed, the electronic device 101 can download the application 143 from the software deployment server 149.

[0144] Please see Figure 5 , Figure 5 This is a flowchart illustrating a method for determining a model structure according to an embodiment of this application. Figure 5 As shown, the method for determining the model structure includes the following steps 501-504.

[0145] Step 501: Obtain the first model, which is obtained by adding multiple candidate network structures to the pre-trained model.

[0146] In this embodiment, based on the pre-trained model, a first model can be obtained by adding multiple candidate network structures to the pre-trained model. Adding candidate network structures to the pre-trained model means inserting a candidate network structure at a certain position in the pre-trained model, thus making the candidate network structure a part of the pre-trained model.

[0147] For example, a candidate network structure can be inserted between any two adjacent neural network layers in a pre-trained model. The output of the preceding neural network layer becomes the output of the candidate network structure, and the output of the candidate network structure becomes the input of the following neural network layer. In other words, before adding the candidate network structure, the input of the following neural network layer in the pre-trained model is the output of the preceding neural network layer; after adding the candidate network structure, the input of the following neural network layer in the pre-trained model becomes the output of the added candidate network structure.

[0148] For a particular candidate network structure among multiple candidate network structures, the candidate network structure may be added between two adjacent neural network layers in the pre-trained model; the candidate network structure may also be added before or after a specific neural network layer, and this embodiment does not make specific limitations on this.

[0149] Furthermore, the candidate network structure can specifically be a neural network structure, such as an attention network structure, an MLP structure, a convolutional neural network structure, or a residual network structure. The candidate network structure can also be a module that includes specific computational operations, capable of performing specific operations on the input of the candidate network structure. For example, the candidate network structure can be a module that includes multiplication, addition, and / or normalization operations.

[0150] In summary, adding a candidate network structure at a certain position in a pre-trained model allows the operation corresponding to the candidate network structure to be performed on the input at that position, thereby obtaining the corresponding output.

[0151] In one possible implementation, the multiple candidate network structures added to the pre-trained model can all be of the same type, such as multiple candidate network structures being a module that includes addition operations, or multiple candidate network structures being MLP structures.

[0152] In another possible implementation, the multiple candidate network structures added to the pre-trained model can include various types of network structures, and each type of network structure includes one or more candidate network structures. That is, the multiple candidate network structures can be divided into multiple parts, with candidate network structures in the same part belonging to the same type of network structure, and candidate network structures in different parts belonging to different types of network structures. For example, suppose 100 candidate network structures are added to the pre-trained model, of which 10 are MLP structures, 40 are modules including addition operations, and 50 are modules including multiplication operations.

[0153] Generally, current parametrically efficient methods add the same network structure to different locations in the pre-trained model, and the network structures added by different parametrically efficient methods are often different. Therefore, in this embodiment, the network structures used by various parametrically efficient methods can be combined as multiple candidate network structures to be added to the pre-trained model, which can effectively improve the diversity of candidate network structures, thereby improving the performance of the model obtained by adding candidate network structures.

[0154] Optionally, among multiple candidate network structures, candidate network structures of different types can be deployed in the same location or in different locations. That is, some candidate network structures of different types can be deployed in the same location in the first model.

[0155] Specifically, for any type of candidate network structure, the placement of that type of candidate network structure in the pre-trained model is often limited. That is, a specific type of candidate network structure can only positively impact the model's performance when placed at specific locations within the pre-trained model. For example, type 1 candidate network structures can only be placed in linear layers of the pre-trained model; type 2 candidate network structures can only be placed in a single module of the pre-trained model; and type 3 candidate network structures can be placed between any two modules of the pre-trained model. Therefore, while each type of candidate network structure has corresponding placement locations, some types of candidate network structures may overlap, allowing some different types of candidate network structures to be deployed in the same locations within the first model.

[0156] Step 502: Based on performance constraints, determine the selection probability of each candidate network structure in the first model, wherein the performance consumption increment of the first model satisfies the performance constraints, and the performance consumption increment is related to the selection probability of each candidate network structure and the performance consumption of each candidate network structure.

[0157] In this step, the process of determining the selection probability of each candidate network structure in the first model can be performed synchronously during the training of the first model. That is, during the training of the first model, the selection probability of each candidate network structure in the first model is updated synchronously based on performance constraints, so that the performance consumption increment of the first model after training satisfies the performance constraints.

[0158] Specifically, before training the first model, a performance constraint is pre-set, and the training process of the first model is guided based on this performance constraint, so as to ensure that the performance consumption increment of the first model after training can meet the performance constraint.

[0159] The performance consumption increment of the first model can be the sum of the product of the selection probability of each candidate network structure and its performance consumption. The performance constraint is that the performance consumption increment of the first model should not exceed a preset increment.

[0160] Since the performance cost of each candidate network structure is fixed, the higher the selection probability of each candidate network structure, the greater the performance cost increment of the first model. Therefore, to ensure that the performance cost increment of the first model does not exceed the preset increment, each candidate network structure in the first model needs to be assigned an appropriate selection probability, so that the performance cost increment of the first model can meet the performance constraints.

[0161] For example, the performance consumption of each candidate network structure can be related to the target parameters of each candidate network structure, which include one or more of the following parameters: the number of network structure parameters, the computational cost of the network structure, and the computation time of the network structure.

[0162] For example, when the performance consumption of a candidate network structure is only related to the number of network structure parameters, the larger the number of network structure parameters of the candidate network structure, the greater the performance consumption of the candidate network structure. That is, adding the candidate network structure to the pre-trained model consumes more storage resources of the processing device.

[0163] For example, when the performance consumption of a candidate network structure is only related to the computational cost of the network structure, the greater the computational cost of the candidate network structure, the greater the performance consumption of the candidate network structure. In other words, adding the candidate network structure to the pre-trained model consumes more processing resources from the processing device.

[0164] For example, when the performance consumption of a candidate network structure is only related to the computation time of the network structure, the greater the computation time of the candidate network structure, the greater the performance consumption of the candidate network structure. That is, the longer it takes to consume the processing resources of the processing device when the candidate network structure is added to the pre-trained model.

[0165] As discussed above, the performance consumption of a candidate network structure is positively correlated with any one of the parameters. Therefore, when the performance consumption of a candidate network structure is only related to one parameter (e.g., the number of network structure parameters, the computational cost of the network structure, or the computation time of the network structure), the performance consumption of the candidate network structure can be represented by that parameter. For example, when the performance consumption of a candidate network structure is only related to the number of network structure parameters, the performance consumption of the candidate network structure can be represented by the number of network structure parameters.

[0166] When the performance consumption of a candidate network structure is related to multiple parameters (such as the number of network structure parameters, the computational cost, and the computation time), the performance consumption can be represented by a weighted sum of the various parameters corresponding to the candidate network structure. For example, if the performance consumption of a candidate network structure is related to the number of network structure parameters, the computational cost, and the computation time, and assuming that the number of network structure parameters is A, the computational cost is B, and the computation time is C, then the performance consumption of this candidate network structure can be a*A + b*B + c*C, where a, b, and c are the weights corresponding to the number of network structure parameters, the computational cost, and the computation time, respectively.

[0167] Optionally, during the training of the first model, the parameters of the original structure of the pre-trained model in the first model can be kept unchanged, while only the parameters of the multiple newly added candidate network structures in the first model are adjusted, thereby reducing the amount of parameter adjustment and alleviating the processing pressure on the processing device.

[0168] Step 503: Select at least one candidate network structure from multiple candidate network structures based on the selection probability of each candidate network structure.

[0169] In this embodiment, after determining the selection probability of each candidate network structure in the first model, at least one candidate network structure that actually needs to be added to the pre-trained model can be further selected from multiple candidate network structures based on performance constraints.

[0170] For example, in one possible implementation, at least one candidate network structure can be selected from multiple candidate network structures in descending order of selection probability. Furthermore, during the selection process, it is ensured that the sum of the performance costs of the selected at least one candidate network structure does not exceed a preset increment. In other words, after adding the selected at least one candidate network structure to the pre-trained model, the increased performance cost of the pre-trained model will not exceed the preset increment, ensuring that the model after adding the candidate network structure can meet the performance constraints.

[0171] In another possible implementation, when selecting candidate network structures, the selection can be based on the selection probability of each candidate network structure, randomly choosing from multiple candidate network structures to obtain at least one candidate network structure. It is worth noting that during the selection process, it can also be ensured that the sum of the performance costs of the selected at least one candidate network structure does not exceed a preset increment. For example, suppose the first model includes 100 candidate network structures, and the selection probabilities of these 100 candidate network structures are A1-A... 100Then, N candidate network structures can be randomly selected based on the selection probability of each candidate network structure, and the sum of the performance consumption of the selected N candidate network structures is guaranteed not to exceed the preset increment.

[0172] Step 504: Add at least one candidate network structure to the pre-trained model to obtain the second model.

[0173] After selecting at least one candidate network structure, referencing the position where each candidate network structure was added in the first model, add that at least one candidate network structure to the same position in the pre-trained model to obtain the second model.

[0174] In other words, compared to the first model, the second model actually retains at least one candidate network structure selected in the first model and removes other candidate network structures that were not selected. For example, if the first model is obtained by adding candidate network structures 1 to 100 to the pre-trained model, and assuming that the selected candidate network structures are candidate network structures 1 to 50, then the second model is actually obtained by adding candidate network structures 1 to 50 at the same positions in the pre-trained model, referencing the positions where candidate network structures 1 to 50 were added in the first model. That is, compared to the first model, the second model actually retains candidate network structures 1 to 50 and removes candidate network structures 51 to 100.

[0175] In this embodiment, the second model can be used to perform artificial intelligence tasks, such as NLP tasks, image processing tasks, speech recognition tasks, etc. For example, if the pre-trained model is a pre-trained language model, the second model can be used to perform NLP tasks. This embodiment does not limit the specific task type performed by the second model.

[0176] In this embodiment, after adding multiple candidate network structures to the pre-trained model to obtain the first model, the network structure search process of the first model is constrained by pre-setting performance constraints. This allows for the selection probability of each candidate network structure in the first model, ensuring that the network structures to be added based on these probabilities meet the performance constraints. Therefore, when selecting the network structures actually needed to be added to the pre-trained model based on the selection probabilities of each candidate network structure, a network structure with appropriate performance consumption can be selected, ensuring that the performance resources consumed by the second model obtained by adding network structures meet the device requirements.

[0177] It is understandable that the first model is obtained by adding multiple candidate network structures to a pre-trained model, while the second model is obtained by adding at least one candidate network structure from among multiple candidate network structures to a pre-trained model. That is, the structures of the first and second models are different. Therefore, even if the trained first model includes some structures identical to those in the second model, the parameters in the trained first model may not be directly applicable to the second model. Based on this, in this embodiment, after obtaining the second model, it can be retrained to ensure its performance.

[0178] For example, after obtaining the second model, it can be trained based on the training dataset to obtain the trained second model. The training dataset for training the second model can be the same as the training dataset for training the first model. Furthermore, the training dataset for training the second model can be specifically related to the task the second model is required to perform. For example, if the second model is used to process NLP tasks, the training dataset for training the second model can be a text dataset.

[0179] Optionally, during the training of the second model, partial parameter fine-tuning can be employed. Simply put, the second model is obtained by adding at least one candidate network structure to the original structure of the pre-trained model. Therefore, when training the second model, the parameters belonging to the original structure of the pre-trained model can be kept unchanged, while only the parameters of the newly added candidate network structure are adjusted, thereby achieving partial parameter fine-tuning.

[0180] In this scheme, after selecting the target network structure from multiple candidate network structures to determine the structure of the second model, the second model is retrained based on the training dataset. This allows for the adjustment of the parameters in the second model, further improving its performance and ensuring that the performance of the second model meets the accuracy requirements.

[0181] The above describes the process in this embodiment of training a first model with multiple candidate network structures based on performance constraints, selecting the actual network structure to be added based on the selection probability of each candidate network structure, and thus determining the final model structure. For ease of understanding, the process of training the first model based on performance constraints in step 502 above will be described in detail below.

[0182] Specifically, in any training iteration of the first model, training data can be input into the first model, and a loss function can be constructed based on the output of the first model. This loss function characterizes the difference between the output of the first model and the true value of the training data. For example, assuming the first model is used to handle translation tasks in NLP, the training data of the first model is English text, and the true value of the training data is the actual Chinese translation text corresponding to the English text, then the loss function can be constructed based on the Chinese translation prediction text output by the first model and the actual Chinese translation text, used to characterize the difference between the Chinese translation prediction text output by the first model and the actual Chinese translation text. Then, based on the loss function and performance constraints, the weight parameters and selection probabilities of multiple candidate network structures in the first model are updated. Specifically, the method for updating the weight parameters and selection probabilities of multiple candidate network structures in the first model can be gradient descent.

[0183] The above describes one round of training iterations for the first model. In reality, training the first model involves multiple rounds of training iterations. Therefore, training the first model essentially repeats the above training iteration process until the iteration termination condition is met. The iteration termination condition can be either that the accuracy of the trained first model reaches a preset accuracy requirement and that the performance consumption of the trained first model meets the performance constraints; or it can be that the number of training iterations for the first model reaches a preset number and that the performance consumption of the trained first model meets the performance constraints.

[0184] In summary, compared to traditional model training, the training objectives of the first model include not only improving the prediction accuracy of the first model, but also ensuring that the performance consumption increment of the first model after training can meet the performance constraints.

[0185] Optionally, in determining the selection probability of each candidate network structure in the first model, the adjustment directions of the selection probabilities of multiple candidate network structures are competitive. Specifically, for any candidate network structure, its selection probability is determined within the range [0, 1], meaning that the selection probability of any candidate network structure is greater than or equal to 0 and less than or equal to 1. Therefore, during the iterative training of the first model, the adjustment direction of the selection probability of each candidate network structure is to either increase or decrease it compared to the value in the previous iteration, i.e., decrease or increase the selection probability of the candidate network structure. However, due to performance constraints, it is impossible for the selection probabilities of all candidate network structures to continuously increase during the iteration process.

[0186] Therefore, in this embodiment, during the process of determining the selection probability of each candidate network structure in the first model, a competitive relationship is established between the adjustment directions of the selection probabilities of multiple candidate network structures. The adjustment of the selection probabilities of multiple candidate network structures is guided by a common optimization objective, which is that the performance consumption increment of the first model satisfies the performance constraints. In this way, when adjusting the selection probabilities of multiple candidate network structures, their selection probabilities cannot all be adjusted in the same direction; that is, the selection probabilities of multiple candidate network structures cannot increase or decrease simultaneously. If the selection probability of some candidate network structures increases, due to the existence of competition, the selection probabilities of other candidate network structures will inevitably need to decrease.

[0187] In this scheme, the adjustment directions of the selection probabilities of all candidate network structures in the first model are made competitive during the process of determining the selection probability of each candidate network structure. This allows all candidate network structures to compete and select the globally optimal set of candidate network structures, thus ensuring the performance of the final generated model.

[0188] To facilitate understanding, the following will provide a detailed explanation of the specific implementation process of the model structure determination method provided in this embodiment in practical applications, using concrete examples.

[0189] In this embodiment, taking a pre-trained model as an example, we introduce the process of determining the structure of a new model based on a pre-trained language model.

[0190] Currently, most pre-trained language models are built by stacking Transformers, resulting in nearly identical structures for each module. Different parameter-efficient methods differ only in the network structures added for fine-tuning within the pre-trained language model; however, the network structures added to each layer of the pre-trained language model are the same for each method, and each method has its own advantages and disadvantages. Therefore, this embodiment proposes a combined optimization approach by adding different network structures to different layers of the pre-trained language model to achieve better results.

[0191] Specifically, to expand the model's search space and integrate more parameter-efficient methods, this embodiment designs a unified paradigm for adding candidate network structures to the pre-trained language model. This paradigm unifies different parameter-efficient methods and various possible network structures. Specifically, the unified paradigm designed in this embodiment is shown in Equation 1.

[0192]

[0193] in, This refers to the neural unit that serves as the output (hereinafter referred to as the output neural unit); This refers to the neural unit that serves as input (hereinafter referred to as the input neural unit); This represents the neural units in the original pre-trained language model. To neural units Operation; Δ i Represents a candidate network structure; z i This is the weight parameter, which takes a value of 0 or 1; z i An equal value of 1 indicates that from a neural unit To neural units In addition to the mapping() operation, an additional operation is added (i.e., the computational operation corresponding to the candidate network structure); z i An equal value of 0 indicates that the signal originates from a neural unit. To neural units It still only has the mapping() operation; p i Indicate z i The probability that it equals 1.

[0194] In other words, the output neural unit is related not only to the original operations in the pre-trained language model, but also to the operations corresponding to newly added candidate network structures (depending on whether there are new candidate network structures between the input and output neural units). Therefore, the network structure search of the model can be transformed into the weight parameter z. i The search, that is, by searching z i This determines whether to add candidate network structures at specific locations.

[0195] In pre-trained language models, candidate network structures cannot be added between any two neural units; different types of candidate network structures can be added to different positions in the pre-trained language model.

[0196] Based on the computational operations corresponding to the candidate network structures used in this embodiment, after adapting various candidate network structures to the unified paradigm shown in Formula 1 above, we can obtain the unified paradigm, the representation of various candidate network structures, and the position of candidate network structures in the pre-trained language model.

[0197] For example, the correspondence between various candidate network structures and the unified paradigm can be shown in Table 1.

[0198] Table 1

[0199]

[0200] In Table 1, Δ represents the computational operation corresponding to the candidate network structure; H inIndicates the input neural unit; H out Represents the output neural unit; mapping() represents the output neural unit H in the original pre-trained language model. in To neural unit H out Operations; f() represents the activation function; A, B, W down W up ,b δ ,s δ The parameters represent the candidate network structure; W, b, s represent the parameters in the original pre-trained language model; Norm() represents normalization of the last dimension of the input.

[0201] Specifically, as shown in Table 1, adding the first type of candidate network structure to the pre-trained language model essentially involves adding a multiplication operation to the existing neural unit's computational operations. This means adding the result obtained from the multiplication operation to the original neural unit's computational result. Specifically, this multiplication operation (H...) in AB) is the multiplication of the input neuron value with parameters A and B.

[0202] Adding a second type of candidate network structure to a pre-trained language model involves adding an activation function operation to the existing neural unit operations. This means adding the result obtained from the activation function operation to the original neural unit computation. Specifically, the activation function operation (f(mapping(H))... in W down W up ) is the value of the input neuron and the parameter W. down The products are multiplied, and an activation function is used to calculate the product. Finally, the result of the activation function and the parameter W are obtained. up The product of.

[0203] Adding a third type of candidate network structure to a pre-trained language model involves adding an activation function operation to the existing neural unit operations. This means adding the result obtained from the activation function operation to the original neural unit computation result. Specifically, the activation function operation (f(H)) in W down W up This involves combining the computational results of the original neural unit with the parameter W. down The products are multiplied, and an activation function is used to calculate the product. Finally, the result of the activation function and the parameter W are obtained. up The product of.

[0204] Adding a fourth type of candidate network structure to the pre-trained language model involves adding a parameter value to the original neural unit's computational operations; that is, adding a parameter b to the original neural unit's computational result. δ .

[0205] Adding a fifth type of candidate network structure to the pre-trained language model involves adding a normalization operation to the existing neural unit operations. This means adding a result obtained from the normalization operation to the original neural unit computation results. Specifically, the normalization operation (Norm(H)) is... in )s δ ) performs a normalization operation on the value of the input neuron and obtains the normalization result and parameter s. δ The product of.

[0206] As shown in Table 1, the positions where each type of candidate network structure can be added to the pre-trained language model are usually fixed, and the positions that can be added by different types of candidate network structures are often different. For example, candidate network structure 1 can be added to any linear layer in the pre-trained language model; candidate network structure 2 can be added after any module in the pre-trained language model, such as after the feed forward neural network (FNN) module; candidate network structure 3 can be added between any two modules in the pre-trained language model, such as between the self attention module and the FNN module; candidate network structure 4 can be added to any linear layer in the pre-trained language model; and candidate network structure 5 can only be added to the layer normalization (LayerNorm) module in the pre-trained language model.

[0207] As shown in Table 1, after determining the candidate network structures that need to be added, we can determine the locations on the pre-trained language model where candidate network structures can be added, as well as the types of candidate network structures that can be added at each location, thus determining the search space of the model.

[0208] In simple terms, for any position in the pre-trained language model, if there are no candidate network structures that can be added at that position, then no candidate network structure is added at that position; if there are candidate network structures that can be added at that position, then all possible candidate network structures are added at that position. Therefore, among all the positions in the pre-trained language model where candidate network structures have been added, some positions may only have one candidate network structure added, while other positions may have multiple candidate network structures added.

[0209] For example, please refer to Figure 6 , Figure 6 This is a schematic diagram of a search space provided for an embodiment of this application. For example... Figure 6 As shown, the pre-trained language model includes multiple neural units, namely neural unit X0, neural unit X1, neural unit X2, and neural unit X3. Figure 6 In the diagram, solid lines represent the primitive computational operations between neural units in the pre-trained language model. Specifically, there are primitive computational operations between neural units X0 and X1, X0 and X2, X1 and X2, X1 and X3, X2 and X3, and X0 and X3. In other words, Figure 6 The CCP includes 6 locations with primitive computational operations.

[0210] In addition, Figure 6 In the diagram, different types of dashed lines represent the computational operations corresponding to different types of candidate network structures added to the pre-trained language model. Therefore, the locations in the pre-trained language model where candidate network structures can be added include: the position between neurons X0 and X1, the position between neurons X0 and X2, the position between neurons X1 and X2, the position between neurons X1 and X3, and the position between neurons X2 and X3. That is, there are a total of 5 locations in the pre-trained language model where candidate network structures can be added; the position between neurons X2 and X3 cannot add candidate network structures.

[0211] Furthermore, the types of candidate network structures that can be added at different locations are different. Specifically, the location between neural units X0 and X1 can only add two types of candidate network structures, namely type 1 and type 2; the location between neural units X0 and X2 can only add one type of candidate network structure, namely type 1; the location between neural units X1 and X2 can add two types of candidate network structures, namely type 1 and type 3; the location between neural units X1 and X3 can only add one type of candidate network structure, namely type 3; and the location between neural units X2 and X3 can only add two types of candidate network structures, namely type 2 and type 3.

[0212] Based on the above Figure 6It can be seen that by adding all possible candidate network structures to all possible locations in the pre-trained language model, the search space of the pre-trained language model can be obtained. After adding candidate network structures to the pre-trained language model, each added candidate network structure will be regarded as an independent candidate network structure. Subsequently, it is necessary to determine the selection probability corresponding to each candidate network structure, so as to select the candidate network structure to be retained based on the selection probability.

[0213] By unifying the various candidate network structures into a unified paradigm, the search space of the pre-trained language model is determined by adding multiple candidate network structures to the pre-trained language model.

[0214] Next, performance constraints are set to constrain the model structure search of the pre-trained language model so that the selection probability of each candidate network structure can meet the requirements of the performance constraints.

[0215] Specifically, the model structure search process of a pre-trained language model is actually to determine whether each newly added candidate network structure in the pre-trained language model needs to be selected (i.e. whether it needs to be retained), and then select a finite number of candidate network structures from all the newly added candidate network structures in the pre-trained language model so that the selected candidate network structures can meet the performance constraints.

[0216] In this embodiment, the selection probability of each candidate network structure is introduced to determine whether each candidate network structure needs to be selected, so that a finite number of candidate network structures can be selected from all the newly added candidate network structures in the pre-trained language model based on the selection probability of each candidate network structure.

[0217] Specifically, the process of determining the selection probability of each candidate network structure in the pre-trained language model can be carried out synchronously with the training process of the pre-trained language model. That is, the training objective of the pre-trained language model becomes: determining the weight parameters of the pre-trained language model and the selection probability of each candidate network structure.

[0218] For example, taking the performance constraint that the parameter consumption increment of the pre-trained language model is not higher than the target increment as an example (i.e., the total number of parameters of the newly added candidate network structures in the pre-trained language model is not higher than the target increment), the process of determining the selection probability of each candidate network structure in the pre-trained language model may include the following steps 1-4.

[0219] Step 1, introduce the adjustable parameter α iTo stabilize the optimization process of the selection probability of each candidate network structure in the pre-trained language model, a dynamic threshold ζ is introduced to constrain the performance consumption increment of the pre-trained language model. Specifically, the selection probability p of each candidate network structure can be obtained through Equation 2. i .

[0220]

[0221] Where, p i τ represents the selection probability of the i-th candidate network structure, where i is greater than or equal to 1; sigmoid() represents the sigmoid function, used to map variables to the range (0, 1); τ is a constant; α i ζ is an adjustable parameter; ζ is a dynamic threshold.

[0222] Step 2: Before training the pre-trained language model, initialize the weight parameters and adjustable parameters α of each candidate network structure. i .

[0223] Step 3, repeat steps a)-e) in a loop:

[0224] a) The dynamic threshold ζ is obtained by minimizing the difference between the expected value E[N] of the current parameter consumption increment of the pre-trained language model and the target increment B. Specifically, ζ = ζ can be obtained through the following formula 3-4. *

[0225] E[N]=∑ i p i |δ i |Formula 3

[0226] ζ * =argmin ζ (E[N]-B),whereE[N]≤B Formula 4

[0227] Where, |δ i | represents the number of network structure parameters corresponding to the i-th candidate network structure. Therefore, Equation 3 actually expresses the expected value E[N] of the current parameter consumption increment of the pre-trained language model, which is the sum of the product between the selection probability of each candidate network structure and the number of parameters. argmin ζ () indicates that a value of ζ is calculated so that E[N] is close to B.

[0228] b) From α i z is calculated from ζ i The probability p that equals 1 i :

[0229]

[0230] c) Construct a competitive relationship between different candidate network structures, and use... Replace p i Specifically, it is obtained through the following formula 5.

[0231]

[0232] Among them, Detach(p i The parentheses indicate that the gradient of the variable in parentheses is removed, and only the value of that variable is retained.

[0233] d) Since the training of neural networks is based on the gradient descent algorithm, each variable in the neural network needs to be continuously differentiable in order to calculate the gradient.

[0234] However, z i The value of z is only 0 and 1, and it is a discrete value, so z i If it is not differentiable, training based on gradient descent is impossible. Therefore, in this embodiment, we use... Replace z i , It is continuously differentiable, meaning that a differentiable approximate sample is used instead of the actual sample. This is obtained through the following Equation 6.

[0235]

[0236] Where σ() represents the distribution or dispersion of the average value of any process parameter; β is a hyperparameter constant; and log() represents the logarithmic function.

[0237] e) Calculate the loss function and gradient of the pre-trained language model through forward computation and backpropagation, and update the weight parameters and adjustable parameters α of each candidate network structure in the pre-trained language model based on the gradient. i .

[0238] 4. By repeatedly executing steps a)-e) in step 3 above, a pre-trained language model can be obtained. Each candidate network structure in this pre-trained language model has corresponding weight parameters and selection probability p. i .

[0239] In summary, the model structure search process essentially involves finding candidate network structures with fewer parameters that can guarantee model performance. Therefore, in this embodiment, the adjustable total number of parameters (i.e., the total number of parameters of all selected candidate network structures) is used as a global constraint—the objective increment B—to automatically search for the neural structure with the best performance. Since the number of parameters for each candidate network structure is fixed, i.e., the number of parameters for the i-th candidate network structure is |δ... iTherefore, by pre-setting the value of the target increment B, a global constraint can be implemented on the model structure search process based on the target increment B. Through the search algorithm under the above global constraints, the neural structure with the best search performance can be automatically searched.

[0240] For example, please refer to Figure 7 , Figure 7 This diagram illustrates the selection probabilities of various candidate network structures obtained during a model structure search process under global constraints, as provided in an embodiment of this application. Figure 7 As shown, Figure 7 Is Figure 6 The model structure search process was performed based on the pre-trained language model shown, thereby obtaining the selection probabilities of each newly added candidate network structure.

[0241] Specifically, the selection probabilities of the two candidate network structures added at the position between neural unit X0 and neural unit X1 are 0.67 and 0.41, respectively; the selection probability of the candidate network structure added at the position between neural unit X0 and neural unit X2 is 0.09; the selection probabilities of the two candidate network structures added at the position between neural unit X1 and neural unit X2 are 0.58 and 0.32, respectively; the selection probability of the candidate network structure that can be added at the position between neural unit X1 and neural unit X3 is 0.12; and the probabilities of the two candidate network structures that can be added at the position between neural unit X2 and neural unit X3 are 0.20 and 0.51, respectively.

[0242] After obtaining the selection probability of each candidate network structure, the selection probabilities of each candidate network structure can be sorted in descending order. Then, each candidate network structure is selected in descending order of selection probability, while keeping the total number of parameters of the selected candidate network structures less than the target increment B, that is, satisfying the performance constraint condition. Finally, the target model is obtained by adding the selected candidate network structures to the original pre-trained language model.

[0243] For example, please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating how a candidate network structure is selected based on the selection probability of each candidate network structure to construct a target model, as provided in an embodiment of this application. Figure 8 As shown, after determining the selection probability of each candidate network structure in the pre-trained language model, multiple candidate network structures can be selected to form the final target model in descending order of selection probability. Then, the final target model is obtained by removing the unselected candidate network structures.

[0244] exist Figure 8In the initial training, two candidate network structures with selection probabilities of 0.67 and 0.41 between neurons X0 and X1 were selected; a candidate network structure with a selection probability of 0.58 between neurons X1 and X2 was selected; and a candidate network structure with a selection probability of 0.51 between neurons X2 and X3 was selected. In other words, out of the eight candidate network structures added to the pre-trained language model, only four were ultimately selected.

[0245] Furthermore, after constructing the final target model based on the selected candidate network structures, the parameters of the newly added candidate network structures in the target model can be fine-tuned using the corresponding training dataset based on the specific downstream task to which the target model is applied (such as text translation task) in order to verify and improve the performance of the target model.

[0246] It should be noted that the above example, using the performance constraint that the parameter consumption increment of the pre-trained language model should not exceed the target increment (i.e., the total number of parameters of newly added candidate network structures in the pre-trained language model should not exceed the target increment), illustrates how to determine the selection probability of each candidate network structure in the pre-trained language model. In other words, when users are more concerned with the total number of parameters in the final searched model, using the parameter consumption increment of the pre-trained language model as a performance constraint can effectively determine the model structure that ultimately meets the parameter total requirement.

[0247] However, in some cases, users may be concerned with multiple metrics of the searched models simultaneously, such as the total number of parameters, computational cost, and computation time. That is, users hope that the total number of parameters, computational cost, and computation time of the final model are all within a certain range. Furthermore, the degree of concern users have for different model metrics may also vary. For example, while simultaneously caring about the total number of parameters, computational cost, and computation time, users may prefer that the total number of parameters be as small as possible.

[0248] In this context, in practical applications, performance constraints can be set according to actual needs. This ensures that when searching for model structures based on performance constraints, the search simultaneously considers multiple parameter indicators of each candidate network structure (such as the number of network structure parameters, computational cost, and computation time), ultimately yielding a model that meets all requirements.

[0249] For example, a performance constraint could be set such that the weighted sum of various parameters (e.g., number of network structure parameters, computational cost, and computation time) of multiple candidate network structures does not exceed the target increment B. Then, the performance cost of each candidate network structure can be calculated based on its corresponding number of parameters, computational cost, and computation time. For instance, assuming a candidate network structure has a number of network structure parameters A, a computational cost B, and a computation time C, its performance cost can be a*A + b*B + c*C, where a, b, and c are the weights corresponding to the number of network structure parameters, computational cost, and computation time, respectively.

[0250] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.

[0251] Please see Figure 9 , Figure 9 This is a schematic diagram of a model structure determination device provided in an embodiment of this application. Figure 9 As shown, the device for determining the model structure includes:

[0252] The acquisition module 901 is used to acquire the first model, which is obtained by adding multiple candidate network structures to the pre-trained model.

[0253] The processing module 902 is used to determine the selection probability of each candidate network structure in the first model based on performance constraints, wherein the performance consumption increment of the first model satisfies the performance constraints, and the performance consumption increment is related to the selection probability of each candidate network structure and the performance consumption of each candidate network structure.

[0254] The processing module 902 is also configured to select at least one candidate network structure from multiple candidate network structures based on the selection probability of each candidate network structure.

[0255] The processing module 902 is also used to add at least one candidate network structure to the pre-trained model to obtain a second model.

[0256] In one possible implementation, the performance cost of each candidate network structure is related to the target parameters of each candidate network structure, which include one or more of the following parameters: the number of network structure parameters, the computational cost of the network structure, and the computation time of the network structure.

[0257] In one possible implementation, the performance constraint is that the performance consumption increment of the first model is not higher than a preset increment.

[0258] In one possible implementation, the processing module 902 is further configured to select at least one candidate network structure from multiple candidate network structures in descending order of selection probability.

[0259] Among them, the sum of the performance consumption of at least one candidate network structure is not higher than the preset increment.

[0260] In one possible implementation, the multiple candidate network structures include multiple types of network structures, and each type of network structure includes one or more candidate network structures.

[0261] In one possible implementation, several candidate network structures of different types are deployed in the same location in the first model.

[0262] In one possible implementation, the adjustment directions of the selection probabilities of multiple candidate network structures are competitive in the process of determining the selection probability of each candidate network structure in the first model.

[0263] In one possible implementation, the processing module 902 is further configured to:

[0264] The training data is input into the first model, and a loss function is constructed based on the output of the first model;

[0265] Based on the loss function and performance constraints, the weight parameters and selection probabilities of multiple candidate network structures in the first model are updated.

[0266] In one possible implementation, the processing module 902 is further configured to: train the second model based on the training dataset to obtain the trained second model.

[0267] In one possible implementation, the second model is used to perform natural language processing tasks.

[0268] Please see Figure 10 , Figure 10 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1000 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. Specifically, the execution device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003, and a memory 1004 (wherein the execution device 1000 may have one or more processors 1003). Figure 10 (Taking a processor as an example), processor 1003 may include application processor 10031 and communication processor 10032. In some embodiments of this application, receiver 1001, transmitter 1002, processor 1003 and memory 1004 may be connected via bus or other means.

[0269] Memory 1004 may include read-only memory and random access memory, and provides instructions and data to processor 1003. A portion of memory 1004 may also include non-volatile random access memory (NVRAM). Memory 1004 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0270] Processor 1003 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0271] The methods disclosed in the embodiments of this application described above can be applied to processor 1003, or implemented by processor 1003. Processor 1003 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 1003 or by instructions in the form of software. The processor 1003 described above can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0272] The processor 1003 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1004. The processor 1003 reads information from memory 1004 and, in conjunction with its hardware, completes the steps of the above methods.

[0273] Receiver 1001 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1002 can be used to output digital or character information through the first interface; transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1002 may also include a display device such as a display screen.

[0274] The electronic device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the model structure determination method described in the above embodiments, or to cause the chip in the training device to execute the model structure determination method described in the above embodiments. Optionally, the storage unit can be an internal storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be an external storage unit located within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0275] For details, please refer to Figure 11 , Figure 11 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1100. The NPU 1100 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.

[0276] In some implementations, the arithmetic circuit 1103 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1103 is a two-dimensional pulsating array. The arithmetic circuit 1103 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1103 is a general-purpose matrix processor.

[0277] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1102 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1101 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1108.

[0278] Unified memory 1106 is used to store input and output data. Weight data is directly transferred to weight memory 1102 via Direct Memory Access Controller (DMAC) 1105. Input data is also transferred to unified memory 1106 via DMAC.

[0279] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1109.

[0280] The Bus Interface Unit (BIU) 1110 is used by the instruction fetch memory 1109 to fetch instructions from external memory, and also by the memory access controller 1105 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0281] The DMAC is mainly used to move input data from external memory DDR to unified memory 1106, or to move weight data to weight memory 1102, or to move input data to input memory 1101.

[0282] The vector computation unit 1107 includes multiple processing units that, when needed, further process the output of the computation circuit 1103, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0283] In some implementations, vector computation unit 1107 can store the processed output vector in unified memory 1106. For example, vector computation unit 1107 can apply a linear function, or a nonlinear function, to the output of computation circuit 1103, such as linear interpolation of feature planes extracted by convolutional layers, or, for example, a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 1107 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to computation circuit 1103, for example, for use in subsequent layers of the neural network.

[0284] The instruction fetch buffer 1109 connected to the controller 1104 is used to store the instructions used by the controller 1104;

[0285] Unified memory 1106, input memory 1101, weight memory 1102, and instruction fetch memory 1109 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0286] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0287] Please refer to Figure 12 , Figure 12 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 5 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0288] Figure 12 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0289] In one embodiment, the computer-readable storage medium 1200 is provided using a signal bearer medium 1201. The signal bearer medium 1201 may include one or more program instructions 1202, which, when executed by one or more processors, can provide the above-mentioned... Figure 5 The described function or part of the function.

[0290] In some examples, signal carrying medium 1201 may include computer-readable medium 1203, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0291] In some embodiments, the signal-bearing medium 1201 may comprise a computer-recordable medium 1204, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1201 may comprise a communication medium 1205, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1201 may be transmitted by a wireless form of communication medium 1205 (e.g., a wireless communication medium conforming to the IEEE 802.11 standard or other transmission protocols).

[0292] One or more program instructions 1202 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1202 conveyed to the computing device via a computer-readable medium 1203, a computer-recordable medium 1204, and / or a communication medium 1205.

[0293] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0294] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0295] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0296] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for determining a model structure, characterized in that, include: Obtain the first model, which is obtained by adding multiple candidate network structures to the pre-trained model; Based on performance constraints, the selection probability of each candidate network structure in the first model is determined, wherein the performance consumption increment of the first model satisfies the performance constraints, and the performance consumption increment is related to the selection probability of each candidate network structure and the performance consumption of each candidate network structure. Based on the selection probability of each candidate network structure, at least one candidate network structure is selected from the plurality of candidate network structures, wherein the sum of the performance consumption of the at least one candidate network structure is not higher than a preset increment; By adding at least one candidate network structure to the pre-trained model, a second model is obtained. The first model and the second model are used to process text processing tasks, image processing tasks, or speech recognition tasks.

2. The method according to claim 1, characterized in that, The performance consumption of each candidate network structure is related to the target parameters of each candidate network structure, which include one or more of the following parameters: number of network structure parameters, computational cost of network structure, and computation time of network structure.

3. The method according to claim 1 or 2, characterized in that, The performance constraint is that the performance consumption increment of the first model is not higher than a preset increment.

4. The method according to claim 1 or 2, characterized in that, The step of selecting at least one candidate network structure from the plurality of candidate network structures based on the selection probability of each candidate network structure includes: At least one candidate network structure is selected from the plurality of candidate network structures in descending order of selection probability.

5. The method according to claim 1 or 2, characterized in that, The plurality of candidate network structures includes multiple types of network structures, and each type of network structure includes one or more candidate network structures.

6. The method according to claim 5, characterized in that, Among the multiple candidate network structures, some of different types are deployed in the same position in the first model.

7. The method according to claim 1 or 2, characterized in that, In determining the selection probability of each candidate network structure in the first model, the adjustment directions of the selection probabilities of the multiple candidate network structures are in competition with each other.

8. The method according to claim 1 or 2, characterized in that, The determination of the selection probability of each candidate network structure in the first model based on performance constraints includes: The training data is input into the first model, and a loss function is constructed based on the output of the first model; Based on the loss function and the performance constraints, the weight parameters and selection probabilities of the multiple candidate network structures in the first model are updated.

9. The method according to claim 1 or 2, characterized in that, The method further includes: The second model is trained based on the training dataset to obtain the trained second model.

10. The method according to claim 1 or 2, characterized in that, The second model is used to perform natural language processing tasks.

11. A device for determining a model structure, characterized in that, include: The acquisition module is used to acquire the first model, which is obtained by adding multiple candidate network structures to the pre-trained model. The processing module is used to determine the selection probability of each candidate network structure in the first model based on performance constraints, wherein the performance consumption increment of the first model satisfies the performance constraints, and the performance consumption increment is related to the selection probability of each candidate network structure and the performance consumption of each candidate network structure. The processing module is further configured to select at least one candidate network structure from the plurality of candidate network structures according to the selection probability of each candidate network structure, wherein the sum of the performance consumption of the at least one candidate network structure is not higher than a preset increment. The processing module is further configured to add at least one candidate network structure to the pre-trained model to obtain a second model, wherein the first model and the second model are used to process text processing tasks, image processing tasks, or speech recognition tasks.

12. The apparatus according to claim 11, characterized in that, The performance consumption of each candidate network structure is related to the target parameters of each candidate network structure, which include one or more of the following parameters: number of network structure parameters, computational cost of network structure, and computation time of network structure.

13. The apparatus according to claim 11 or 12, characterized in that, The performance constraint is that the performance consumption increment of the first model is not higher than a preset increment.

14. The apparatus according to claim 11 or 12, characterized in that, The processing module is further configured to select at least one candidate network structure from the plurality of candidate network structures in descending order of selection probability.

15. The apparatus according to claim 11 or 12, characterized in that, The plurality of candidate network structures includes multiple types of network structures, and each type of network structure includes one or more candidate network structures.

16. The apparatus according to claim 15, characterized in that, Among the multiple candidate network structures, some of different types are deployed in the same position in the first model.

17. The apparatus according to claim 11 or 12, characterized in that, In determining the selection probability of each candidate network structure in the first model, the adjustment directions of the selection probabilities of the multiple candidate network structures are in competition with each other.

18. The apparatus according to claim 11 or 12, characterized in that, The processing module is further configured to: The training data is input into the first model, and a loss function is constructed based on the output of the first model; Based on the loss function and the performance constraints, the weight parameters and selection probabilities of the multiple candidate network structures in the first model are updated.

19. A device for determining a model structure, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1 to 10.

20. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10.

21. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images