Neural network model optimization method and related device

Abstract

The method obtains a second neural network model by processing a first neural network model, the second neural network model comprising an optimized attention layer and at least two preceding network layers. The input of an optimized query feature transformation module is obtained according to the output features of at least one preceding network layer of the optimized attention layer; the input of an optimized key feature transformation module is obtained according to the output features of at least one preceding network layer of the optimized attention layer; the input of an optimized value feature transformation module is obtained according to the output features of at least one preceding network layer of the optimized attention layer; and the input of at least one of the optimized query feature transformation module, the optimized key feature transformation module and the optimized value feature transformation module is obtained according to the output features of at least one non-adjacent preceding network layer of the optimized attention layer. The method enhances the expression capability of the second neural network model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method for optimizing a neural network model and related equipment. Background Technology

[0002] Neural network models can perform tasks such as object detection, object classification, machine translation, and speech recognition, and are therefore widely used in various fields such as security, transportation, and industrial production.

[0003] The Transformer network model is a deep neural network model that does not contain convolutional layers and consists entirely of self-attention layers, encoder-attention layers, and feed-forward layers. Thanks to the ability of the self-attention operation to extract features from the global receptive field, the Transformer network model has wide applications in computer vision, natural language processing, and other fields. The standard Transformer network structure is as follows: Figure 1a As shown, Figure 1a , Figure 1b This is a schematic diagram of the standard Transformer network model; the standard Transformer network model consists of 6 encoders and 6 decoders. Feedforward layer ( Figure 1a The F layer in the diagram consists of fully connected operations, responsible for extracting the correlation between data along the channel dimension; while the Self-Attention layer (…) Figure 1a The middle S layer mainly consists of Self-Attention operations, which extract features from another dimension of the data (patch or token). The encoder attention layer performs an attention operation between the encoder's output features and the decoder's intermediate features.

[0004] Furthermore, in the Transformer network model, the inputs to the query, key, and value feature transformation modules of all Self-Attention layers are exactly the same; that is, the inputs to these modules are the outputs of the previous layer. (Reference) Figure 1b Taking the self-attention layer as the sixth layer of the model as an example, the question feature transformation module of the self-attention layer ( Figure 1b Q in the key feature conversion module (Q) Figure 1b K) and value feature transformation module ( Figure 1bThe inputs to V in the Transformer network are all output features from the fifth layer of the model. Therefore, the Transformer network is generally cylindrical. This network design results in a fixed input connection method, which cannot guarantee good performance of the Transformer network on all tasks. Summary of the Invention

[0005] This application provides a neural network model optimization method and related equipment, which can optimize the neural network model, improve the model's expressive power, and thus enhance the model's performance.

[0006] A first aspect provides a neural network model optimization method, comprising the following steps: optimizing a first neural network model to obtain a second neural network model; wherein the second neural network model includes an optimized attention layer and at least two preceding network layers located before the optimized attention layer, the at least two preceding network layers being cascaded; the optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module, wherein the input of the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input of the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input of the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; wherein the input of at least one of the optimized query feature transformation module, the optimized key feature transformation module, and the optimized value feature transformation module is obtained based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer.

[0007] The optimization method in this embodiment of the invention optimizes a first neural network model to obtain a second neural network model. The second neural network model includes an optimized attention layer and at least two preceding network layers, each of which can be a single network layer or composed of multiple parallel network layers. The optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The input to the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input to the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; and the input to the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. At least one of these feature transformation modules is obtained based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer. This optimized second neural network model, by supplementing the inter-layer feature information of the network, enhances the model's expressive power and improves its task processing performance.

[0008] Optionally, the input to the target feature transformation module is at least one output feature of the attention layer in the preceding network layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0009] Optionally, the first neural network model includes a cascaded attention layer and at least two preceding network layers located before the attention layer, wherein the at least two preceding network layers are cascaded. The first neural network model is then optimized to obtain a second neural network model, specifically including the following steps: determining the search space of the first neural network model, wherein the elements in the search space include preceding network layers that can be connected to the first query feature transformation module, the first key feature transformation module, and the first value feature transformation module in the attention layer; determining the optimized attention layer using a search algorithm based on the search space, wherein the search algorithm is used to determine, according to search conditions, the first preceding network layer connected to the optimized query feature transformation module, the second preceding network layer connected to the optimized key feature transformation module, and the third preceding network layer connected to the optimized value feature transformation module, wherein at least one of the first, second, and third preceding network layers is a non-adjacent preceding network layer of the optimized attention layer.

[0010] In this embodiment of the invention, after determining the search space corresponding to the first neural network model, a search algorithm based on the search space is used to determine the optimized attention layer. Specifically, the optimized attention layer is defined as follows: the first preceding network layer connected by the optimized query feature transformation module, the second preceding network layer connected by the optimized key feature transformation module, and the third preceding network layer connected by the optimized value feature transformation module. At least one of the first, second, and third preceding network layers is a non-adjacent preceding network layer of the optimized attention layer. It is evident that the connection structure between the optimized attention layer and the preceding network layers differs from the connection structure between the attention layer and the preceding network layers in the first neural network model. The optimized attention layer focuses on inter-layer feature fusion; therefore, it enhances the expressive power of the second neural network model and helps improve its performance.

[0011] Optionally, the search algorithm may include any one of evolutionary algorithms, reinforcement learning algorithms, or network structure search algorithms.

[0012] Among them, evolutionary algorithms have a fast training convergence speed, which is beneficial to improving the optimization speed of the model.

[0013] Optionally, the elements in the search space may also include at least one of the following: an activation function of the first neural network model, a normalization operation of the first neural network model, an operation type of the feature map of the first neural network model, a number of parallel branches of the first neural network model, a number of modules in the search unit, and optional connection methods between the preceding network layers, excluding the attention layer.

[0014] In this embodiment of the invention, when determining the search space corresponding to the first neural network model, in addition to using the possible connection relationships between the attention layer and the preceding network layers as elements of the search space, the elements in the search space can also be expanded to improve the model performance of the second neural network model determined after the search.

[0015] Optionally, the input to the target feature transformation module is the input feature obtained by weighted summation of at least two output features of the previous network layers and the weights of the previous network layers in optimizing the attention layer; the target feature transformation module can be any one of the query feature transformation module, the key feature transformation module, or the value feature transformation module.

[0016] Optionally, the second neural network model further includes a first fusion module, a second fusion module, and a third fusion module. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the first fusion module, and the output of the first fusion module is connected to the input of the optimized query feature transformation module. The first fusion module is used to obtain the first input feature of the optimized query feature transformation module by performing weighted summation processing on the preceding network layers connected to it. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the second fusion module, and the output of the second fusion module is connected to the input of the optimized key feature transformation module. The input of the first fusion module is connected to the second fusion module; the second fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the second input feature of the optimized key feature transformation module. The parameters of the second fusion module include the second weights corresponding to the preceding network layers connected to it; the output of all preceding network layers of the optimized attention layer is connected to the input of the third fusion module, and the output of the third fusion module is connected to the input of the optimized value feature transformation module; the third fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the third input feature of the optimized value feature transformation module. The parameters of the second fusion module include the third weights corresponding to the preceding network layers connected to it.

[0017] In this embodiment of the invention, taking the optimized query feature transformation module as an example, a first fusion module is added before it. This first fusion module is used to perform a weighted summation of all preceding network layers of the optimized attention layer to obtain the first input features of the optimized query feature transformation module. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. Thus, by using the first fusion module, the features of the preceding network layers of the optimized attention layer can be fused to enhance the expressive power of the optimized attention layer and improve the performance of the second neural network model.

[0018] Optionally, any one of the first fusion module, the second fusion module, and the third fusion module includes any one of the following structures: a static weighting module, a multilayer perceptron module, and an attention module.

[0019] Secondly, this application also provides a neural network model, which includes an optimized attention layer and at least two preceding network layers located before the optimized attention layer. The at least two preceding network layers are connected in series. The optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The input of the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. The input of the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. The input of the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. At least one of the optimized query feature transformation module, the optimized key feature transformation module, and the optimized value feature transformation module receives its input based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer.

[0020] The neural network model provided in this embodiment of the invention, by supplementing the inter-layer feature information of the network, enhances the expressive power of the model and is beneficial to improving the task processing performance of the model.

[0021] Optionally, the input to the target feature transformation module is at least one output feature of the attention layer in the preceding network layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0022] Optionally, the input to the target feature transformation module is the input feature obtained by weighted summation of the output features of at least one or two preceding network layers and the weights of the preceding network layers of the optimized attention layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0023] Optionally, the neural network model further includes a first fusion module, a second fusion module, and a third fusion module. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the first fusion module, and the output of the first fusion module is connected to the input of the optimized query feature transformation module. The first fusion module is used to obtain the first input feature of the optimized query feature transformation module by performing weighted summation processing on the preceding network layers connected to it. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the second fusion module, and the output of the second fusion module is connected to the input of the optimized key feature transformation module. The input terminals of the block are connected; the second fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the second input feature of the input optimization key feature transformation module. The parameters of the second fusion module include the second weights corresponding to the preceding network layers connected to it; the output terminals of all preceding network layers of the optimization attention layer are connected to the input terminal of the third fusion module, and the output terminal of the third fusion module is connected to the input terminal of the optimization value feature transformation module; the third fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the third input feature of the input optimization value feature transformation module. The parameters of the second fusion module include the third weights corresponding to the preceding network layers connected to it.

[0024] Optionally, any one of the first fusion module, the second fusion module, and the third fusion module includes any one of the following structures: a static weighting module, a multilayer perceptron module, and an attention module.

[0025] Thirdly, this application also provides a model optimization apparatus, comprising: a processing module for optimizing a first neural network model to obtain a second neural network model; wherein the second neural network model includes an optimized attention layer and at least two preceding network layers located before the optimized attention layer, the at least two preceding network layers being connected in series, the optimized attention layer including an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module; the input of the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input of the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input of the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; wherein the input of at least one of the optimized query feature transformation module, the optimized key feature transformation module, and the optimized value feature transformation module is obtained based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer.

[0026] Fourthly, this application also provides a model optimization device, including a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the neural network model optimization method as described in the first aspect.

[0027] Fifthly, this application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the neural network model optimization method as described in the first aspect.

[0028] Sixthly, this application also provides a computer program product containing instructions that, when the computer program product is run on a computer, cause the computer to execute the neural network model optimization method described in the first aspect.

[0029] In a seventh aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the neural network model optimization method described in the first aspect.

[0030] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the neural network model optimization method described in the first aspect.

[0031] Eighthly, this application also provides a terminal device on which the neural network model described in the second aspect is running. Attached Figure Description

[0032] The accompanying drawings used in the embodiments of this application are described below.

[0033] Figure 1a , Figure 1b This is a schematic diagram of the standard Transformer network model;

[0034] Figure 2 This is a schematic diagram of the attention layer;

[0035] Figure 3 This is a schematic diagram of a system architecture provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application;

[0038] Figure 6This is a flowchart illustrating a neural network model optimization method provided in an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram of an optimized attention layer structure provided in an embodiment of the present invention;

[0040] Figure 8 This is a schematic diagram of the framework of an evolutionary algorithm provided in an embodiment of the present invention;

[0041] Figure 9a , Figure 9b , Figure 9c This is a schematic diagram of a network structure search process provided in an embodiment of the present invention;

[0042] Figure 10 This is a schematic diagram of the structure of an optimized Transformer network model provided in an embodiment of the present invention;

[0043] Figure 11 This is a schematic diagram of the framework of another evolutionary algorithm provided in an embodiment of the present invention;

[0044] Figure 12 This is a schematic diagram of another optimized attention layer structure provided in an embodiment of the present invention;

[0045] Figure 13a , Figure 13b , Figure 13c This is a schematic diagram of the structure of a first fusion module provided in an embodiment of the present invention;

[0046] Figure 14 This is a schematic diagram of the structure of a model optimization device provided in an embodiment of the present invention;

[0047] Figure 15 This is a schematic diagram of the structure of a model optimization device provided in an embodiment of the present invention;

[0048] Figure 16 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0050] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0051] (1) Neural Networks

[0052] Neural networks can be composed of neural units, which can refer to units represented by x. sThe arithmetic unit that takes an intercept of 1 as input can output the following:

[0053]

[0054] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0055] (2) Deep Neural Networks

[0056] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0057] (3) Convolutional Neural Networks

[0058] A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units within the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in an image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0059] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0060] (4) Recurrent Neural Networks

[0061] Recurrent Neural Networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains inadequate for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output. That is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. Theoretically, RNNs can process sequential data of any length. Training an RNN is similar to training a traditional CNN or DNN. This algorithm also uses the backpropagation algorithm, but with one key difference: when an RNN is expanded, its parameters, such as W, are shared; however, this is not the case with traditional neural networks as illustrated above. Furthermore, in gradient descent, the output at each step depends not only on the network at the current step but also on the states of the network in previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0062] Since we already have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. CNNs rely on the fundamental assumption that elements are independent of each other, and that input and output are also independent—like a cat and a dog. However, in the real world, many elements are interconnected. For example, stock prices fluctuate over time. Or, imagine someone saying, "I love traveling, and my favorite place is Yunnan. I definitely want to go there someday." Humans know the answer to this question is "Yunnan." Humans can infer from context, but how can machines do the same? This is where RNNs come in. RNNs aim to give machines the ability to remember, just like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0063] (5) Loss Function

[0064] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0065] (6) Backpropagation algorithm

[0066] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0067] (7) Attention

[0068] Attention operations extract weights from the vast amount of information contained in the data to describe the importance of features, and combine these weights with the features, allowing the network to focus on information with higher weights. The network layer that performs attention operations is called an attention layer; specifically, refer to... Figure 2 , Figure 2 This is a schematic diagram of the attention layer; the attention layer includes the first query feature transformation module (i.e. Figure 2 The Q module in the first key feature transformation module (i.e.) Figure 2 The K module), the first value feature transformation module (i.e. Figure 2 The V module in the attention layer, x is the input to the attention layer. Q x K x VAfter feature transformations by the Q module, K module, and V module respectively, it becomes K. out Q out V out According to K out and Q out Generate relevant weights, and then apply them to V based on the weight coefficients. out The weighted summation is performed, and the weighted feature `out` is output. Here, self-attention is a special type of attention operation, where self-attention occurs when the x-axis of the attention layer... Q =x K =x V When x = , the attention layer at this time is the self-attention layer.

[0069] refer to Figure 2 The basic process is described below. The input to the self-attention layer is x. First, three feature transformations are performed on x to obtain Q. out K out and V out :

[0070] Q out =xW Q K out =xW k V out =xW V (1)

[0071] In equation (1), x is the input of the self-attention layer, W Q W k W v W is the characteristic transformation matrix. Q W k W v In a convolutional neural network, these are fully connected layers, and their weights are learnable parameters. Let x be correlated with W... Q W K W v Multiply to get Q out K out and V out The goal is to map the x features to another dimension, so that the mapped features satisfy the Q-axis in the Attention operation. out K out and V out Requirements, W Q W K W v The functionality will gradually become apparent as the deep neural network training process progresses. Generally, after W... K After transformation, K is obtained out It can describe the content features of the input, representing what the input features are like; after W QQ after feature transformation out It can contain guiding features about the input, representing what information the model needs for processing; and after W v V after feature transformation out It is a vector that represents the input features.

[0072] Then Q out With K out Matrix multiplication (MatMul) is performed, followed by numerical scaling using a scaling layer. Optionally, in text processing, a mask layer is also used to process the scaled data during attention operations. Finally, after Softmax normalization, an Attention Map describing the autocorrelation between dimensions x is obtained, i.e., the Attention Map is:

[0073]

[0074] In equation (2), d is a constant, which is a custom value and is set according to the different needs of the model. Matrix multiplication is equivalent to finding guiding features within content features.

[0075] Finally, an Attention Map is used to apply V. out Weighted, i.e., V out The matrix multiplication with the Attention Map, followed by feature transformation, is used as the output of the attention layer. W in equation (3) out This is the characteristic transformation matrix.

[0076] out = AV out W out (3)

[0077] Based on the above introduction to the Attention operation, we can denote the Attention operation as:

[0078] Attention Q x K x V )

[0079] The Self-Attention (SA) operation, based on the Attention operation, sets the inputs of the Q module, K module, and V module to the same input x. The Self-Attention operation is denoted as:

[0080] SA() = Attention(x,x,x)

[0081] In existing technologies, network models that apply attention mechanisms, such as the Transformer network model, refer to... Figure 1a , Figure 1b In this network, the inputs to the feature transformation modules (question, key, and value) of all Self-Attention layers are the outputs of the previous layer. Therefore, the Transformer network has an overall "cylindrical" shape. This design results in a fixed input connection method, which cannot guarantee good performance for all tasks.

[0082] To address the aforementioned technical problems, this application provides a neural network model optimization method that optimizes a first neural network model to obtain a second neural network model. This second neural network model differs from the "straight cylinder" model in the prior art, as it considers the inter-layer feature information of the network, effectively enhancing the model's expressive power and improving its task processing performance.

[0083] In particular, the neural network model optimization method in this application embodiment can be applied to all neural network models with attention mechanisms, such as neural network models corresponding to task scenarios such as object detection, object classification, image classification, machine translation, speech recognition, and text recognition. By using the neural network model optimization method in this application embodiment to optimize the original neural network model, an optimized neural network model can be obtained, thereby improving the performance of the model.

[0084] Example 1

[0085] The system architecture provided in the embodiments of this application is described below.

[0086] See appendix Figure 3This invention provides a system architecture 300. As shown in the system architecture 300, the data acquisition device 360 ​​is used to collect training data. In this embodiment, the training data varies depending on the model task. For example, when the model task is speech recognition, the training data is speech data; when the model task is text recognition, the training data is text data; and for another example, when the model task is image classification, taking animal image classification as an example, the training data is animal images, such as images of cats, dogs, rabbits, etc. The training data is stored in the database 330. The model optimization device 320 can optimize the neural network model to be optimized (i.e., the first neural network model) to obtain the optimized neural network model 301 (here, 301 is the second neural network model obtained through optimization as described above). For example, the model optimization device 320 optimizes the neural network model to be optimized (i.e., the first neural network model) based on the training data maintained in the database 330 to obtain the optimized neural network model 301 (here, 301 is the second neural network model obtained through optimization as described above). The model optimization device 320 can be a terminal, such as a mobile terminal, tablet computer, laptop computer, AR / VR, vehicle terminal, monitoring equipment, vehicle autonomous driving system, vending machine, etc., or it can be a server or cloud. The following will describe in more detail, using Embodiment 2, how the model optimization device 320 optimizes the neural network model to obtain a neural network model 301, which can be used to implement various model tasks. It should be noted that in practical applications, the training data maintained in the database 330 may not all come from the data acquisition device 360; it may also be received from other devices, such as cameras. Furthermore, it should be noted that the model optimization device 320 may not necessarily optimize the model entirely based on the training data maintained in the database 330; it may also obtain training data from the cloud or other sources for model optimization. The above description should not be construed as limiting the embodiments of this application.

[0087] The neural network model 301 obtained after optimization by the model optimization device 320 can be applied to different systems or devices, such as... Figure 3 The terminal device 310 shown can be a terminal, such as a mobile phone, tablet computer, laptop computer, AR / VR device, vehicle terminal, monitoring equipment, vehicle autonomous driving system, vending machine, etc., or it can be a server or cloud service. (See attached...) Figure 3In this embodiment, terminal device 310 is equipped with I / O interface 312 for data interaction with external devices. Users can input data into I / O interface 312 through client device 340. The input data varies depending on the model task. For example, for a neural network model whose task is image classification, the input data is an image to be processed, including the target object; this image can be input by the user or retrieved from a database. Client device 340 can be an image acquisition device, such as a camera. For a neural network model whose task is speech recognition, the input data is the speech data to be recognized; and for a neural network model whose task is text recognition, the input data is the text data to be recognized.

[0088] The preprocessing module 313 is used to preprocess the input data (such as the image to be processed) received by the I / O interface 312. In this embodiment, the preprocessing module 313 is used to preprocess the input data (such as the image to be processed) received by the I / O interface 312, and the preprocessed data enters the calculation module 311. In this embodiment, for a neural network model whose model task is image classification, the preprocessing module 313 can be used to perform at least one of the following processing on the image to be processed: filtering, enhancement, and denoising, to obtain an image to be processed that meets the requirements.

[0089] During the preprocessing of input data by the terminal device 310, or the calculation module 311 of the terminal device 310 performing calculations and other related processes, the terminal device 310 can call data, code, etc. in the data storage system 350 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 350.

[0090] Finally, the I / O interface 312 returns the model processing results of the input data to the client device 340, which is then provided to the user. At this time, the client device 340 can be a display.

[0091] It is worth noting that the model optimization device 320 can be used for different tasks, and the corresponding neural network model 301 can be used to complete the above tasks, thereby providing the user with the required results.

[0092] In the appendix Figure 3In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 312. Alternatively, the client device 340 can automatically send input data to I / O interface 312. If user authorization is required for the client device 340 to automatically send input data, the user can set the corresponding permissions in the client device 340. The user can view the results output by the terminal device 310 on the client device 340, which can be presented in various ways such as display, sound, or animation. The client device 340 can also act as a data acquisition terminal, collecting data such as... Figure 3 The input data and output results of the input I / O interface 312 are used as new sample data and stored in the database 330. Alternatively, data can be collected directly from the I / O interface 312 without going through the client device 340. Figure 3 The input data of the input I / O interface 312 and the output results of the output I / O interface 312 are stored in the database 330 as new sample data.

[0093] It is worth noting that, attached Figure 3 This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 3 In this configuration, the data storage system 350 is an external storage device relative to the terminal device 310. However, in other cases, the data storage system 350 can also be placed within the terminal device 310. Furthermore, the model optimization device 320 and the terminal device 310 can be the same device.

[0094] like Figure 3 As shown, a neural network model 301 is obtained through optimization processing by the model optimization device 320. Taking image classification as an example, this neural network model 301 in this embodiment may include a feature extractor and a predictor. The feature extractor is used to extract feature vectors from the image to be processed, while the predictor is used to process the feature vectors of the image to obtain the corresponding processing result. In this embodiment, both the feature extractor and the predictor can be convolutional neural networks. In particular, an attention mechanism can be applied in the feature extractor to extract features that are more relevant to the model task.

[0095] As introduced in the basic concepts above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It's a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network where each neuron responds to an input image. Figure 4 As shown, the convolutional neural network (CNN) 400 may include an input layer 410, a convolutional / pooling layer 420 (where the pooling layer is optional), and a neural network layer 430.

[0096] Convolutional / pooling layers 420:

[0097] Convolutional layers:

[0098] like Figure 4 The convolutional / pooling layer 420 shown may include layers as in Examples 421-426. For instance, in one implementation, layer 421 is a convolutional layer, layer 422 is a pooling layer, layer 423 is a convolutional layer, layer 424 is a pooling layer, layer 425 is a convolutional layer, and layer 426 is a pooling layer; in another implementation, layers 421 and 422 are convolutional layers, layer 423 is a pooling layer, layers 424 and 425 are convolutional layers, and layer 426 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0099] The following section will use convolutional layer 421 as an example to introduce the internal working principle of a convolutional layer.

[0100] Convolutional layer 421 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0101] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 400 to make correct predictions.

[0102] When a convolutional neural network 400 has multiple convolutional layers, the initial convolutional layers (e.g., 421) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 400 increases, the features extracted by later convolutional layers (e.g., 426) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0103] Pooling layer:

[0104] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 4Layers 421-426 in example 420 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0105] Neural network layer 430:

[0106] After processing by the convolutional / pooling layers 420, the convolutional neural network 400 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 420 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 400 needs to utilize neural network layers 430 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 430 can include multiple hidden layers (such as...). Figure 4 As shown in 431, 432 to 43n) and output layer 440, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as pose estimation, image recognition, image classification, image super-resolution reconstruction, etc.

[0107] After the multiple hidden layers in neural network layer 430, the final layer of the entire convolutional neural network 400 is the output layer 440. This output layer 440 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 400 has undergone forward propagation (such as...), the loss function is applied. Figure 4 Propagation from 410 to 440 degrees is considered forward propagation, while reverse propagation (such as...) is completed. Figure 4 The propagation from 440 to 410 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 400 and the error between the output of the convolutional neural network 400 through the output layer and the ideal result.

[0108] It should be noted that, as Figure 4The convolutional neural network 400 shown is merely an example of a convolutional neural network; in specific applications, convolutional neural networks can also exist in the form of other network models. Furthermore, Figure 4 The convolutional neural network shown is only an example of a neural network model. Neural network model 301 can also be a network model with other structures, as long as the model applies an attention mechanism, that is, the model includes attention layers, self-attention layers and other attention mechanism network layers.

[0109] The following describes a chip hardware structure provided by an embodiment of this application.

[0110] Figure 5 A chip hardware structure provided in this embodiment of the invention includes a neural network processor 50. This chip can be configured as follows: Figure 3 In the terminal device 310 shown, this chip is used to perform the calculations of the computing module 311. This chip can also be located in, for example... Figure 3 The model optimization device 320 shown is used to complete the model optimization processing and output the neural network model 301. For example... Figure 4 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 5 This is achieved in the chip shown.

[0111] The Neural Processing Unit (NPU) 50 is mounted as a coprocessor on the host CPU, which allocates tasks to it. The core of the NPU is the arithmetic circuit 503, which is controlled by the controller 504 to retrieve matrix data from the memory (weight memory 502 or input memory 501) and perform multiplication operations.

[0112] In some implementations, the arithmetic circuit 503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0113] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit 503. The arithmetic circuit 503 retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0114] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0115] In some implementations, vector computation unit 507 stores the processed output vector in unified memory 506. For example, vector computation unit 507 can apply a nonlinear function to the output of arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0116] The unified memory 506 is used to store input data and output data.

[0117] The weight data is directly transferred from external memory to input memory 501 and / or unified memory 506 via the Direct Memory Access Controller (DMAC), the weight data in external memory is stored in the weight memory 502, and the data in unified memory 506 is stored in external memory. Input data is also transferred to unified memory 506 via the DMAC.

[0118] The Bus Interface Unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0119] The instruction fetch buffer 509, which is connected to the controller 504, is used to store the instructions used by the controller 504.

[0120] The controller 504 is used to call the instructions cached in the instruction fetch memory 509 to control the operation of the computing accelerator.

[0121] Generally, the unified memory 506, input memory 501, weighted memory 502, and instruction fetch memory 509 are all on-chip memories. External memory is a proprietary memory hardware architecture located outside the NPU. This external memory can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), High Bandwidth Memory (HBM), or other readable and writable memories.

[0122] in, Figure 3 and Figure 4 The program algorithm is completed by the main CPU and NPU working together. Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0123] Example 2

[0124] The following describes a neural network model optimization method provided by an embodiment of this application.

[0125] refer to Figure 6 , Figure 6 This is a flowchart illustrating a neural network model optimization method provided in an embodiment of the present invention. The neural network model optimization method 600 includes the following steps:

[0126] 601. The first neural network model is optimized to obtain the second neural network model;

[0127] The second neural network model includes an optimized attention layer and at least two preceding network layers, which are cascaded together. These preceding network layers are located between the input of the second neural network model and the input of the optimized attention layer. (Reference) Figure 7 , Figure 7 This is a schematic diagram of an optimized attention layer structure provided by an embodiment of the present invention; wherein, layers 1 to 5 connected in series in the model are all preceding network layers of the optimized attention layer. Specifically, the specific structure of the preceding network layers can be a single network layer or composed of multiple parallel network layers, to... Figure 7 Taking the 5th layer as an example, it can be composed of multiple parallel network layers.

[0128] This optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The optimized query feature transformation module utilizes W in the optimized attention layer. QThe module that performs feature transformation using the feature transformation matrix is ​​similar to the module that optimizes the key feature transformation in the attention layer by utilizing W. K The module that performs feature transformation using the feature transformation matrix; the optimized value feature transformation module optimizes the attention layer by utilizing W. V The module for feature transformation using the feature transformation matrix. Regarding W... Q Feature transformation matrix, W K Feature transformation matrix, W V For a detailed description of the feature transformation matrix, please refer to the description of the Attention operation in point (7) above. It will not be repeated here.

[0129] The input to the optimized query feature transformation module is obtained from at least one output feature of the optimized attention layer in the previous network layer;

[0130] The input to the optimized key feature transformation module is obtained from at least one output feature of the optimized attention layer in the previous network layer;

[0131] The input to the Value feature transformation module is obtained based on at least one output feature of the attention layer in the preceding network layer;

[0132] Specifically, the input to at least one of the feature transformation modules—the Query feature transformation module, the Key feature transformation module, and the Value feature transformation module—is obtained based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer. Specifically, the input to the aforementioned feature transformation modules can be a feature map. For example, in scenarios such as speech recognition and machine recognition, the feature map can be a three-dimensional feature map, namely, batch dimension B, channel dimension C, and patch dimension P. The non-adjacent preceding network layer refers to all preceding network layers of the optimized attention layer, excluding the preceding network layers adjacent to the optimized attention layer. (Reference) Figure 7 Among them, for Figure 7 In the context of the optimized attention layer, all preceding network layers are layers 1 to 5, and layer 5 is the adjacent preceding network layer of the optimized attention layer. Therefore, layers 1 to 4 in the network are the non-adjacent preceding network layers of the optimized attention layer. Figure 7 The Q module optimizes the query feature transformation module, the K module optimizes the key feature transformation module, and the V module optimizes the value feature transformation module. Figure 7 The Q, K, and V modules in the text are... Figure 2 The Q module, K module, and V module.

[0133] The optimization method in this embodiment of the invention optimizes a first neural network model to obtain a second neural network model. In the second neural network model, the input to the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input to the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; the input to the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer; and the input to at least one of the optimized query, key, and value feature transformation modules is obtained based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer. This optimized second neural network model, by supplementing the inter-layer feature information of the network, enhances the model's expressive power and improves its task processing performance.

[0134] In particular, optimizing the attention layer in Figure 7 The diagram only illustrates the optimization of the query feature transformation module, the optimization of the key feature transformation module, and the optimization of the value feature transformation module. Figure 7 The optimized attention layer shown in the diagram is a simplified illustration. In reality, besides optimizing the query feature transformation module, the key feature transformation module, and the value feature transformation module, the optimized attention layer can also include, for example, […]. Figure 2 Other network layers shown, such as matrix multiplication layers, numerical scaling layers, and normalization layers, may also include masking layers in the optimized attention layer, depending on the processing task of the second neural network model.

[0135] In some possible embodiments, the input to the target feature transformation module is at least one output feature of the attention layer in the preceding network layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0136] Specifically, optionally, the input to the target feature transformation module can be the output feature of a preceding network layer of the optimized attention layer. Simply put, for each target feature transformation module, it selects the output feature of a preceding network layer from the optimized attention layer as its own input feature. Assume the preceding network layer connected to the optimized query feature transformation module is the first preceding network layer, the preceding network layer connected to the optimized key feature transformation module is the second preceding network layer, and the preceding network layer connected to the optimized value feature transformation module is the third preceding network layer. The first, second, and third preceding network layers can be the same or different. For the same optimized attention layer, it is only necessary to satisfy that at least one of the first, second, and third preceding network layers is a non-adjacent preceding network layer of that optimized attention layer. Figure 7 As an example of optimizing the attention layer, the preceding network layer connected to the Query feature transformation module is optimized to be the 2nd preceding network layer, the preceding network layer connected to the Key feature transformation module is optimized to be the 5th preceding network layer, and the preceding network layer connected to the Value feature transformation module is optimized to be the 3rd preceding network layer.

[0137] Assuming the second neural network model has L layers (starting from layer 0), the output of the optimized attention layer in layer i (1≤i≤L-1) is represented as:

[0138] y i =AA i (Y 0～(i-1) ,Y 0～(i-1) ,Y 0～(i-1) ) = Attention i (y a ,y b ,y c )

[0139] Where 0≤a,,≤i-1,Y 0～(i-1) ={y0,y1,y2,...,y i-1 In other words, for an optimized attention layer, a certain output feature of the preceding network layer is selected from all the output features of the preceding network layers as the input of the optimized query feature transformation module of the optimized attention layer. The inputs of the optimized key feature transformation module and the optimized value feature transformation module are also determined using the above method.

[0140] Optionally, the input to the target feature transformation module can also be the output features of at least two preceding network layers of the optimized attention layer. Simply put, for each target feature transformation module, at least two output features from the preceding network layers of the optimized attention layer are selected as its own input features. The preceding network layers selected for the optimized query feature transformation module, optimized key feature transformation module, and optimized value feature transformation module can be the same or different. Further, in some possible embodiments, in the optimized attention layer of this embodiment, before performing feature transformation, the optimized query feature transformation module, optimized key feature transformation module, and optimized value feature transformation module need to concatenate and average the output features of at least two preceding network layers.

[0141] In some possible embodiments, the first neural network model includes a cascaded attention layer and at least two preceding network layers, which are cascaded together. The attention layer refers to a network layer that applies an attention mechanism; it can be the attention layer described in point (7) above, or a self-attention layer, etc. The preceding network layer here refers to the network layer between the input of the first neural network model and the input of the attention layer. (See reference...) Figure 1b , Figure 1b The network structure shown is an example of a first neural network model, where the attention layer is a self-attention layer. The self-attention layer has five cascaded preceding network layers (layers 1 to 5), which are cascaded with the self-attention layer. Specifically, the self-attention layer... Figure 1b The diagram only illustrates the query feature transformation module, the key feature transformation module, and the value feature transformation module. Figure 1b The self-attention layer in the diagram is a simplified illustration. Besides the question feature transformation module, key feature transformation module, and value feature transformation module, the self-attention layer also includes, for example,... Figure 2 Other network layers shown include matrix multiplication layers, numerical scaling layers, and normalization layers. Depending on the processing task of the first neural network model, the attention layer may also include a mask layer, etc.

[0142] Furthermore, step 601 specifically includes the following steps:

[0143] 6011. Determine the search space of the first neural network model. The elements in the search space include the first query feature transformation module, the first key feature transformation module, and the first value feature transformation module in the attention layer that can be connected to the previous network layers.

[0144] The attention layer includes a first query feature transformation module, a first key feature transformation module, and a first value feature transformation module. The search space includes the preceding network layers to which the first query feature transformation module, first key feature transformation module, and first value feature transformation module of each attention layer in the first neural network model can be connected. (Reference) Figure 1b ,for Figure 1b For the self-attention layer in the search space, the preceding network layers that the question feature transformation module can connect to are the first to the fifth preceding network layers, the preceding network layers that the key feature transformation module can connect to are the first to the fifth preceding network layers, and the preceding network layers that the value feature transformation module can connect to are the first to the fifth preceding network layers. Therefore, the five preceding network layers from the first to the fifth preceding network layers will be used as elements of the search space.

[0145] 6012. An optimization attention layer is determined using a search space-based search algorithm, wherein the search algorithm is used to determine, based on search conditions, a first preceding network layer connected to the query feature transformation module, a second preceding network layer connected to the key feature transformation module, and a third preceding network layer connected to the value feature transformation module, wherein at least one of the first preceding network layer, the second preceding network layer, and the third preceding network layer is a non-adjacent preceding network layer of the optimization attention layer.

[0146] Specifically, the search algorithm includes any one of evolutionary algorithms, reinforcement learning algorithms, and network structure search algorithms. Among them, evolutionary algorithms have fast training convergence speed, which is beneficial to improving the optimization speed of the model. Of course, the search algorithm can also be other algorithms, as long as it can achieve the network structure search function. In this embodiment of the invention, after determining the search space corresponding to the first neural network model, a search algorithm based on the search space is used to perform a search, which can determine the optimized attention layer, that is, determine the first preceding network layer connected by the optimized query feature transformation module, the second preceding network layer connected by the optimized key feature transformation module, and the third preceding network layer connected by the optimized value feature transformation module in the optimized attention layer, wherein at least one of the first preceding network layer, the second preceding network layer, and the third preceding network layer is a non-adjacent preceding network layer of the optimized attention layer.

[0147] The search algorithm effectively adjusts the input connections of the attention layer in the first neural network model to obtain an optimized attention layer. This allows for the extraction of feature information between different layers in the network simply by optimizing the attention layer itself. It is evident that the connection structure between the optimized attention layer and the preceding network layers differs from that in the first neural network model. The optimized attention layer focuses on the fusion of inter-layer features; therefore, optimizing the attention layer enhances the expressive power of the second neural network model and helps improve its performance.

[0148] It should be noted that, by using the above steps 6011 and 6012, a second neural network model corresponding to the output feature of the previous network layer can be obtained, with the input of the aforementioned target feature transformation module being an optimized attention layer.

[0149] The following section uses the first neural network model as the Transformer network model and the evolutionary algorithm as an example to explain the optimization process in detail:

[0150] The Transformer is a commonly used network structure in machine translation tasks, containing a large number of Self-Attention operations. This embodiment utilizes an optimization method to reselect the input connection methods for all Attention operations in the standard Transformer network, and combines this with an evolutionary algorithm to search for a superior model structure for machine translation tasks.

[0151] The optimization method in this embodiment is based on an evolutionary algorithm to search the network structure, referring to... Figure 8 , Figure 8 This embodiment provides an example framework for the evolutionary algorithm, which mainly consists of three parts: search space, population, and training module.

[0152] Search space: This includes the available options for the structural components of the first neural network model, which can be defined manually. In this embodiment, it is defined as the input source for the challenge feature transformation module (Q module), key feature transformation module (K module), and value feature transformation module (V module) of each self-attention layer in the Transformer network. The evolutionary algorithm obtains the network structure by randomly sampling the search space of the network structure, with each sampling randomly yielding a different network structure.

[0153] Population: Contains a predefined number of samples, each representing a network structure encoding, sampled from the search space. Operations that can be performed on the population include: adding samples from the search space to the population, deleting some samples from the population, and selecting a sample in the population for mutation (mutation refers to replacing each bit of the encoding with another option in the search space with a preset probability).

[0154] Training module: This module is task-related. It takes the network structure and dataset as input to train the network structure, evaluates the network performance on the validation dataset, and returns the corresponding network performance metrics.

[0155] refer to Figure 8 The specific workflow of this embodiment is as follows:

[0156] 1. Define the search space set L composed of Transformer network connections, the maximum number of samples in the population m, the number of samples n collected from the population in each iteration, the number k of the highest-performing samples retained in the subset in each iteration, and the number of iterations T of the evolutionary algorithm. For example... Figure 8 The search space is defined by taking the optional input sources of the Q, K, and V modules of each self-attention layer as elements of the search space. For example, the input sources of QKV for the first self-attention layer are 1a, 1b, and 1c (1a, etc. are the encodings of the preceding network layers of the first self-attention layer), while the input sources of QKV for the second self-attention layer are 2a, 2b, 2c, and 2d (2a, etc. are the encodings of the preceding network layers of the second self-attention layer), and so on.

[0157] 2. Randomly sample m times from the search space L to form an initial population L containing different connection methods. m ={l1,l2,l3,...,l m Each sampling determines the QKV input source for each self-attention layer, i.e. Figure 8 Each line represents one sampling process, and the first box in a line corresponds to the input source sampling of the QKV of the first self-attention layer (e.g., Figure 8 The first box in the first row of the middle population represents the input sources of the first layer's QKV (1c, 1b, 1a), and the second box in the row corresponds to the input source sampling of the second layer's self-attention layer, and so on.

[0158] 3. Randomly select a subset L from the population. n , where |L n |=n, L n Each sample in the training module is used to evaluate the performance P of each structure in the subset on the translation task, thus obtaining P. n ={p1,p2,p3,...,p n}

[0159] 4. Retain the k highest-performing structures in the subset and discard the others. That is, retain sample L. k =argtopk(P n At the same time, samples corresponding to the eliminated structures are removed from the population.

[0160] 5. For the retained k structures L k Mutation is performed to obtain nk new structures L. mutate Update the population to L′ m =L m -(L n -L k )+L mutate .

[0161] Repeat steps 3-5 above for T iterations, and then select the structure with the best performance in the population as the search result.

[0162] refer to Figure 9a , Figure 9b , Figure 9c , Figure 9a , Figure 9b , Figure 9c This is a schematic diagram of a network structure search process provided by an embodiment of the present invention; a specific search process is illustrated below:

[0163] refer to Figure 1b The search space L is defined as the input connection scheme of all Self-Attention operations in a Transformer network with two Self-Attention layers. The maximum population size is m = 4, the subset contains n = 3 samples, and k = 1 samples with the highest performance are retained in each iteration.

[0164] S1. Perform m = 4 samplings in the search space L to obtain L m ={l 1001 ,l 1002 ,l 1003 ,l 1004}

[0165] S2. Randomly sample a subset L of size n = 3 from the population. n ={l 2001 ,l 2002 ,l 2003}, evaluate the performance P of the three structures in the training module. n ={p 2001 ,p 2002 ,p 2003}, assuming p 2002 >p 2003 >p 2001 Then retain the k=1 structure l with the highest performance in the subset. 2002 At the same time, remove sample l from the population 2001 and l 2003 .

[0166] S3, for the retained sample l 3002Mutation is performed to obtain L mutate ={l 3001 , 3003}, update population L′ m for

[0167] L′ m ={l 3001 ,l 3002 ,l 3003}

[0168] Repeat steps S1-S3, and after T iterations, select the structure with the best performance in the population as the search result.

[0169] In summary, the optimization method in the above embodiments encodes the input connections of attention operations in the network and uses an evolutionary algorithm to search for a better-performing attention structure on a specified task.

[0170] Furthermore, to illustrate the optimization effect of the optimization method in this embodiment of the invention, a search was performed on the machine translation task iwslt14 dataset, using the standard Transformer network as a benchmark, to obtain a new Transformer structure (called Aggregate Transformer) with optimized input connection methods, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of the optimized Transformer network model provided in this embodiment of the invention. In the Aggregate Transformer, the input connection method of the Self-Attention layer differs significantly from the standard Transformer, with multiple Key, Query, or Value inputs connected across layers. This is the result of optimizing the input connection method using an evolutionary algorithm. As shown in Table 1, the accuracy of the search results on the iwslt14 dataset (e.g., BLEU value; BLEU stands for Bilingual Evaluation Understudy) is significantly improved compared to the original network, and even more so compared to the existing Evolved Transformer solution with the same number of samples. The tiny-scale network is the Tiny setting, while the small-scale network is the Small setting.

[0171] Model BLEU (Microscale Network) BLEU (Small-Scale Network) Number of samples Transformer 34.79 37.20 -- Evolved Transformer 34.99(+0.20) 37.32(+0.12) 2000 AggregateTransformer 35.30(+0.51) 37.60(+0.40) 2000

[0172] Table 1

[0173] In some possible embodiments, the elements in the search space may further include at least one of the following: an activation function optional for the first neural network model, a normalization operation optional for the first neural network model, an operation type optional for the feature map of the first neural network model, an optional number of parallel branches optional for the first neural network model, an optional number of modules in the search unit, and optional connection methods between the preceding network layers other than the attention layer.

[0174] Specifically, optional activation functions include the Sigmoid function, Tanh function, ReLU function, etc. Optional normalization operations include Layer Normalization, Batch Normalization, Instance Normalization, Switchable Normalization, etc. The operation type on the feature map refers to the specific type of processing operation performed on the feature map. For example, if the feature map processing operation is convolution, optional convolutional layer types include Conv1*1:2048, Sep Conv 9*1:256 (Separable Convolutional), etc. Other optional operations on the feature map include self-attention operations, gated linear processing operations, etc. The optional number of parallel branches refers to the number of parallel branches the network can allow during network search. For example, assuming the number of parallel branches is 2, a convolutional layer can be replaced by two parallel convolutional layers during the search. The search unit refers to the number of modules (or network layers) in each search unit that the search algorithm replaces during the search process. Specifically, one network layer in the first neural network model can be used as the search unit to search and replace one network layer; or two network layers in the first neural network model can be used as the search and replacement unit to search and replace two network layers.

[0175] In summary, besides incorporating possible connections between the attention layer and previous network layers as elements of the search space, the search space can also include elements from existing search spaces, further improving the performance of the determined second neural network model on the corresponding task. Taking evolutionary algorithms as an example, refer to... Figure 11 , Figure 11 This is a schematic diagram of another evolutionary algorithm framework provided by an embodiment of the present invention; wherein, in addition to the input sources of the Q-module, K-module, and V-module of the self-attention layer of the first neural network model, the elements in the search space also include the operation type of the feature map, the activation function, and the connection method between modules (excluding the self-attention layer in the first neural network model). Accordingly, when the evolutionary algorithm performs network search, the sampled population is as follows: Figure 11 As shown, the processing procedure of the evolutionary algorithm is similar to... Figure 8 The processing procedure is similar and will not be repeated here.

[0176] Taking the first neural network model as a Transformer network model as an example, refer to Table 2, which compares the model performance before and after merging the search space. Here, Aggregate Transformer is the second neural network model obtained by using the search space only as the input source of the Q, K, and V modules of the self-attention layer of the first neural network model. It is evident that merging the search space further improves the performance of the searched network in machine translation tasks.

[0177]

[0178] Table 2

[0179] In some possible embodiments, for the second neural network model whose input to the aforementioned target feature transformation module is the output features of at least two preceding network layers of the optimized attention layer, similarly, steps 6011 and 6012 can be used to first determine the search space corresponding to the first neural network model, and then use a search algorithm to determine the optimized attention layer, that is, to determine at least two preceding network layers connected to the optimized Query feature transformation module, at least two preceding network layers connected to the optimized Key feature transformation module, and at least two preceding network layers connected to the optimized Value feature transformation module in each optimized attention layer. (Similar to the above...) Figure 8 The corresponding optimization process differs in that, taking the input of the target feature transformation module as the output features of the two preceding network layers of the attention layer as an example, in this embodiment of the invention, each time the search algorithm samples in the search space, for each target feature transformation module, it needs to sample the two preceding network layers connected to it. Similarly, to further improve the performance of the second neural network model, the composition of elements in the search space can be expanded, for example, by adding activation functions, normalization methods, etc., as elements to the search space. Referring to the foregoing content, it will not be repeated here.

[0180] In some possible embodiments, the input to the target feature transformation module is the input feature obtained by weighted summation of at least two output features of the previous network layers and the weights of the previous network layers of the optimized attention layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0181] Specifically, the output features of the previous network layers are weighted and summed to serve as the input to the target feature transformation module, thereby achieving inter-layer feature fusion. This enhances the expressive power of the optimized attention layer and improves the performance of the second neural network model.

[0182] In some possible embodiments, the second neural network model further includes a first fusion module, a second fusion module, and a third fusion module. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the first fusion module, and the output of the first fusion module is connected to the input of the optimized query feature transformation module. The first fusion module is used to obtain the first input feature of the optimized query feature transformation module by performing weighted summation processing based on the preceding network layers connected to it. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it.

[0183] The outputs of all preceding network layers in the optimized attention layer are connected to the input of the second fusion module, and the output of the second fusion module is connected to the input of the optimized key feature transformation module. The second fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the second input feature of the optimized key feature transformation module. The parameters of the second fusion module include the second weights corresponding to the preceding network layers connected to it.

[0184] The outputs of all preceding network layers in the optimized attention layer are connected to the input of the third fusion module, and the output of the third fusion module is connected to the input of the optimized value feature transformation module. The third fusion module is used to perform weighted summation based on the preceding network layers connected to it to obtain the third input feature of the optimized value feature transformation module. The parameters of the second fusion module include the third weights corresponding to the preceding network layers connected to it.

[0185] Specifically, for each optimized attention layer, refer to Figure 12 , Figure 12 This is a schematic diagram of another optimized attention layer structure provided in an embodiment of the present invention; to optimize the query feature transformation module (such as...). Figure 12 Taking the Q module as an example, a first fusion module is added before it. This first fusion module is used to perform a weighted summation of all the preceding network layers of the optimized attention layer to obtain the first input features of the optimized query feature transformation module. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. In this way, by using the first fusion module, the features of the preceding network layers of the optimized attention layer can be fused to enhance the expressive power of the optimized attention layer and improve the performance of the second neural network model. Similarly, for the optimized key feature transformation module (such as...) Figure 12The K module in the model (in the model) has a second fusion module preceding it, while the optimized value feature transformation module (such as...) Figure 12 Before the V module, a third fusion module is set up. It should be noted that the difference between the optimized attention layer and the attention layer in the prior art is that the optimized attention layer has three fusion modules before it: a first fusion module, a second fusion module, and a third fusion module.

[0186] Taking the first neural network model as an example, a standard Transformer network with a total of L layers (starting from layer 0) is provided. The output features of each layer are: The input to the i-th optimized attention layer (1≤i≤L-1) is the output features of all its preceding network layers, denoted as:

[0187] Y 0～(i-1) ={y0,y1,y2,...,y i-1}

[0188] The output features of all preceding network layers in the optimized attention layer are input into the fusion module. Taking the optimized query feature transformation module as an example, all output features of preceding network layers are input into the first fusion module. The first fusion module generates the weighted weights of the output features of each preceding network layer. The weighted weight matrix of all output features of preceding network layers is denoted as W. agg Then the output characteristics of the first fusion module are:

[0189] Agg(Y 0～(i-1) ) = W agg Y c W agg ∈R (i-1) × (i-1)

[0190] Y c =Concat(Y 0～(i-1) )∈R (i-1)×z

[0191] Among them, Y c For cascading features, Concat(·) is the cascading operation.

[0192] Finally, the i-1×i-1 dimension weight matrix generated by the first fusion module is used to perform a weighted summation on the i-1 output features of the preceding network layers to obtain the input features that combine the output features of all the preceding network layers. This input feature is used as the input to the query feature transformation module for optimization.

[0193] In fact, reference Figure 12 When determining the first, second, and third weights, the first neural network model is first optimized into... Figure 12 The second neural network model shown ( Figure 12 (For example only), the second neural network model is then trained using the training data. In particular, the optimization method of this embodiment can improve the model performance in a single training session. For example, if the model task of the second neural network model is image classification, a training image with a real label is input into the second neural network model for training. After the second neural network model outputs the predicted label, the first weight, the second weight, and the third weight can be extracted from the first fusion module, the second fusion module, and the third fusion module.

[0194] In some possible embodiments, any one of the first fusion module, the second fusion module, and the third fusion module includes any of the following structures: a static weighting module, a multilayer perceptron module, and an attention module.

[0195] In this embodiment of the invention, taking the first fusion module as an example, the first fusion module can be implemented using any one of the following structures: a static weighting module, a multilayer perceptron module, or an attention module. Wherein:

[0196] ① The static weighting module implements weighting through static weighting parameters. Taking the first fusion module as an example, assuming the first fusion module has i-1 preceding network layers, then the first fusion module includes i-1 static parameters, each corresponding to one of the i-1 preceding network layers, denoted as W. agg =Diag(α1,2,..., i-1 ). Figure 13a , Figure 13b , Figure 13c This is a schematic diagram of the structure of a first fusion module provided in an embodiment of the present invention; see reference. Figure 13a Taking a first fusion module with five preceding network layers as an example, the first fusion module includes five static weighting parameters: a1, a2, a3, a4, and a5, each corresponding to one of the five preceding network layers. After training the second neural network model using training data, the corresponding static weighting parameters can be obtained from the static weighting module, and multiple static weighting parameters can form a weighting matrix.

[0197] ② The multilayer perceptron module mainly achieves weighting through a multilayer perceptron. Taking the first fusion module as an example, the multilayer perceptron uses a two-layer perceptron as an example, refer to... Figure 13b Assuming the first fusion module has i-1 preceding network layers, and the size of the output feature of each preceding network layer is (p, c), the multilayer perceptron module first concatenates the output features of the i-1 preceding network layers to obtain concatenated features (p, c, i-1), and then averages the concatenated features through a pooling layer. Figure 13bThe p dimension is removed from Avg to obtain (c, i-1), and then the averaged features (c, i-1) are input into a two-layer perceptron MLP containing activation functions. After training the second neural network model using training data, the corresponding weighted weight matrix W can be obtained from the multilayer perceptron module. agg =MLP(AvgPool(Y c )).

[0198] ③ The attention module mainly achieves weighting through dynamic generation of attention operations, see reference. Figure 13c Taking the first fusion module as an example, assuming that the first fusion module has i-1 preceding network layers, the attention module first performs concatenation processing on the output features of the i-1 preceding network layers to obtain the concatenated feature Y. c Then, for the cascaded feature Y c Perform an Attention operation (see reference) Figure 2 (Related description), and then averaged to obtain the i-1 dimension weight, denoted as: W agg =AvgPool(Attention(Y) c ,Y c ,Y c )).

[0199] Additionally, DeiT is a deep learning model used for image classification tasks, which extensively utilizes Self-Attention operations. Taking the first, second, and third fusion modules as attention modules as an example, it leverages... Figure 12 The corresponding optimization method was used to process the DeiT model to obtain the second neural network model. The optimized attention layer in the second neural network model is Aggregate-Attention. The performance comparison table of the DeiT model and the second neural network model is shown in Table 3. It can be seen that, compared with the DeiT model, the optimized attention layer in the second neural network model significantly improves the model performance with a small amount of computational and parameter overhead.

[0200]

[0201]

[0202] Table 3

[0203] It is particularly noteworthy that the model optimization method described in this application can supplement inter-layer feature information of the network with minimal computational overhead, further enhancing the expressive power of the optimized network model. Figure 7 The corresponding optimization method embodiment, since it does not add additional convolutional layers, does not increase the computational overhead of the second neural network model. However, for... Figure 12The corresponding optimization method embodiment requires only the calculation of the weighted weights of each preceding network layer, which minimizes the computational cost.

[0204] Example 3

[0205] Based on the neural network model optimization method described in Embodiment 2, this application also provides a model optimization apparatus, referencing... Figure 14 , Figure 14 This is a schematic diagram of a model optimization device provided in an embodiment of the present invention; the model optimization device 1400 includes:

[0206] Processing module 1401 is used to optimize the first neural network model to obtain the second neural network model;

[0207] The second neural network model includes an optimized attention layer and at least two preceding network layers connected in series. The optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The input to the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. The input to the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. The input to the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. At least one of the optimized query feature transformation module, optimized key feature transformation module, and optimized value feature transformation module receives its input based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer.

[0208] The specific execution process of the model optimization device 1400 can be referred to the description in Embodiment 2, and will not be repeated here.

[0209] In addition, this application also provides a model optimization device. Figure 15 This is a schematic diagram of the structure of a model optimization device provided in an embodiment of this application; Figure 15 The model optimization device 1500 shown (which may specifically be a computer device) includes a memory 1501, a processor 1502, a communication interface 1504, and a bus 1503. The memory 1501, processor 1502, and communication interface 1504 are interconnected via the bus 1503.

[0210] The memory 1501 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1501 may store a program. When the program stored in the memory 1501 is executed by the processor 1502, the processor 1502 and the communication interface 1504 are used to execute the various steps of the neural network model optimization method of Embodiment 2 of this application.

[0211] The processor 1502 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the model optimization device described in the above embodiments, or to execute the neural network model optimization method described in Embodiment 2.

[0212] The processor 1502 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the neural network model optimization method in Embodiment 2 of this application can be completed by the integrated logic circuits in the hardware of the processor 1502 or by instructions in software form. The processor 1502 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the neural network model optimization method disclosed in Embodiment 2 of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1501. The processor 1502 reads the information in the memory 1501 and, in conjunction with its hardware, performs the functions required by the units included in the model optimization device described in the above embodiments, or executes the neural network model optimization method of the second embodiment of this application.

[0213] The communication interface 1504 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the model optimization device 1500 and other devices or communication networks. For example, training data can be acquired through the communication interface 1504.

[0214] Bus 1503 may include a pathway for transmitting information between various components of model optimization device 1500 (e.g., memory 1501, processor 1502, communication interface 1504).

[0215] It should be understood that the processing module 1401 in the model optimization device 1400 is equivalent to the processor 1502 in the model optimization equipment 1500. Figure 15 The model optimization device 1500 shown is equivalent to Figure 3 The model optimization device 320 is used in this system.

[0216] It should be noted that, although Figure 15 The model optimization device 1500 shown only illustrates the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the model optimization device 1500 also includes other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the model optimization device 1500 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the model optimization device 1500 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 15 All the devices shown.

[0217] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0219] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0220] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0221] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a computer program product. This computer program product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0222] This invention also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes the neural network model optimization method described in Embodiment 2.

[0223] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the neural network model optimization method described in Embodiment 2.

[0224] Example 4

[0225] Based on the neural network model optimization method described in Embodiment 2, this application also provides a neural network model, which includes an optimized attention layer and at least two preceding network layers located before the optimized attention layer. The at least two preceding network layers are connected in series. The optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The input of the optimized query feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. The input of the optimized key feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. The input of the optimized value feature transformation module is obtained based on the output features of at least one preceding network layer of the optimized attention layer. At least one of the feature transformation modules, namely the optimized query feature transformation module, the optimized key feature transformation module, and the optimized value feature transformation module, receives its input based on the output features of at least one non-adjacent preceding network layer of the optimized attention layer.

[0226] The neural network model provided in this embodiment of the invention, by supplementing the inter-layer feature information of the network, enhances the expressive power of the model and is beneficial to improving the task processing performance of the model.

[0227] In some possible embodiments, the input to the target feature transformation module is at least one output feature of the attention layer in the preceding network layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0228] In some possible embodiments, the input to the target feature transformation module is the input feature obtained by weighted summation of at least one or two output features of the preceding network layers and the weights of the preceding network layers in optimizing the attention layer; the target feature transformation module is any one of the query feature transformation module, the key feature transformation module, and the value feature transformation module.

[0229] In some possible embodiments, the neural network model further includes a first fusion module, a second fusion module, and a third fusion module. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the first fusion module, and the output of the first fusion module is connected to the input of the optimized query feature transformation module. The first fusion module is used to obtain the first input feature of the optimized query feature transformation module by performing a weighted summation based on the preceding network layers connected to it. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the second fusion module, and the output of the second fusion module is connected to the optimized key feature. The input of the transformation module is connected; the second fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the second input feature of the input optimization key feature transformation module. The parameters of the second fusion module include the second weights corresponding to the preceding network layers connected to it; the output of all preceding network layers of the optimization attention layer is connected to the input of the third fusion module, and the output of the third fusion module is connected to the input of the optimization value feature transformation module; the third fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the third input feature of the input optimization value feature transformation module. The parameters of the second fusion module include the third weights corresponding to the preceding network layers connected to it.

[0230] In some possible embodiments, any one of the first fusion module, the second fusion module, and the third fusion module includes any of the following structures: a static weighting module, a multilayer perceptron module, and an attention module.

[0231] The neural network model in this embodiment of the invention is the second neural network model in Embodiment 2. Therefore, for a detailed description of the neural network model in this embodiment of the invention, please refer to the description in Embodiment 2, and it will not be repeated here.

[0232] This application also provides a terminal device on which the neural network model described in this embodiment is running. Figure 16 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. Figure 16 The terminal device 1600 shown (which may specifically be a computer device) includes a memory 1601, a processor 1602, a communication interface 1604, and a bus 1603. The memory 1601, processor 1602, and communication interface 1604 are interconnected via the bus 1603.

[0233] The memory 1601 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1601 may store programs, and when the program stored in the memory 1601 is executed by the processor 1602, the processor 1602 and the communication interface 1604 are used to run the neural network model described in this embodiment of the invention.

[0234] The processor 1602 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to realize the function of the neural network model described in the embodiments of this application.

[0235] Processor 1602 can also be an integrated circuit chip with signal processing capabilities. In implementation, the functions of the neural network model described in this application embodiment can be accomplished through integrated logic circuits in the hardware of processor 1602 or through software instructions. The processor 1602 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. Software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1601. Processor 1602 reads information from memory 1601 and, in conjunction with its hardware, completes the functions of the neural network model in this application embodiment.

[0236] The communication interface 1604 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the terminal device 1600 and other devices or communication networks. For example, an image to be processed can be acquired through the communication interface 1604.

[0237] Bus 1603 may include a pathway for transmitting information between various components of terminal device 1600 (e.g., memory 1601, processor 1602, communication interface 1604).

[0238] It should be noted that, although Figure 16 The terminal device 1600 shown only illustrates the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the terminal device 1600 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the terminal device 1600 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the terminal device 1600 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 16 All the devices shown.

[0239] Understandable. Figure 16 The terminal device 1600 shown is equivalent to Figure 3 Terminal device 310 in the middle.

[0240] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0241] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for optimizing a neural network model, characterized in that, Includes the following steps: A second neural network model is obtained by optimizing the first neural network model; the task of the first neural network model includes at least one of the following: object detection, object classification, image classification, machine translation, speech recognition, or text recognition; wherein, The second neural network model includes an optimized attention layer and at least two preceding network layers, which are cascaded together. The optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The input to the optimized query feature transformation module is obtained based on at least one output feature of the optimized attention layer in the previous network layer; The input to the optimized key feature transformation module is obtained based on at least one output feature of the optimized attention layer in the previous network layer; The input to the optimized value feature transformation module is obtained based on at least one output feature of the optimized attention layer in the previous network layer; The input of at least one of the following feature transformation modules—the Query feature transformation module, the Key feature transformation module, and the Value feature transformation module—is obtained based on the output features of at least one non-adjacent preceding network layer of the Optimized Attention Layer.

2. The method according to claim 1, characterized in that, The first neural network model includes a cascaded attention layer and at least two preceding network layers located before the attention layer, wherein the at least two preceding network layers are cascaded. The optimization of the first neural network model to obtain the second neural network model specifically includes the following steps: Determine the search space of the first neural network model, wherein the elements in the search space include the preceding network layers to which the first query feature transformation module, the first key feature transformation module, and the first value feature transformation module in the attention layer can be connected; The optimized attention layer is determined using a search algorithm based on the search space, wherein the search algorithm is used to determine, according to search conditions, a first preceding network layer connected by the optimized query feature transformation module, a second preceding network layer connected by the optimized key feature transformation module, and a third preceding network layer connected by the optimized value feature transformation module, wherein at least one of the first preceding network layer, the second preceding network layer, and the third preceding network layer is a non-adjacent preceding network layer of the optimized attention layer.

3. The method according to claim 2, characterized in that, The search algorithm includes any one of evolutionary algorithms, reinforcement learning algorithms, and network structure search algorithms.

4. The method according to claim 2 or 3, characterized in that, The elements in the search space also include at least one of the following: an activation function optional for the first neural network model, a normalization operation optional for the first neural network model, an operation type optional for the feature map of the first neural network model, an optional number of parallel branches optional for the first neural network model, an optional number of modules in the search unit, and optional connection methods between the preceding network layers other than the attention layer.

5. The method according to claim 1, characterized in that, The input to the target feature transformation module is the input feature obtained by weighted summation of the output features of at least two preceding network layers of the optimized attention layer and the weights of the preceding network layers; the target feature transformation module is any one of the optimized query feature transformation module, the optimized key feature transformation module, and the optimized value feature transformation module.

6. The method according to claim 5, characterized in that, The second neural network model further includes a first fusion module, a second fusion module, and a third fusion module. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the first fusion module, and the output of the first fusion module is connected to the input of the optimized query feature transformation module. The first fusion module is used to obtain the first input feature of the optimized query feature transformation module by performing weighted summation processing based on the preceding network layers connected to it. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the second fusion module, and the output of the second fusion module is connected to the input of the optimized key feature transformation module. The second fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the second input feature of the optimized key feature transformation module. The parameters of the second fusion module include the second weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the third fusion module, and the output of the third fusion module is connected to the input of the optimized value feature transformation module. The third fusion module is used to obtain the third input feature of the optimized value feature transformation module by performing weighted summation processing based on the preceding network layers connected to it. The parameters of the second fusion module include the third weights corresponding to the preceding network layers connected to it.

7. The method according to claim 6, characterized in that, The first fusion module, the second fusion module, and the third fusion module each include any of the following structures: a static weighting module, a multilayer perceptron module, and an attention module.

8. A model optimization device, characterized in that, include: The processing module is used to optimize the first neural network model to obtain a second neural network model; the task of the first neural network model includes at least one of the following: object detection, object classification, image classification, machine translation, speech recognition, or text recognition; wherein, The second neural network model includes an optimized attention layer and at least two preceding network layers, which are connected in series. The optimized attention layer includes an optimized query feature transformation module, an optimized key feature transformation module, and an optimized value feature transformation module. The input to the optimized query feature transformation module is obtained based on at least one output feature of the optimized attention layer in the previous network layer; The input to the optimized key feature transformation module is obtained based on at least one output feature of the optimized attention layer in the previous network layer; The input to the optimized value feature transformation module is obtained based on at least one output feature of the optimized attention layer in the previous network layer; The input of at least one of the following feature transformation modules—the Query feature transformation module, the Key feature transformation module, and the Value feature transformation module—is obtained based on the output features of at least one non-adjacent preceding network layer of the Optimized Attention Layer.

9. The apparatus according to claim 8, characterized in that, The first neural network model includes a cascaded attention layer and at least two preceding network layers located before the attention layer, wherein the at least two preceding network layers are cascaded together; the processing module, in optimizing the first neural network model to obtain the second neural network model, is specifically used for: Determine the search space of the first neural network model, wherein the elements in the search space include the preceding network layers to which the first query feature transformation module, the first key feature transformation module, and the first value feature transformation module in the attention layer can be connected; The optimized attention layer is determined using a search algorithm based on the search space, wherein the search algorithm is used to determine, according to search conditions, a first preceding network layer connected by the optimized query feature transformation module, a second preceding network layer connected by the optimized key feature transformation module, and a third preceding network layer connected by the optimized value feature transformation module, wherein at least one of the first preceding network layer, the second preceding network layer, and the third preceding network layer is a non-adjacent preceding network layer of the optimized attention layer.

10. The apparatus according to claim 9, characterized in that, The search algorithm includes any one of evolutionary algorithms, reinforcement learning algorithms, and network structure search algorithms.

11. The apparatus according to claim 9 or 10, characterized in that, The elements in the search space also include at least one of the following: an activation function optional for the first neural network model, a normalization operation optional for the first neural network model, an operation type optional for the feature map of the first neural network model, an optional number of parallel branches optional for the first neural network model, an optional number of modules in the search unit, and optional connection methods between the preceding network layers other than the attention layer.

12. The apparatus according to claim 8, characterized in that, The input to the target feature transformation module is the input feature obtained by weighted summation of the output features of at least two preceding network layers of the optimized attention layer and the weights of the preceding network layers; the target feature transformation module is any one of the optimized query feature transformation module, the optimized key feature transformation module, and the optimized value feature transformation module.

13. The apparatus according to claim 12, characterized in that, The second neural network model further includes a first fusion module, a second fusion module, and a third fusion module. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the first fusion module, and the output of the first fusion module is connected to the input of the optimized query feature transformation module. The first fusion module is used to obtain the first input feature of the optimized query feature transformation module by performing weighted summation processing based on the preceding network layers connected to it. The parameters of the first fusion module include the first weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the second fusion module, and the output of the second fusion module is connected to the input of the optimized key feature transformation module. The second fusion module is used to perform weighted summation processing based on the preceding network layers connected to it to obtain the second input feature of the optimized key feature transformation module. The parameters of the second fusion module include the second weights corresponding to the preceding network layers connected to it. The outputs of all preceding network layers of the optimized attention layer are connected to the input of the third fusion module, and the output of the third fusion module is connected to the input of the optimized value feature transformation module. The third fusion module is used to obtain the third input feature of the optimized value feature transformation module by performing weighted summation processing based on the preceding network layers connected to it. The parameters of the second fusion module include the third weights corresponding to the preceding network layers connected to it.

14. The apparatus according to claim 13, characterized in that, The first fusion module, the second fusion module, and the third fusion module each include any of the following structures: a static weighting module, a multilayer perceptron module, and an attention module.

15. A model optimization device, characterized in that, It includes a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the neural network model optimization method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the neural network model optimization method as described in any one of claims 1 to 7.

17. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the neural network model optimization method according to any one of claims 1 to 7.

18. A terminal device, characterized in that, The terminal device runs the second neural network model as described in any one of claims 1 to 7.

Images