A model training method and device

Abstract

The application discloses a model training method which can be applied to the field of artificial intelligence. The method comprises the following steps: obtaining a first neural network model; replacing the first convolution layer in the first neural network model with a linear operation to obtain a plurality of second neural network models; performing model training on the plurality of second neural network models after training to obtain a neural network model with the highest model precision in the plurality of second neural network models. The application replaces the convolution layer in the neural network to be trained with a linear operation which can be equivalent to the convolution layer, and selects the mode with the highest precision from a plurality of replacement modes, so that the precision of the model after training is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model training method and apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] To improve model accuracy during training, overparameterization training methods can be employed. This involves introducing additional parameters and computations into the original model during training, thereby influencing the training process and ultimately improving model accuracy. ACNet (Asymmetric Convolutional Network) is one such overparameterization training method. During training, the original 3x3 convolutions are replaced with the sum of three convolutions: 3x3, 1x3, and 3x1. However, ACNet only has one fixed overparameterization form, offering limited performance improvement. Summary of the Invention

[0004] Firstly, this application provides a model training method, the method comprising:

[0005] A first neural network model is obtained, which includes a first convolutional layer. The training device can replace some or all of the convolutional layers in the first neural network model with linear operations. Specifically, the replaced convolutional layer can be one of the multiple convolutional layers in the first neural network model, where the first convolutional layer is one of them.

[0006] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation, and the linear operation is equivalent to a convolutional layer;

[0007] In this application embodiment, "equivalence" refers to the relationship between two computational units. Specifically, it means that two computational units that are formally different can produce the same processing result when processing any identical data. One of the computational units can be transformed into the form of the other computational unit through mathematical derivation. For this application embodiment, the sub-linear operations included in a linear operation can be transformed into the form of a convolutional layer through mathematical derivation, and the resulting convolutional layer and the linear operation produce the same processing result when processing the same data.

[0008] A linear operation is composed of multiple sub-linear operations. Here, a sub-linear operation refers to a basic linear operation, not a composite of multiple basic linear operations. For example, the operation type of a sub-linear operation can be, but is not limited to, addition, null operation, identity, convolution, batch normalization (BN), or pooling. Correspondingly, a linear operation can refer to a composite of at least one of these sub-linear operations. It should be understood that "composition" here means that the number of sub-linear operations is greater than or equal to two, and that there are connections between them; there are no isolated sub-linear operations. A connection means that the output of one sub-linear operation is used as the input of another (except for sub-linear operations located on the output side of a linear operation, where the output of that sub-linear operation is used as the output of the linear operation).

[0009] It should be understood that the linear operations in each second neural network model are different from those in the first convolutional layer, and different second neural network models include different linear operations;

[0010] The plurality of second neural network models are trained to obtain a target neural network model, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of trained second neural network models.

[0011] When training the second neural network model, the model accuracy (or validation accuracy) of each trained second neural network model can be obtained. Based on the model accuracy of each second neural network model, the second neural network model with the highest model accuracy can be selected from multiple second neural network models.

[0012] By using the above method, the convolutional layers in the neural network to be trained are replaced with linear operations that can be equivalent to convolutional layers, and the method with the highest accuracy is selected from multiple replacement methods, thereby improving the accuracy of the trained model.

[0013] In one possible implementation, the receptive field of the convolutional layer equivalent to the linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0014] To ensure that a linear operation can be equivalent to a convolutional layer, at least one of the multiple sub-linear operations within a linear operation must be a convolutional operation. During subsequent model inference, to avoid slowing down the inference stage or increasing its resource consumption, linear operations are not used for model inference. Instead, a convolutional layer equivalent to the linear operation (which can be referred to as the second convolutional layer in subsequent embodiments) is used for model inference. Furthermore, the receptive field of this equivalent convolutional layer must be less than or equal to the receptive field of the first convolutional layer.

[0015] In one possible implementation, the linear operation includes multiple operation branches, each of which takes as input to the linear operation. That is, each operation branch processes the input data of the linear operation. Each operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer; or,

[0016] The linear operation includes an operation branch for processing the input data of the linear operation. The operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer.

[0017] In this linear operation, the input and output are considered as two endpoints. A data path between these endpoints can be considered an operation branch. The starting point of the operation branch is the input of the linear operation, and the ending point is the output of the linear operation. In one implementation, the linear operation can include multiple operation branches. Each operation branch processes the input data of the linear operation, which is equivalent to each operation branch starting as the input of the linear operation. Furthermore, the input of the sub-linear operation closest to the input of the linear operation in each operation branch is the input data of the linear operation, meaning each operation branch processes the input data of the linear operation. Each operation branch includes at least one serial sub-linear operation. Alternatively, the linear operation can be represented as a computation graph that defines the input sources and output data flow of each sub-linear operation. Any path from input to output in this computation graph can be defined as an operation branch of the linear operation.

[0018] For a single sublinear operation, such as k*k convolution or pooling, the receptive field is k, while the receptive field of the summation operation and BN operation is 1. The equivalent receptive field of an operation branch is defined as k: each output of the operation branch is affected by kxk inputs.

[0019] To ensure that the equivalent receptive field of a linear operation is less than or equal to the receptive field of the first convolutional layer, the equivalent receptive field of each operation branch in the linear operation needs to be less than or equal to the receptive field of the first convolutional layer. In one implementation, the linear operation may include only one operation branch, which is used to process the input data of the linear operation. The operation branch includes at least one serial sub-linear operation. In this case, the equivalent receptive field of the operation branch included in the linear operation is less than or equal to the receptive field of the first convolutional layer.

[0020] In one possible implementation, the equivalent receptive field of at least one of the plurality of parallel operation branches is equal to the receptive field of the first convolutional layer; or,

[0021] The equivalent receptive field of the linear operation, which includes only one branch of the operation, is equal to the receptive field of the first convolutional layer.

[0022] In one implementation, the equivalent receptive field of at least one of the multiple parallel operation branches is equal to the receptive field of the first convolutional layer. Then, the receptive field of the linear operation is equal to the receptive field of the first convolutional layer. Consequently, the receptive field of the convolutional layer equivalent to the linear operation (hereinafter referred to as the second convolutional layer) is equal to the receptive field of the first convolutional layer. The second convolutional layer can be used for subsequent model inference. Since the receptive field of the second convolutional layer is consistent with that of the first convolutional layer, under the premise of ensuring that the size specifications of the neural network model that has not been replaced are consistent, that is, under the premise of ensuring that the speed and resource consumption of the inference stage remain unchanged, the receptive field of the second convolutional layer is smaller than that of the first convolutional layer, which increases the number of training parameters and improves the accuracy of the model.

[0023] In one possible implementation, the linear operation in each second neural network model is different from that in the first convolutional layer, and different second neural network models include different linear operations.

[0024] In one possible implementation, the convolutional layer equivalent to the linear operation and the linear operation produce the same processing results when processing the same data.

[0025] In one possible implementation, the target neural network model includes a trained target linear operation, and the method further includes:

[0026] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0027] Since the target linear operation includes multiple sub-linear operations compared to the first convolutional layer, directly using this target neural network model for model inference would reduce the inference speed and increase the resource consumption required for model inference. Therefore, in this embodiment, an equivalent second convolutional layer to the trained target linear operation can be obtained, and the trained target linear operation in the target neural network model can be replaced with the second convolutional layer to obtain a third neural network model, which can be used for model inference.

[0028] The so-called model reasoning refers to the actual data processing process using the model during its application.

[0029] It should be understood that, in this embodiment of the application, the step of obtaining the second convolutional layer equivalent to the trained target linear operation and replacing the trained target linear operation in the target neural network model with the second convolutional layer to obtain the third neural network model can be performed by a training device. After training is completed, the training device can directly feed back the third neural network model. Specifically, the training device can send the third neural network model to a terminal device or server so that the terminal device or server can perform model inference based on the third neural network model. Alternatively, the terminal device or server can obtain the second convolutional layer equivalent to the trained target linear operation and replace the trained target linear operation in the target neural network model with the second convolutional layer before performing model inference to obtain the action execution of the third neural network model.

[0030] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0031] In order to ensure that the model used for inference has the same specifications as the first neural network model before training, the size of the second convolutional layer needs to be the same as the size of the first convolutional layer.

[0032] In one implementation, if the receptive field of the target linear operation is equal to the receptive field of the first convolutional layer, then the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0033] In one implementation, if the receptive field of the target linear operation is smaller than the receptive field of the first convolutional layer, the size of the calculated equivalent convolutional layer will be smaller than the size of the first convolutional layer. In this case, zero-padding can be performed on the calculated equivalent convolutional layer to obtain a second convolutional layer with the same size as the first convolutional layer.

[0034] In one possible implementation, the method further includes:

[0035] Based on the order in which the multiple sub-linear operations included in the trained target linear operation process data, each sub-linear operation is merged into an adjacent sub-linear operation that is later in the order, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0036] If a sub-linear operation is not directly connected to the input side of a linear operation, then its fusion parameter is its own operation parameter;

[0037] If the sub-linear operation is not directly connected to the input side of the linear operation, its fusion parameter is obtained based on the fusion parameter of the adjacent preceding sub-linear operation, or based on the fusion parameter of the adjacent preceding operation and its own operation parameter;

[0038] For each sub-linear operation, it can be merged into an adjacent sub-linear operation that is later in the order in which multiple sub-linear operations process data, until the merging is completed to the last sub-linear operation (the sub-linear operation closest to the output).

[0039] It should be understood that the determination of the input of a sub-linear operation depends on the completion of data processing and the corresponding output of other sub-linear operations. For example, if the output of operation A is the input of operation B, and the output of operation B is the input of operation C, then operation C can only perform data processing after operation A and operation B have completed data processing and obtained the corresponding output. Therefore, the sub-linear operation needs to complete the parameter fusion of the sub-linear operations before it can perform its own parameter fusion.

[0040] It should be understood that the determination of the inputs of some sub-linear operations does not depend on certain sub-linear operations completing data processing and obtaining corresponding outputs. For example, the input of operation A1 is the input of the overall linear operation, the output of operation A1 is the input of operation A2, the output of operation A2 is the input of operation B, the input of operation C1 is the input of the overall linear operation, the output of operation C1 is the input of operation C2, and the output of operation C2 is also the input of operation B. In this case, there is no strict time constraint between the data processing of operation A1 and the data processing of operation C1. Therefore, the process of merging operation A1 into operation A2 can be simultaneous with, before or after the process of merging operation C1 into operation C2.

[0041] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0042] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0043] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0044] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0045] In this embodiment, the first sub-linear operation and the second sub-linear operation can be any adjacent sub-linear operations in the trained target linear operations, and the second sub-linear operation is the sub-linear operation following the first sub-linear operation in the sequence. The first sub-linear operation includes a first operation parameter, and the first sub-linear operation is used to process the input data of the first sub-linear operation according to the first operation parameter, corresponding to the operation type of the first sub-linear operation. The second sub-linear operation includes a second operation parameter, and the second sub-linear operation is used to process the input data of the second sub-linear operation according to the second operation parameter, corresponding to the operation type of the second sub-linear operation. The step of merging each sub-linear operation into an adjacent sub-linear operation that is later in the sequence includes:

[0046] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters.

[0047] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0048] For linear operations in the trained target neural network, the fusion parameter is equal to the fusion (output node). The fusion process is performed on every linear operation in the model, ultimately resulting in a fully fused model. This model has the same structure as the original model, thus maintaining the same speed and resource consumption during the inference phase. Furthermore, the models before and after fusion are mathematically equivalent, ensuring that the accuracy of the fused model remains consistent with that of the original model.

[0049] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0050] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0051] Secondly, this application provides a model training method, the method comprising:

[0052] Obtain a first neural network model, the first neural network model including a first convolutional layer, the first neural network model being used to achieve the target task;

[0053] A target linear operation for replacing the first convolutional layer is determined based on at least one of the following information, wherein the information includes the network structure of the first neural network model, the target task, and the position of the first convolutional layer in the first neural network model, and the target linear operation is equivalent to a convolutional layer;

[0054] Among them, different linear operations can be selected for neural network models with different network structures, neural network models that achieve different target tasks, and convolutional layers at different positions in the neural network model, so that the replaced neural network model has higher accuracy after training.

[0055] The target linear operation can be determined based on the network structure of the first neural network model and / or the position of the first convolutional layer within the first neural network model. Specifically, the structure of the target linear operation can be determined based on the network structure of the first neural network model. The network structure of the first neural network model can include the number of sub-network layers, the types of sub-network layers, the connection relationships between sub-network layers, and the position of the first convolutional layer within the first neural network model. The structure of the target linear operation can refer to the number of sub-linear operations included in the target linear operation, the types of sub-linear operations, and the connection relationships between sub-linear operations. For example, it can be based on model search, performing linear operation replacement on convolutional layers of neural network models with different network structures, and... The replaced neural network model is trained to determine the optimal or relatively optimal linear operation corresponding to each convolutional layer in the network structure of each neural network model. The optimal or relatively optimal linear operation refers to the model with higher accuracy obtained by training the replaced neural network model. After obtaining the first neural network model, based on the network structure of the first neural network model, a neural network model with the same or similar structure can be selected from the network structures of the neural network models obtained in the previous search. The linear operation corresponding to a convolutional layer in the same or similar neural network model is determined as the target linear operation. The relative position of the "convolutional layer" in the same or similar neural network model is the same as or similar to the relative position of the first convolutional layer in the first neural network model.

[0056] The target linear operation can be determined based on the network structure of the first neural network model and the target task it achieves. Similar to the above determination based on the network structure of the first neural network model, linear operation replacement can be performed on the convolutional layers of neural network models with different network structures and different target tasks through model search. The replaced neural network models are then trained to determine the optimal or relatively optimal linear operation corresponding to each convolutional layer in the network structure of each neural network model. The optimal or relatively optimal linear operation refers to the model with higher accuracy obtained by training the replaced neural network model.

[0057] The target linear operation can be determined based on the target task implemented by the first neural network model. Similar to the above determination based on the network structure of the first neural network model, linear operations can be replaced by model search for the convolutional layers of neural network models that implement different target tasks. The replaced neural network models are then trained to determine the optimal or relatively optimal linear operation corresponding to each convolutional layer in the network structure of each neural network model. The optimal or relatively optimal linear operation refers to the model with higher accuracy obtained by training the replaced neural network model.

[0058] It should be understood that the above-described network structure based on the first neural network model and / or the method for determining the target linear operation of the target task is only an illustration and can be implemented in other ways, as long as the accuracy of the replaced first neural network model (i.e., the second neural network model) is high. It does not limit the specific structure and method of determining the target linear operation.

[0059] Based on the first neural network model, a second neural network model is obtained, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation;

[0060] The second neural network model is trained to obtain the target neural network model.

[0061] In this embodiment, the convolutional layers in the neural network to be trained are replaced with target linear operations. The structure of the target linear operations is determined according to the structure of the first neural network model and / or the target task. Compared with the linear operations used in the prior art when replacing convolutional layers, the structure of the linear operations in this embodiment can be more adapted to the first neural network model and is more flexible. Different linear operations can be designed for different model structures and task types, thereby improving the accuracy of the trained model.

[0062] In one possible implementation, the target linear operation comprises multiple sub-linear operations, and the target linear operation comprises M operation branches, each operation branch having its input as the input of the target linear operation, wherein the M operation branches satisfy at least one of the following conditions:

[0063] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0064] At least two of the M operation branches have a different number of sub-linear operations; or,

[0065] At least two of the M operation branches contain sub-linear operations of different types.

[0066] Compared to the existing technology that uses linear operations to replace convolutional layers, the target linear operation structure provided in this embodiment is more complex and can improve the accuracy of the trained model.

[0067] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0068] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0069] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0070] In one possible implementation, the target neural network model includes a trained target linear operation, and the method further includes:

[0071] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0072] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0073] In one possible implementation, the method further includes:

[0074] Based on the order in which the multiple sub-linear operations included in the trained target linear operation process data, each sub-linear operation is merged into an adjacent sub-linear operation that is later in the order, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0075] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0076] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0077] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0078] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0079] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0080] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0081] Furthermore, this application provides a model training method, characterized in that the method includes:

[0082] Obtain a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0083] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation. The target linear operation is equivalent to a convolutional layer, and the target linear operation includes multiple sub-linear operations. The target linear operation includes M operation branches, and the input of each operation branch is the input of the target linear operation. The M operation branches satisfy at least one of the following conditions:

[0084] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0085] At least two of the M operation branches have a different number of sub-linear operations; or,

[0086] At least two of the M operation branches contain sub-linear operations of different types.

[0087] The second neural network model is trained to obtain the target neural network model.

[0088] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0089] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0090] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0091] In one possible implementation, the target neural network model includes a trained target linear operation, and the method further includes:

[0092] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0093] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0094] In one possible implementation, the method further includes:

[0095] Based on the order in which the multiple sub-linear operations included in the trained target linear operation process data, each sub-linear operation is merged into an adjacent sub-linear operation that is later in the order, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0096] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0097] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0098] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0099] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0100] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0101] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0102] This application provides a model training method, the method comprising: obtaining a first neural network model, the first neural network model including a first convolutional layer; obtaining a plurality of second neural network models based on the first neural network model, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation, the target linear operation being equivalent to a convolutional layer, the target linear operation including a plurality of sub-linear operations, the target linear operation including M operation branches, the input of each operation branch being the input of the target linear operation, the M operation branches satisfying at least one of the following conditions: the input of at least one sub-linear operation among the plurality of sub-linear operations included in the M operation branches is the output of the plurality of sub-linear operations; the number of sub-linear operations included between at least two operation branches among the M operation branches is different; or, the operation types of the sub-linear operations included between at least two operation branches among the M operation branches are different; and training the second neural network models to obtain a target neural network model. Compared with the structure of linear operations used to replace convolutional layers in the prior art, the structure of the target linear operation provided in this embodiment is more complex, which can improve the accuracy of the trained model.

[0103] Thirdly, this application provides a model training apparatus, the apparatus comprising:

[0104] An acquisition module is used to acquire a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0105] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation, and the linear operation is equivalent to a convolutional layer;

[0106] The model training module is used to train the plurality of second neural network models to obtain a target neural network model, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of trained second neural network models.

[0107] By using the above method, the convolutional layers in the neural network to be trained are replaced with linear operations that can be equivalent to convolutional layers, and the method with the highest accuracy is selected from multiple replacement methods, thereby improving the accuracy of the trained model.

[0108] In one possible implementation, the receptive field of the convolutional layer equivalent to the linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0109] To ensure that a linear operation can be equivalent to a convolutional layer, at least one of the multiple sub-linear operations within a linear operation must be a convolutional operation. During subsequent model inference, to avoid slowing down the inference stage or increasing its resource consumption, linear operations are not used for model inference. Instead, a convolutional layer equivalent to the linear operation (which can be referred to as the second convolutional layer in subsequent embodiments) is used for model inference. Furthermore, the receptive field of this equivalent convolutional layer must be less than or equal to the receptive field of the first convolutional layer.

[0110] In one possible implementation, the linear operation includes multiple operation branches, each operation branch receiving the input of the linear operation. Each operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer; or,

[0111] The linear operation includes an operation branch for processing the input data of the linear operation. The operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer.

[0112] In one implementation, the equivalent receptive field of at least one of the multiple parallel operation branches is equal to the receptive field of the first convolutional layer. Then, the receptive field of the linear operation is equal to the receptive field of the first convolutional layer. Consequently, the receptive field of the convolutional layer equivalent to the linear operation (hereinafter referred to as the second convolutional layer) is equal to the receptive field of the first convolutional layer. The second convolutional layer can be used for subsequent model inference. Since the receptive field of the second convolutional layer is consistent with that of the first convolutional layer, under the premise of ensuring that the size specifications of the neural network model that has not been replaced are consistent, that is, under the premise of ensuring that the speed and resource consumption of the inference stage remain unchanged, the receptive field of the second convolutional layer is smaller than that of the first convolutional layer, which increases the number of training parameters and improves the accuracy of the model.

[0113] In one possible implementation, the linear operation in each second neural network model is different from that in the first convolutional layer, and different second neural network models include different linear operations.

[0114] In one possible implementation, the convolutional layer equivalent to the linear operation and the linear operation produce the same processing results when processing the same data.

[0115] In one possible implementation, the target neural network model includes a trained target linear operation, and the acquisition module is configured to:

[0116] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0117] Since the target linear operation includes multiple sub-linear operations compared to the first convolutional layer, directly using this target neural network model for model inference would reduce the inference speed and increase the resource consumption required for model inference. Therefore, in this embodiment, an equivalent second convolutional layer to the trained target linear operation can be obtained, and the trained target linear operation in the target neural network model can be replaced with the second convolutional layer to obtain a third neural network model, which can be used for model inference.

[0118] The so-called model reasoning refers to the actual data processing process using the model during its application.

[0119] It should be understood that, in this embodiment of the application, the step of obtaining the second convolutional layer equivalent to the trained target linear operation and replacing the trained target linear operation in the target neural network model with the second convolutional layer to obtain the third neural network model can be performed by a training device. After training is completed, the training device can directly feed back the third neural network model. Specifically, the training device can send the third neural network model to a terminal device or server so that the terminal device or server can perform model inference based on the third neural network model. Alternatively, the terminal device or server can obtain the second convolutional layer equivalent to the trained target linear operation and replace the trained target linear operation in the target neural network model with the second convolutional layer before performing model inference to obtain the action execution of the third neural network model.

[0120] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0121] In order to ensure that the model used for inference has the same specifications as the first neural network model before training, the size of the second convolutional layer needs to be the same as the size of the first convolutional layer.

[0122] In one implementation, if the receptive field of the target linear operation is equal to the receptive field of the first convolutional layer, then the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0123] In one implementation, if the receptive field of the target linear operation is smaller than the receptive field of the first convolutional layer, the size of the calculated equivalent convolutional layer will be smaller than the size of the first convolutional layer. In this case, zero-padding can be performed on the calculated equivalent convolutional layer to obtain a second convolutional layer with the same size as the first convolutional layer.

[0124] In one possible implementation, the device further includes:

[0125] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0126] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0127] The fusion module is used for:

[0128] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0129] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0130] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0131] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0132] Fourthly, this application provides a model training apparatus, the apparatus comprising:

[0133] An acquisition module is used to acquire a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0134] A target linear operation for replacing the first convolutional layer is determined based on at least one of the following information, wherein the information includes the network structure of the first neural network model, the target task, and the position of the first convolutional layer in the first neural network model, and the target linear operation is equivalent to a convolutional layer;

[0135] Based on the first neural network model, a second neural network model is obtained, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation;

[0136] The model training module is used to train the second neural network model to obtain the target neural network model.

[0137] In this embodiment, the convolutional layers in the neural network to be trained are replaced with target linear operations. The structure of the target linear operations is determined according to the structure of the first neural network model and / or the target task. Compared with the linear operations used in the prior art when replacing convolutional layers, the structure of the linear operations in this embodiment can be more adapted to the first neural network model and is more flexible. Different linear operations can be designed for different model structures and task types, thereby improving the accuracy of the trained model.

[0138] In one possible implementation, the target linear operation comprises multiple sub-linear operations, and the target linear operation comprises M operation branches, each operation branch having its input as the input of the target linear operation, wherein the M operation branches satisfy at least one of the following conditions:

[0139] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0140] At least two of the M operation branches have a different number of sub-linear operations; or,

[0141] At least two of the M operation branches contain sub-linear operations of different types.

[0142] Compared to the existing technology that uses linear operations to replace convolutional layers, the target linear operation structure provided in this embodiment is more complex and can improve the accuracy of the trained model.

[0143] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0144] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0145] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0146] In one possible implementation, the acquisition module is used to replace the trained target linear operation in the target neural network model with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0147] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0148] In one possible implementation, the device further includes:

[0149] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0150] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0151] The fusion module is used to obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of a third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0152] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0153] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0154] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0155] This application embodiment also provides a model training apparatus, the apparatus comprising:

[0156] An acquisition module is used to acquire a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0157] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation. The target linear operation is equivalent to a convolutional layer, and the target linear operation includes multiple sub-linear operations. The target linear operation includes M operation branches, and the input of each operation branch is the input of the target linear operation. The M operation branches satisfy at least one of the following conditions:

[0158] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0159] At least two of the M operation branches have a different number of sub-linear operations; or,

[0160] At least two of the M operation branches contain sub-linear operations of different types.

[0161] The model training module is used to train the second neural network model to obtain the target neural network model.

[0162] Compared to the existing technology that uses linear operations to replace convolutional layers, the target linear operation structure provided in this embodiment is more complex and can improve the accuracy of the trained model.

[0163] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0164] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0165] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0166] In one possible implementation, the target neural network model includes a trained target linear operation, and the acquisition module is configured to:

[0167] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0168] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0169] In one possible implementation, the device further includes:

[0170] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0171] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0172] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0173] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0174] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0175] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0176] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0177] Fifthly, embodiments of this application provide a model training apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect, the third aspect, and any optional method thereof.

[0178] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect, the third aspect, and any optional method thereof.

[0179] In a seventh aspect, embodiments of this application provide a computer program, including code, which, when executed, is used to implement the first aspect, the third aspect, and any optional method thereof described above.

[0180] Eighthly, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices.

[0181] This application provides a model training method, comprising: obtaining a first neural network model, the first neural network model including a first convolutional layer; obtaining a plurality of second neural network models based on the first neural network model, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation, the linear operation being equivalent to a convolutional layer; training the plurality of second neural network models to obtain a target neural network model, the target neural network model being the neural network model with the highest model accuracy among the plurality of trained second neural network models. By replacing the convolutional layer in the neural network to be trained with a linear operation that can be equivalent to a convolutional layer, and selecting the most accurate replacement method from multiple replacement methods, the accuracy of the trained model is improved. Attached Figure Description

[0182] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0183] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0184] Figure 3 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0185] Figure 4 A schematic diagram of a system architecture provided for an embodiment of this application;

[0186] Figure 5 This is an illustration of an embodiment of a model training method provided in this application.

[0187] Figure 6a A schematic diagram of a linear operation provided for an embodiment of this application;

[0188] Figure 6b A schematic diagram of a linear operation provided for an embodiment of this application;

[0189] Figure 6c A schematic diagram of a linear operation provided for an embodiment of this application;

[0190] Figure 7 A schematic diagram of the receptive field of a convolutional layer provided in an embodiment of this application;

[0191] Figure 8 A schematic diagram of the receptive field of a convolutional layer provided in an embodiment of this application;

[0192] Figure 9 A schematic diagram of a convolutional layer provided for an embodiment of this application;

[0193] Figure 10 A schematic diagram of a convolution kernel provided in an embodiment of this application;

[0194] Figure 11 This application provides a schematic diagram of linear operation fusion as an embodiment of the present application;

[0195] Figure 12 This application provides a schematic diagram of a linear operation replacement.

[0196] Figure 13 A schematic diagram of a linear operation provided for an embodiment of this application;

[0197] Figure 14 This is a schematic diagram of a zero-padding operation provided in an embodiment of this application;

[0198] Figure 15a This application provides an illustration of an application scenario for a model training method.

[0199] Figure 15b This application provides an illustration of an application scenario for a model training method.

[0200] Figure 16a This application provides an illustration of an application scenario for a model training method.

[0201] Figure 16b This is an illustration of an embodiment of a model training method provided in this application.

[0202] Figure 17 A schematic diagram of a model training device provided in an embodiment of this application;

[0203] Figure 18A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0204] Figure 19 This is a schematic diagram of a training device provided in an embodiment of this application;

[0205] Figure 20 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0206] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0207] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0208] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0209] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0210] (1) Infrastructure

[0211] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0212] (2) Data

[0213] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0214] (3) Data processing

[0215] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0216] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0217] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0218] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0219] (4) General ability

[0220] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0221] (5) Smart Products and Industry Applications

[0222] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, and safe cities.

[0223] The method provided in this application is described below from the perspectives of model training and model application:

[0224] The model training method provided in this application can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data to finally obtain a trained neural network model (such as the target neural network model in this application embodiment). The target neural network model can be used for model inference. Specifically, input data can be input into the target neural network model to obtain output data.

[0225] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0226] (1) Neural Network

[0227] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0228] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0229] (2) A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer refers to the layer of neurons in a CNN that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in neighboring layers. A convolutional layer typically contains several feature planes, each of which can be composed of several rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernels. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of a certain part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, we can use the same learned image information for all locations in the image. Within the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0230] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0231] Specifically, such as Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0232] The structure consisting of the convolutional layer / pooling layer 120 and the neural network layer 130 can be the first convolutional layer and the second convolutional layer described in this application. The input layer 110 is connected to the convolutional layer / pooling layer 120, and the convolutional layer / pooling layer 120 is connected to the neural network layer 130. The output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform non-linear processing on the output of the neural network layer 130.

[0233] Convolutional / pooling layers 120:

[0234] Convolutional layers:

[0235] like Figure 2The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0236] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0237] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0238] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0239] Pooling layer:

[0240] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 2 In the example of 120, each layer 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0241] Neural network layer 130:

[0242] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 2 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0243] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0244] It should be noted that, as Figure 2The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 3 The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0245] (3) Deep Neural Networks

[0246] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0247] (4) Loss Function

[0248] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0249] (5) Backpropagation algorithm

[0250] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0251] (6) Linear operations

[0252] Linearity refers to a proportional, linear relationship between quantities. Mathematically, it can be understood as a function whose first derivative is a constant. Linear operations include, but are not limited to, addition, empty operations, identity operations, convolution, batch normalization (BN), and pooling. Linear operations can also be called linear mappings. Linear mappings must satisfy two conditions: homogeneity and additivity. If either condition is not met, it is nonlinear.

[0253] Homogeneity refers to f(ax) = af(x); additivity refers to f(x+y) = f(x) + f(y); for example, f(x) = ax is linear. It's important to note that x, a, and f(x) are not necessarily scalars; they can be vectors or matrices, forming a linear space of arbitrary dimensions. If x and f(x) are n-dimensional vectors, homogeneity is equivalent to a constant when a is a constant, and additivity is equivalent to a matrix when a is a matrix. Conversely, functions whose graphs are straight lines do not necessarily conform to linear mappings. For example, f(x) = ax + b does not satisfy either homogeneity or additivity, and therefore belongs to nonlinear mappings.

[0254] In the embodiments of this application, the composition of multiple linear operations can be referred to as a linear operation, and each linear operation included in a linear operation can also be referred to as a sub-linear operation.

[0255] (7) BN: By normalizing in small batches, the differences in parameter optimization between different levels of input are eliminated, reducing the possibility of overfitting in a certain layer of the model, making training more stable.

[0256] Figure 4 This is a schematic diagram of a system architecture provided in an embodiment of this application. Figure 4 In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140.

[0257] During the preprocessing of input data by the execution device 120, or during the calculation module 111 of the execution device 120 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 120 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0258] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0259] Optionally, the customer device 140 may be, for example, a control unit in an autonomous driving system or a functional algorithm module in a mobile terminal, such as a functional algorithm module that can be used to perform related tasks.

[0260] It is worth noting that the training device 120 can generate corresponding target models / rules (such as the target neural network model in this embodiment) based on different training data for different objectives or tasks. The corresponding target models / rules can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0261] exist Figure 4 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can take the form of display, sound, or action. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0262] It is worth noting that, Figure 4 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 4 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0263] First, the model training method provided in this application embodiment will be explained using the model training stage as an example.

[0264] Reference Figure 5 , Figure 5 An embodiment of a model training method provided in this application is illustrated below. Figure 5 As shown, an embodiment of this application provides a model training method comprising:

[0265] 501. Obtain a first neural network model, wherein the first neural network model includes a first convolutional layer.

[0266] In this embodiment of the application, the training device can acquire a first neural network model to be trained, which may be a model to be trained given by the user.

[0267] In this embodiment, the training device can replace some or all of the convolutional layers in the first neural network model with linear operations. The replaced convolutional layer object can be a first convolutional layer included in the first neural network model. Specifically, the first neural network model can include multiple convolutional layers, and the first convolutional layer is one of those multiple convolutional layers.

[0268] In this embodiment of the application, the training device can select the convolutional layer (including the first convolutional layer) that needs to be replaced from the first neural network model.

[0269] In one implementation, the administrator can specify the convolutional layers in the first neural network model that need to be replaced, or the training device can determine the convolutional layers in the first neural network model that need to be replaced through model structure search. How the training device determines the convolutional layers that need to be replaced through model structure search will be described in subsequent embodiments and will not be repeated here.

[0270] 502. Based on the first neural network model, obtain a plurality of second neural network models, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation, and the linear operation is equivalent to a convolutional layer.

[0271] In this embodiment of the application, the training device can replace the first convolutional layer in the first neural network model with a linear operation to obtain a second neural network model, and then obtain multiple second neural network models, each of which is obtained by replacing the first convolutional layer in the first neural network model with a linear operation.

[0272] In this embodiment of the application, the linear operation is equivalent to a convolutional layer.

[0273] In this application's embodiments, "equivalence" refers to the relationship between two computational units. Specifically, it means that two formally different computational units produce the same processing result when processing any identical data. One of the computational units can be transformed into the form of the other through mathematical derivation. For this application's embodiments, sub-linear operations included in a linear operation can be transformed into a convolutional layer through mathematical derivation, and the resulting convolutional layer and the linear operation produce the same processing result when processing the same data.

[0274] In this embodiment, to enable a linear operation to be equivalent to a convolutional layer, at least one of the multiple sub-linear operations included in the linear operation must be a convolutional operation. Specifically, a linear operation is composed of multiple sub-linear operations. Here, a sub-linear operation can refer to a basic linear operation, not an operation composed of multiple basic linear operations. For example, the operation type of the sub-linear operation can be, but is not limited to, addition, null operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation. Correspondingly, a linear operation can refer to a combination of at least one of the following sub-linear operations: addition, null operation, identity operation, convolution operation, batch normalization (BN) operation, and pooling operation. It should be understood that "composition" here means that the number of sub-linear operations is greater than or equal to two, and there is a connection between the sub-linear operations; there are no isolated sub-linear operations. A connection means that the output of one sub-linear operation is used as the input of another sub-linear operation (except for sub-linear operations located on the output side of the linear operation, whose output is used as the output of the linear operation).

[0275] For example, you can refer to Figure 6a , Figure 6b as well as Figure 6c , Figure 6a , Figure 6b as well as Figure 6c These are schematic diagrams of several linear operations in the embodiments of this application, wherein, Figure 6a The linear operation shown includes four sub-linear operations: convolution operation 1 (convolution size k*k), convolution operation 2 (convolution size 1*1), convolution operation 3 (convolution size k*k), and a summation operation. Convolution operation 1 processes the input data of the linear operation to obtain output 1, convolution operation 2 processes the input data of the linear operation to obtain output 2, convolution operation 3 processes output 2 to obtain output 3, and the summation operation sums output 1 and output 3 to obtain the output of the linear operation.

[0276] in, Figure 6bThe linear operation shown includes seven sub-linear operations: convolution operation 1 (convolution size k*k), convolution operation 2 (convolution size 1*1), convolution operation 3 (convolution size k*k), convolution operation 4 (convolution size 1*1), convolution operation 5 (convolution size k*k), convolution operation 6 (convolution size 1*1), and a summation operation. Convolution operation 1 processes the input data of the linear operation to obtain output 1; convolution operation 2 processes the input data of the linear operation to obtain output 2; convolution operation 3 processes output 2 to obtain output 3; convolution operation 4 processes the input data of the linear operation to obtain output 4; convolution operation 5 processes output 4 to obtain output 5; convolution operation 6 processes output 5 to obtain output 6; and the summation operation sums output 1, output 3, and output 6 to obtain the output of the linear operation.

[0277] in, Figure 6c The linear operation shown includes eight sub-linear operations: convolution operation 1 (convolution size k*k), convolution operation 2 (convolution size 1*1), convolution operation 3 (convolution size k*k), convolution operation 4 (convolution size 1*1), convolution operation 5 (convolution size 1*1), convolution operation 6 (convolution size k*k), summation operation 1, and summation operation 2. Convolution operation 1 processes the input data of the linear operation to obtain output 1; convolution operation 2 processes the input data of the linear operation to obtain output 2; convolution operation 3 processes output 2 to obtain output 3; convolution operation 4 processes output 2 to obtain output 4; convolution operation 5 processes the input data of the linear operation to obtain output 5; summation operation 1 sums output 4 and output 5 to obtain output 6; convolution operation 6 processes output 6 to obtain output 7; and summation operation 2 sums output 1, output 3, and output 7 to obtain the output of the linear operation.

[0278] The following describes the linear operation used to replace the first convolutional layer:

[0279] In this embodiment, to ensure that a linear operation can be equivalent to a convolutional layer, at least one of the multiple sub-linear operations included in the linear operation must be a convolutional operation. During subsequent model inference, to avoid reducing the speed or increasing resource consumption of the inference stage, the linear operation is not used for model inference. Instead, a convolutional layer equivalent to the linear operation (which can be referred to as the second convolutional layer in subsequent embodiments) is used for model inference. Furthermore, it is necessary to ensure that the receptive field of the convolutional layer equivalent to the linear operation is less than or equal to the receptive field of the first convolutional layer.

[0280] The following describes how to ensure that the receptive field of a linear operation is less than or equal to the receptive field of the first convolutional layer:

[0281] In this embodiment, to ensure that the equivalent receptive field of a linear operation is less than or equal to the receptive field of the first convolutional layer, the equivalent receptive field of each operation branch in the linear operation needs to be less than or equal to the receptive field of the first convolutional layer. The receptive field of each operation branch in the linear operation will be described in detail below.

[0282] First, let's describe the concept of operation branches:

[0283] Taking the input and output of a linear operation as its two endpoints, a data path between these endpoints can be considered an operation branch. The starting point of the operation branch is the input of the linear operation, and the ending point is the output of the linear operation. In one implementation, the linear operation can include multiple parallel operation branches. Each operation branch processes the input data of the linear operation, which is equivalent to each operation branch starting from the input of the linear operation. Furthermore, the input of the sub-linear operation closest to the input of the linear operation in each operation branch is the input data of the linear operation, meaning each operation branch processes the input data of the linear operation. Each operation branch includes at least one serial sub-linear operation. Alternatively, a linear operation can be represented as a computation graph that defines the input sources and output data flow of each sub-linear operation. Any path from input to output in this computation graph can be defined as an operation branch of the linear operation.

[0284] For example, you can refer to Figure 6a , Figure 6a The linear operation shown may include two operation branches (represented as operation branch 1 and operation branch 2 in this embodiment). Operation branch 1 includes convolution operation 1 and addition operation, and operation branch 2 includes convolution operation 2, convolution operation 3 and addition operation. Both operation branch 1 and operation branch 2 are used to process the input data of the linear operation. The data flow of operation branch 1 is from convolution operation 1 to addition operation, that is, the input data of the linear operation is processed by convolution operation 1 and addition operation in sequence. The data flow of operation branch 2 is from convolution operation 2, convolution operation 3 to addition operation, that is, the input data of the linear operation is processed by convolution operation 2, convolution operation 3 and addition operation in sequence.

[0285] For example, you can refer to Figure 6b , Figure 6bThe linear operation shown may include three operation branches (represented as operation branch 1, operation branch 2, and operation branch 3 in this embodiment). Operation branch 1 includes convolution operation 1 and addition operation; operation branch 2 includes convolution operation 2, convolution operation 3, and addition operation; and operation branch 3 includes convolution operation 4, convolution operation 5, convolution operation 6, and addition operation. Operation branches 1, 2, and 3 are all used to process the input data of the linear operation. The data flow of operation branch 1 is from convolution operation 1 to addition operation, that is, the input data of the linear operation is processed by convolution operation 1 and addition operation in sequence. The data flow of operation branch 2 is from convolution operation 2 and convolution operation 3 to addition operation, that is, the input data of the linear operation is processed by convolution operation 2, convolution operation 3, and addition operation in sequence. The data flow of operation branch 3 is from convolution operation 4, convolution operation 5, convolution operation 6 to addition operation, that is, the input data of the linear operation is processed by convolution operation 4, convolution operation 5, convolution operation 6, and addition operation in sequence.

[0286] For example, you can refer to Figure 6c , Figure 6c The linear operation shown may include four operation branches (represented as operation branch 1, operation branch 2, operation branch 3, and operation branch 4 in this embodiment). Operation branch 1 includes convolution operation 1 and addition operation 2; operation branch 2 includes convolution operation 2, convolution operation 3, and addition operation 2; operation branch 3 includes convolution operation 2, convolution operation 4, addition operation 1, convolution operation 6, and addition operation 1; and operation branch 4 includes convolution operation 5, addition operation 1, convolution operation 6, and addition operation 2. Operation branches 1, 2, 3, and 4 are all used to process the input data of the linear operation. The data flow of operation branch 1 is from convolution operation 1 to addition operation 2, meaning the input data of the linear operation is sequentially processed through convolution operation 1. The data flow of operation branch 2 is from convolution operation 2, convolution operation 3 to addition operation 2. That is, the input data of the linear operation is processed by convolution operation 2, convolution operation 3 and addition operation 2 in sequence. The data flow of operation branch 3 is from convolution operation 2, convolution operation 4, summation operation 1, convolution operation 6 to summation operation 1. That is, the input data of the linear operation is processed by convolution operation 2, convolution operation 4, summation operation 1, convolution operation 6 and summation operation 1 in sequence. The data flow of operation branch 4 is from convolution operation 5, summation operation 1, convolution operation 6 to summation operation 2. That is, the input data of the linear operation is processed by convolution operation 5, summation operation 1, convolution operation 6 and summation operation 2 in sequence.

[0287] The equivalent receptive field of each operation branch in the linear operation will be described next.

[0288] For a single sub-linear operation, such as a k*k convolution or pooling operation, the receptive field is k, while the receptive field of an addition operation and a BN operation is 1. The equivalent receptive field k of an operation branch is defined as follows: each output of the operation branch is influenced by kxk inputs. The receptive field of an operation branch is calculated as follows: assuming an operation branch includes N sub-linear operations, and each of the N sub-linear operations has a receptive field ki (i is a positive integer less than or equal to N), then the equivalent receptive field of the N sub-linear operations is k1 + k2 + ... + kN - (N-1). For example, the equivalent receptive field of two 3x3 convolution operations is 3 + 3 - 1 = 5.

[0289] For example, Figure 6a The equivalent receptive field of operation branch 1 in the linear operation is k (calculated as k+1-1=k).

[0290] For example, Figure 6a The equivalent receptive field of operation branch 2 in the linear operation is k (calculated as 1+k+1-2=k).

[0291] For example, Figure 6b The equivalent receptive field of operation branch 1 in the linear operation is k (calculated as k+1-1=k).

[0292] For example, Figure 6b The equivalent receptive field of operation branch 2 in the linear operation is k (calculated as 1+k+1-2=k).

[0293] For example, Figure 6b The equivalent receptive field of operation branch 3 in the linear operation is k (calculated as 1+k+1+1-3=k).

[0294] For example, Figure 6c The equivalent receptive field of operation branch 1 in the linear operation is k (calculated as k+1-1=k).

[0295] For example, Figure 6c The equivalent receptive field of operation branch 2 in the linear operation is k (calculated as 1+k+1-2=k).

[0296] For example, Figure 6c The equivalent receptive field of operation branch 3 in the linear operation is k (calculated as 1+1+1+k+1-4=k).

[0297] For example, Figure 6c The equivalent receptive field of operation branch 4 in the linear operation is k (calculated as 1+1+k+1-2=k).

[0298] In this embodiment, the receptive field of the convolutional layer equivalent to the linear operation is consistent with the receptive field of the linear operation, and the receptive field of the linear operation is equal to the largest receptive field among the various operation branches. For example, if the receptive fields of the various operation branches included in the linear operation are 3, 5, 5, 5, and 7 respectively, then the receptive field of the linear operation is equal to 7.

[0299] In order to ensure that the receptive field of the equivalent convolutional layer for a linear operation is less than or equal to the receptive field of the first convolutional layer, it is necessary to guarantee that the receptive field of the linear operation is less than or equal to the receptive field of the first convolutional kernel. In other words, the equivalent receptive field of each branch of the linear operation is less than or equal to the receptive field of the first convolutional layer.

[0300] In one implementation, the linear operation may include only one operation branch for processing the input data of the linear operation. The operation branch includes at least one sequential sub-linear operation. In this case, the equivalent receptive field of the operation branch included in the linear operation is less than or equal to the receptive field of the first convolutional layer.

[0301] The concept of the receptive field of a convolutional layer will be described next.

[0302] Taking images as the object of processing as an example, the receptive field refers to the perceptual area (receptive range) of a feature in the input image. If the pixels within this receptive range change, the value of that feature will change accordingly. For example... Figure 7 As shown, the convolutional kernel slides across the input image, and the extracted features constitute convolutional layer 101. Similarly, the convolutional kernel slides across convolutional layer 101, and the extracted features constitute convolutional layer 102. Therefore, each feature in convolutional layer 101 is extracted from pixels of the input image within the size of the convolutional slice of the sliding convolutional kernel; this size is also the receptive field of convolutional layer 101. Thus, the receptive field of convolutional layer 101 is as follows: Figure 7 As shown.

[0303] Correspondingly, the extent to which each feature in convolutional layer 102 is mapped onto the input image (i.e., how many pixels in the input image are used) is also the receptive field of convolutional layer 102. For example... Figure 8 As shown, each feature in convolutional layer 102 is extracted from pixels of the input image within the size of the convolutional sheet of the sliding convolutional kernel on convolutional layer 101. Conversely, each feature in convolutional layer 101 is extracted from pixels of the input image within the range of the convolutional sheet of the sliding convolutional kernel on the input image. Therefore, the receptive field of convolutional layer 102 is larger than that of convolutional layer 101.

[0304] In one implementation, the equivalent receptive field of at least one of the multiple parallel operation branches is equal to the receptive field of the first convolutional layer. Then, the receptive field of the linear operation is equal to the receptive field of the first convolutional layer. Consequently, the receptive field of the convolutional layer equivalent to the linear operation (hereinafter referred to as the second convolutional layer) is equal to the receptive field of the first convolutional layer. The second convolutional layer can be used for subsequent model inference. Since the receptive field of the second convolutional layer is consistent with that of the first convolutional layer, under the premise of ensuring that the size specifications of the neural network model that has not been replaced are consistent, that is, under the premise of ensuring that the speed and resource consumption of the inference stage remain unchanged, the receptive field of the second convolutional layer is smaller than that of the first convolutional layer, which increases the number of training parameters and improves the accuracy of the model.

[0305] The above describes the linear operations used to replace convolutional layers. In this embodiment, the training device can obtain multiple linear operations and replace the first convolutional layer in the first neural network model with one of the multiple linear operations (or replace multiple convolutional layers (including the first convolutional layer) in the first neural network model with one of the multiple linear operations), and so on, to obtain multiple second neural network models, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation.

[0306] The following describes how to obtain multiple linear operations:

[0307] In this embodiment, a sampling-based search algorithm, such as reinforcement learning or genetic algorithm, can be selected, and the search space including linear operations can be encoded. For example, a feasible encoding method is to first sequentially encode the optional sub-linear operations, for example, encoding the no-op, identity operation, 1x1 convolution, 3x3 convolution, BN, and 3x3 pooling as 0, 1, 2, 3, 4, and 5 respectively, and then using the adjacency matrix M to represent the computation graph of a set of linear operations. For a computation graph with N nodes (excluding input nodes), the adjacency matrix M is an N*(N+1) matrix, with row numbers 1-N and column numbers 0-N. The value M[i,j] in the i-th row and j-th column of the matrix represents the result of the operation corresponding to M[i,j] from the output of the j-th node, added to the i-th node. M[i,j] = 0 indicates that there is no direct operation connecting the j-th node to the i-th node. Based on this encoding scheme, Figure 11 The codes corresponding to the linear operations shown in Table 1 can be represented as follows: (assuming k = 3):

[0308] Table 1

[0309] 3 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 3 0 0 0 1 0 1 0 0 0 1 0

[0310] Then, the encoding of linear operations can be sampled according to the search algorithm. For each sampled linear operation encoding, the first convolution in the first neural network model can be replaced with the linear operation corresponding to the linear operation encoding.

[0311] In one implementation, only one second neural network model can be obtained. This involves determining a target linear operation and replacing the first convolutional layer in the first neural network model with the determined target linear operation to obtain the second neural network model. Specifically, the training device can obtain the second neural network model based on the first neural network model. The second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation. The target linear operation includes multiple sub-linear operations, which are equivalent to a single convolutional layer. The target linear operation includes M operation branches, and the input of each operation branch is the input of the target linear operation. The multiple sub-linear operations satisfy at least one of the following conditions:

[0312] The plurality of sub-linear operations include at least three types of operations; M is not 3; at least one of the M operation branches includes a number of sub-linear operations that are not equal to 2, and M is a positive integer; or, at least one of the M operation branches has a number of sub-linear operations of type convolution that are not 1.

[0313] 503. Train the plurality of second neural network models to obtain a target neural network model, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of trained second neural network models.

[0314] In this embodiment of the application, the training device can perform model training on the acquired plurality of second neural network models to obtain a plurality of trained second neural network models, and determine a target neural network model from the plurality of trained second neural network models, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of second neural network models.

[0315] It should be understood that the execution of the action of acquiring multiple second neural networks in step 502 is not strictly after the execution time of the action of training the multiple second neural network models in step 503. For example, after the training device can acquire a second neural network model, it can train the second neural network model, and after the training is completed, it can acquire the next second neural network model, and so on. Thus, the training device can acquire multiple second neural network models and train multiple second neural network models.

[0316] The number of second neural network models can be pre-specified by the administrator, or it can be the number of second neural network models that have been trained when the search resource limit is reached during the training process of the training equipment.

[0317] In this embodiment of the application, when training the second neural network model, the model accuracy (or verification accuracy) of each trained second neural network model can be obtained. Based on the model accuracy of each second neural network model, the second neural network model with the highest model accuracy can be selected from multiple second neural network models.

[0318] Taking the second neural network model with the highest model accuracy as the target neural network model as an example, the second neural network model corresponding to the target neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation, and the neural network model with the highest accuracy includes the target linear operation after training.

[0319] Since the target linear operation includes multiple sub-linear operations compared to the first convolutional layer, directly using this target neural network model for model inference would reduce the inference speed and increase the resource consumption required for inference. Therefore, in this embodiment, an equivalent second convolutional layer to the trained target linear operation can be obtained, and the trained target linear operation in the target neural network model can be replaced with the second convolutional layer to obtain a third neural network model, which can be used for model inference.

[0320] It should be understood that, in this embodiment of the application, the step of obtaining the second convolutional layer equivalent to the trained target linear operation and replacing the trained target linear operation in the target neural network model with the second convolutional layer to obtain the third neural network model can be performed by a training device. After training is completed, the training device can directly feed back the third neural network model. Specifically, the training device can send the third neural network model to a terminal device or server so that the terminal device or server can perform model inference based on the third neural network model. Alternatively, the terminal device or server can obtain the second convolutional layer equivalent to the trained target linear operation and replace the trained target linear operation in the target neural network model with the second convolutional layer before performing model inference to obtain the action execution of the third neural network model.

[0321] The following describes how to obtain the equivalent second convolutional layer for the linear operation of the target after training:

[0322] In this embodiment of the application, each sub-linear operation can be merged into an adjacent sub-linear operation located after the order in which the multiple sub-linear operations included in the trained target linear operation are processed when processing data, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0323] For each sub-linear operation, it can be merged into an adjacent sub-linear operation that is later in the sequence, until the merging is completed up to the last sub-linear operation (the sub-linear operation closest to the output).

[0324] In this embodiment of the application, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation. The first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0325] Specifically, in this embodiment, the first sub-linear operation and the second sub-linear operation are any adjacent sub-linear operations in the trained target linear operation, and the second sub-linear operation is the sub-linear operation following the first sub-linear operation in the sequence. The first sub-linear operation includes a first operation parameter, and the first sub-linear operation is used to process the input data of the first sub-linear operation according to the first operation parameter, corresponding to the operation type of the first sub-linear operation. The second sub-linear operation includes a second operation parameter, and the second sub-linear operation is used to process the input data of the second sub-linear operation according to the second operation parameter, corresponding to the operation type of the second sub-linear operation. Thus, the fusion parameter of the first sub-linear operation can be obtained. If the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameter of the first sub-linear operation is the first operation parameter, and the fusion parameter of the second sub-linear operation is obtained according to the fusion parameter of the first sub-linear operation, the second operation parameter, and the operation type of the second sub-linear operation. If the second sub-linear operation is the last sub-linear operation in the sequence, then the fusion parameter of the second sub-linear operation is used as the operation parameter of the second convolutional layer.

[0326] In one possible implementation, the sub-linear operation type of the linear operation includes at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation or pooling operation. Both convolution operation and BN operation include trainable operation parameters. For the representation of the adjacency matrix, an empty operation (0) is required, which is equivalent to no operation from node i to node j.

[0327] In this embodiment, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0328] For a detailed illustration of the fusion strategy, please refer to [link / reference]. Figure 11 , Figure 11 The second sub-linear operation is of type addition (or summation). Figure 11 The text describes operations such as addition, convolution, pooling, and batch normalization (BN) using examples.

[0329] For linear operations in the trained target neural network, the fusion parameter is equal to the fusion (output node). The fusion process is performed on every linear operation in the model, ultimately resulting in a fully fused model. This model has the same structure as the original model, thus maintaining the same speed and resource consumption during the inference phase. Furthermore, the models before and after fusion are mathematically equivalent, ensuring that the accuracy of the fused model remains consistent with that of the original model.

[0330] The following section uses ResNet18 as the first neural network model as an example to describe the model training method in this application embodiment, with a specific instance:

[0331] like Figure 12 As shown, the convolutional layers in the first neural network model are replaced with linear operations. Here, you can choose to replace only a portion of the convolutional layers, or replace them all. The form of the linear operations replaced by different convolutional layers can be different; here, we only use linear operations as an example. Figure 12 Taking the overparameterized form C as an example, after the replacement is completed, the second neural network model obtained after the replacement is trained according to the training process of the original model to obtain the trained model.

[0332] After obtaining the trained second neural network model, parameter fusion is required for each linear operation. For example... Figure 13 As shown ( Figure 13 Each sub-linear operation is represented as nodes 1-8. Taking the overparameterized form C as an example, the specific fusion process can be as follows:

[0333] For nodes 1, 2, and 4, since nodes 1, 2, and 4 are all used to process the input of linear operations (that is, nodes directly connected to the input node 0), the fusion parameter of node 1 is the operation parameter of node 1, the fusion parameter of node 2 is the operation parameter of node 2, and the fusion parameter of node 4 is the operation parameter of node 4.

[0334] For node 5, node 5 is used to process the output of node 2 according to the operation parameters of node 5, corresponding to the operation type of node 5 (convolution operation). Therefore, the fusion parameter of node 5 is the inner product of the fusion parameter of node 2 and the operation parameter of node 5.

[0335] For node 6, node 6 is used to process the outputs of node 5 and node 4 according to the operation type of node 6 (addition operation). Therefore, the fusion parameter of node 6 is the sum of the fusion parameter of node 5 and the operation parameter of node 4.

[0336] For node 3, node 3 is used to process the output of node 2 according to the operation parameters of node 3, corresponding to the operation type of node 3 (convolution operation). Therefore, the fusion parameter of node 3 is the inner product of the fusion parameter of node 2 and the operation parameter of node 3.

[0337] For node 7, node 7 is used to process the output of node 6 according to the operation parameters of node 7, corresponding to the operation type of node 7 (convolution operation). Therefore, the fusion parameter of node 7 is the inner product of the fusion parameter of node 6 and the operation parameter of node 7.

[0338] For node 8, node 8 is used to process the outputs of node 1, node 3, and node 7 according to the operation type of node 8 (addition operation). Therefore, the fusion parameter of node 8 is the sum of the fusion parameter of node 1, the fusion parameter of node 3, and the operation parameter of node 7.

[0339] The fusion parameters of node 8 can then be used as the operation parameters of the second convolutional layer, which can then perform convolution operations on the input data based on these parameters.

[0340] The following is a description from the perspective of pseudocode. Figure 13 The fusion process of linear operations:

[0341] Merge parameter = Merge (node ​​8): Addition, with predecessor nodes being 1, 3, and 7.

[0342] Node 1 fusion parameter = fusion(node ​​1): convolution, directly connected to the input, returns parameters.

[0343] Node 3 fusion parameter = fusion(node ​​3): convolution, preceding node 2

[0344] Node 2 fusion parameter = fusion(node ​​2): convolution, directly connected to the input, returns parameters.

[0345] Return the inner product of the parameters of node 3 and the fusion parameters of node 2.

[0346] Node 7 fusion parameter = fusion(node ​​7): convolution, preceding node 6

[0347] Node 6 fusion parameter = fusion(node ​​6): addition, with preceding nodes being 5 and 4.

[0348] Node 5 fusion parameter = fusion(node ​​5): convolution, preceding node 2

[0349] Node 2 fusion parameter = fusion(node ​​2): convolution, directly connected to the input, returns parameters.

[0350] Return the inner product of the parameters of node 5 and the fusion parameters of node 2.

[0351] Node 4 fusion parameter = fusion(node ​​4): convolution, directly connected to the input, returns parameters.

[0352] Returns the sum({node 5 fusion parameter, node 4 fusion parameter})

[0353] Return the inner product of the parameters of node 7 and the fusion parameters of node 6.

[0354] Returns the sum({node1 fusion parameter, node3 fusion parameter, node7 fusion parameter})

[0355] For each linear operation, the sub-linear operations are fused following the process described above, ultimately resulting in a fully fused model. This fused model has the same structure as the original ResNet-18 model.

[0356] In this embodiment of the application, in order to ensure that the model used for inference has the same specifications as the first neural network model before training, the size of the second convolutional layer needs to be consistent with the size of the first convolutional layer.

[0357] Next, we will first describe the concept of the size of a convolutional layer.

[0358] The size of a convolutional layer can represent the number of features it includes. For example, the size of a convolutional layer will be explained below in conjunction with the convolutional layer and its kernel. Figure 9As shown, the size of convolutional layer 101 is X*Y*N1, meaning it includes X*Y*N1 features. Here, N1 is the number of channels, with one channel representing one feature dimension, and X*Y representing the number of features per channel. X, Y, and N1 are all positive integers greater than 0. Convolutional kernel 1011 is one of the convolutional kernels used in convolutional layer 101. Since convolutional layer 102 includes N2 channels, convolutional layer 101 uses a total of N2 convolutional kernels. The size and model parameters of these N2 convolutional kernels can be the same or different. Taking convolutional kernel 1011 as an example, its size is X1*X1*N1, meaning it includes X1*X1*N1 model parameters. Convolutional kernel 1011 slides within convolutional layer 101. When it reaches a certain position in convolutional layer 101, the model parameters of convolutional kernel 1011 are multiplied by the features of convolutional layer 101 at the corresponding position. The products of each model parameter of convolutional kernel 1011 and the features of convolutional layer 101 at the corresponding position are combined to obtain a feature for one channel of convolutional layer 102. The product of the features of convolutional layer 101 and convolutional kernel 1011 can be directly used as the feature of convolutional layer 102. Alternatively, after the features of convolutional layer 101 and convolutional kernel 1011 have finished sliding on convolutional layer 101 and all product results have been output, all product results are normalized, and the normalized product result is used as the feature of convolutional layer 102. In simpler terms, convolutional kernel 1011 slides on convolutional layer 101 to perform convolution, and the result of the convolution forms one channel of convolutional layer 102. Each convolutional kernel used in convolutional layer 101 corresponds to one channel of convolutional layer 102. Therefore, the number of channels in convolutional layer 102 is equal to the number of convolutional kernels acting on convolutional layer 101. The design of the model parameters within each convolutional kernel reflects the characteristics of the features that the kernel aims to extract from the convolutional layer. Through N² convolutional kernels, convolutional layer 101 extracts features from N² channels.

[0359] like Figure 10 As shown, convolution kernel 1011 is split into N1 convolutional slices, each of which includes X1*X1 model parameters (P11 to Px1x1). Each model parameter corresponds to a convolution point. The model parameter corresponding to a convolution point is multiplied by the features in the convolutional layer at the corresponding position to obtain the convolution result of that convolution point. The sum of the convolution results of the convolution points of a convolution kernel is the convolution result of the entire convolution kernel.

[0360] In one implementation, if the receptive field of the target linear operation is equal to the receptive field of the first convolutional layer, then the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0361] In one implementation, if the receptive field of the target linear operation is smaller than that of the first convolutional layer, the size of the calculated equivalent convolutional layer will be smaller than that of the first convolutional layer. In this case, zero-padding can be performed on the calculated equivalent convolutional layer to obtain a second convolutional layer with the same size as the first convolutional layer. See [link to details] for more information. Figure 14 , Figure 14 This is a schematic diagram of a zero-padding operation in an embodiment of this application.

[0362] In this embodiment, the accuracy of the trained model is improved by replacing the convolutional layers in the neural network to be trained with linear operations that can be equivalent to convolutional layers, and selecting the most accurate replacement method from multiple options. Referring to Table 2, which shows the accuracy of the network using different replacement methods (represented as overparameterization in Table 2), specifically, in this task, a lower loss indicates a stronger model fitting ability and higher model accuracy. As shown in Table 2, for both model structures, the loss after overparameterization training is lower than the baseline of the original model structure. Furthermore, the optimal overparameterization form differs for different model structures.

[0363] Table 2

[0364] loss Baseline Overparameterized form A Through parameterized form B Overparameterized form C Model Structure 1 1.625 1.581 1.582 1.598 Model Structure 2 1.589 1.574 1.564 1.563

[0365] This application provides a model training method, comprising: obtaining a first neural network model, the first neural network model including a first convolutional layer; obtaining a plurality of second neural network models based on the first neural network model, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation, the linear operation being equivalent to a convolutional layer; training the plurality of second neural network models to obtain a target neural network model, the target neural network model being the neural network model with the highest model accuracy among the plurality of trained second neural network models. By replacing the convolutional layer in the neural network to be trained with a linear operation that can be equivalent to a convolutional layer, and selecting the most accurate replacement method from multiple replacement methods, the accuracy of the trained model is improved.

[0366] Next, from the perspective of product application, we will introduce several application scenarios of the embodiments of this application.

[0367] Typical application scenarios of this application embodiment may include neural network models on terminal devices. Specifically, the model trained using the training method provided in this application embodiment can be deployed on a terminal device (e.g., a smartphone) or a cloud server to provide inference capabilities. Specifically, it can be as follows: Figure 15a As shown, the first neural network model ( Figure 15aThe model (referred to as DNN model in the text) is trained using the training method provided in the embodiments of this application, and the fused overparameterized model is deployed on a terminal device or cloud server to perform inference on the user's data.

[0368] The training method provided in this application can also be applied to AutoML services in the cloud, and combined with other AutoML techniques such as data augmentation strategy search, model structure search, activation function search, and hyperparameter search to further improve model performance. Specifically, it can be as follows: Figure 15b as well as Figure 16a As shown: Users provide training data, model structure, and specify the target task. The AutoML service on the cloud automatically performs a parametric search and finally outputs the searched model and its corresponding parameters. Alternatively, parametric training can be combined with other AutoML techniques, such as data augmentation strategy search, model structure search, activation function search, hyperparameter search, etc., to further improve the model performance.

[0369] Reference Figure 16b , Figure 16b This is a flowchart illustrating a model training method provided in an embodiment of this application, as shown below. Figure 16b As shown in the embodiments of this application, a model training method includes:

[0370] 1601. Obtain a first neural network model, the first neural network model including a first convolutional layer, the first neural network model being used to achieve the target task;

[0371] The specific description of step 1601 can be found in the description of step 501, and will not be repeated here.

[0372] 1602. Determine a target linear operation for replacing the first convolutional layer based on at least one of the following information, wherein the information includes the network structure of the first neural network model, the target task, and the position of the first convolutional layer in the first neural network model, and the target linear operation is equivalent to a convolutional layer;

[0373] Among them, different linear operations can be selected for neural network models with different network structures, neural network models that achieve different target tasks, and convolutional layers at different positions in the neural network model, so that the replaced neural network model has higher accuracy after training.

[0374] The target linear operation can be determined based on the network structure of the first neural network model and / or the position of the first convolutional layer within the first neural network model. Specifically, the structure of the target linear operation can be determined based on the network structure of the first neural network model. The network structure of the first neural network model can include the number of sub-network layers, the types of sub-network layers, the connection relationships between sub-network layers, and the position of the first convolutional layer within the first neural network model. The structure of the target linear operation can refer to the number of sub-linear operations included in the target linear operation, the types of sub-linear operations, and the connection relationships between sub-linear operations. For example, it can be based on model search, performing linear operation replacement on convolutional layers of neural network models with different network structures, and... The replaced neural network model is trained to determine the optimal or relatively optimal linear operation corresponding to each convolutional layer in the network structure of each neural network model. The optimal or relatively optimal linear operation refers to the model with higher accuracy obtained by training the replaced neural network model. After obtaining the first neural network model, based on the network structure of the first neural network model, a neural network model with the same or similar structure can be selected from the network structures of the neural network models obtained in the previous search. The linear operation corresponding to a convolutional layer in the same or similar neural network model is determined as the target linear operation. The relative position of the "convolutional layer" in the same or similar neural network model is the same as or similar to the relative position of the first convolutional layer in the first neural network model.

[0375] The target linear operation can be determined based on the network structure of the first neural network model and the target task it achieves. Similar to the above determination based on the network structure of the first neural network model, linear operation replacement can be performed on the convolutional layers of neural network models with different network structures and different target tasks through model search. The replaced neural network models are then trained to determine the optimal or relatively optimal linear operation corresponding to each convolutional layer in the network structure of each neural network model. The optimal or relatively optimal linear operation refers to the model with higher accuracy obtained by training the replaced neural network model.

[0376] The target linear operation can be determined based on the target task implemented by the first neural network model. Similar to the above determination based on the network structure of the first neural network model, linear operations can be replaced by model search for the convolutional layers of neural network models that implement different target tasks. The replaced neural network models are then trained to determine the optimal or relatively optimal linear operation corresponding to each convolutional layer in the network structure of each neural network model. The optimal or relatively optimal linear operation refers to the model with higher accuracy obtained by training the replaced neural network model.

[0377] It should be understood that the above-described network structure based on the first neural network model and / or the method for determining the target linear operation of the target task is only an illustration and can be implemented in other ways, as long as the accuracy of the replaced first neural network model (i.e., the second neural network model) is high. It does not limit the specific structure and method of determining the target linear operation.

[0378] 1603. Based on the first neural network model, obtain a second neural network model, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation.

[0379] The specific description of step 1603 can be found in the description of step 502, and will not be repeated here.

[0380] 1604. Train the second neural network model to obtain the target neural network model.

[0381] The specific description of step 1604 can be found in the description of the process of training the second neural network model in step 503, and will not be repeated here.

[0382] In this embodiment, the convolutional layers in the neural network to be trained are replaced with target linear operations. The structure of the target linear operations is determined based on the structure of the first neural network model and / or the target task. Compared with the linear operations used in the prior art when replacing convolutional layers, the structure of the linear operations in this embodiment is more flexible and adaptable. Different linear operations can be designed for different model structures and task types, thereby improving the accuracy of the trained model.

[0383] In one possible implementation, the target linear operation comprises multiple sub-linear operations, and the target linear operation comprises M operation branches, each operation branch having its input as the input of the target linear operation, wherein the M operation branches satisfy at least one of the following conditions:

[0384] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0385] At least two of the M operation branches have a different number of sub-linear operations; or,

[0386] At least two of the M operation branches contain sub-linear operations of different types.

[0387] Compared to the existing technology that uses linear operations to replace convolutional layers, the target linear operation structure provided in this embodiment is more complex and can improve the accuracy of the trained model.

[0388] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0389] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0390] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0391] In one possible implementation, the target neural network model includes a trained target linear operation, and the method further includes:

[0392] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0393] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0394] In one possible implementation, the method further includes:

[0395] Based on the order in which the multiple sub-linear operations included in the trained target linear operation process data, each sub-linear operation is merged into an adjacent sub-linear operation that is later in the order, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0396] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0397] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0398] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0399] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0400] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0401] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0402] This application provides a model training method, comprising: obtaining a first neural network model, the first neural network model including a first convolutional layer, the first neural network model being used to implement a target task; determining a target linear operation for replacing the first convolutional layer based on at least one of the following information, wherein the information includes the network structure of the first neural network model, the target task, and the position of the first convolutional layer in the first neural network model, the target linear operation being equivalent to a convolutional layer; obtaining a second neural network model based on the first neural network model, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation; and training the second neural network model to obtain a target neural network model. Through the above method, the convolutional layer in the neural network to be trained is replaced with the target linear operation. The structure of the target linear operation is determined based on the structure of the first neural network model, the target task, and / or the position of the first convolutional layer in the first neural network model. Compared with the linear operations used in prior art when replacing convolutional layers, the structure of the linear operation in this embodiment is more adaptable to the first neural network model and more flexible. Different linear operations can be designed for different model structures and task types, thereby improving the accuracy of the trained model.

[0403] Furthermore, this application provides a model training method, the method comprising:

[0404] Obtain a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0405] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation. The target linear operation is equivalent to a convolutional layer, and the target linear operation includes multiple sub-linear operations. The target linear operation includes M operation branches, and the input of each operation branch is the input of the target linear operation. The M operation branches satisfy at least one of the following conditions:

[0406] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0407] At least two of the M operation branches have a different number of sub-linear operations; or,

[0408] At least two of the M operation branches contain sub-linear operations of different types.

[0409] The second neural network model is trained to obtain the target neural network model.

[0410] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0411] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0412] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0413] In one possible implementation, the target neural network model includes a trained target linear operation, and the method further includes:

[0414] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0415] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0416] In one possible implementation, the method further includes:

[0417] Based on the order in which the multiple sub-linear operations included in the trained target linear operation process data, each sub-linear operation is merged into an adjacent sub-linear operation that is later in the order, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0418] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0419] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0420] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0421] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0422] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0423] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0424] This application provides a model training method, the method comprising: obtaining a first neural network model, the first neural network model including a first convolutional layer; obtaining a plurality of second neural network models based on the first neural network model, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation, the target linear operation being equivalent to a convolutional layer, the target linear operation including a plurality of sub-linear operations, the target linear operation including M operation branches, the input of each operation branch being the input of the target linear operation, the M operation branches satisfying at least one of the following conditions: the input of at least one sub-linear operation among the plurality of sub-linear operations included in the M operation branches is the output of the plurality of sub-linear operations; the number of sub-linear operations included between at least two operation branches among the M operation branches is different; or, the operation types of the sub-linear operations included between at least two operation branches among the M operation branches are different; and training the second neural network models to obtain a target neural network model. Compared with the structure of linear operations used to replace convolutional layers in the prior art, the structure of the target linear operation provided in this embodiment is more complex, which can improve the accuracy of the trained model.

[0425] Reference Figure 17 , Figure 17 This is a schematic diagram of a model training device 1700 provided in an embodiment of this application, as shown below. Figure 17 As shown in the figure, the model training device 1700 provided in this application includes:

[0426] The acquisition module 1701 is used to acquire a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0427] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation, and the linear operation is equivalent to a convolutional layer;

[0428] The relevant description of the acquisition module 1701 can be referred to the description of steps 501 to 502 in the above embodiments, and will not be repeated here.

[0429] The model training module 1702 is used to train the plurality of second neural network models to obtain a target neural network model, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of trained second neural network models.

[0430] The description of the model training module 1702 can be referred to the description of step 503 in the above embodiment, and will not be repeated here.

[0431] In one possible implementation, the receptive field of the convolutional layer equivalent to the linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0432] In one possible implementation, the linear operation includes multiple operation branches, each operation branch receiving the input of the linear operation. Each operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer; or,

[0433] The linear operation includes an operation branch for processing the input data of the linear operation. The operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer.

[0434] In one possible implementation, the linear operation in each second neural network model is different from that in the first convolutional layer, and different second neural network models include different linear operations.

[0435] In one possible implementation, the convolutional layer equivalent to the linear operation and the linear operation produce the same processing results when processing the same data.

[0436] In one possible implementation, the second neural network model corresponding to the target neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation. The target neural network model includes the trained target linear operation. The acquisition module is used for:

[0437] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0438] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0439] In one possible implementation, the device further includes:

[0440] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0441] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0442] The fusion module is used for:

[0443] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0444] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0445] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0446] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0447] In one implementation, the model training device acquires module 1701, which can be used to acquire a first neural network model, the first neural network model including a first convolutional layer;

[0448] Based on the first neural network model, a second neural network model is obtained, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation. The target linear operation includes multiple sub-linear operations, and the target linear operation is equivalent to a convolutional layer. The target linear operation includes M operation branches, and the input of each operation branch is the input of the target linear operation. The multiple sub-linear operations satisfy at least one of the following conditions:

[0449] The plurality of sub-linear operations include at least three types of operations;

[0450] The value of M is not 3;

[0451] At least one of the M operation branches includes sub-linear operations with a number not equal to 2, where M is a positive integer; or,

[0452] In at least one of the M operation branches, the number of sub-linear operations of type convolution is not 1.

[0453] The model training module 1702 can be used to train the second neural network model to obtain the target neural network model.

[0454] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0455] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0456] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0457] In one possible implementation, the acquisition module is used to replace the trained target linear operation in the target neural network model with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0458] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0459] In one possible implementation, the device further includes:

[0460] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0461] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0462] The fusion module is used to obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of a third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0463] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0464] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0465] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0466] This application embodiment also provides a model training apparatus, the apparatus comprising:

[0467] An acquisition module is used to acquire a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0468] A target linear operation for replacing the first convolutional layer is determined based on at least one of the following information, wherein the information includes the network structure of the first neural network model, the target task, and the position of the first convolutional layer in the first neural network model, and the target linear operation is equivalent to a convolutional layer;

[0469] Based on the first neural network model, a second neural network model is obtained, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation;

[0470] The model training module is used to train the second neural network model to obtain the target neural network model.

[0471] In one possible implementation, the target linear operation comprises multiple sub-linear operations, and the target linear operation comprises M operation branches, each operation branch having its input as the input of the target linear operation, wherein the M operation branches satisfy at least one of the following conditions:

[0472] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0473] At least two of the M operation branches have a different number of sub-linear operations; or,

[0474] At least two of the M operation branches contain sub-linear operations of different types.

[0475] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0476] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0477] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0478] In one possible implementation, the acquisition module is used to replace the trained target linear operation in the target neural network model with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0479] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0480] In one possible implementation, the device further includes:

[0481] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0482] In one possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation, the first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0483] The fusion module is used to obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of a third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0484] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0485] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0486] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0487] This application embodiment also provides a model training apparatus, the apparatus comprising:

[0488] An acquisition module is used to acquire a first neural network model, wherein the first neural network model includes a first convolutional layer;

[0489] Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a target linear operation. The target linear operation is equivalent to a convolutional layer, and the target linear operation includes multiple sub-linear operations. The target linear operation includes M operation branches, and the input of each operation branch is the input of the target linear operation. The M operation branches satisfy at least one of the following conditions:

[0490] The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations.

[0491] At least two of the M operation branches have a different number of sub-linear operations; or,

[0492] At least two of the M operation branches contain sub-linear operations of different types.

[0493] The model training module is used to train the second neural network model to obtain the target neural network model.

[0494] In one possible implementation, the receptive field of the convolutional layer equivalent to the target linear operation is smaller than or equal to the receptive field of the first convolutional layer.

[0495] In one possible implementation, the target linear operation is different from that of the first convolutional layer.

[0496] In one possible implementation, the convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

[0497] In one possible implementation, the target neural network model includes a trained target linear operation, and the acquisition module is configured to:

[0498] The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

[0499] In one possible implementation, the size of the second convolutional layer is the same as the size of the first convolutional layer.

[0500] In one possible implementation, the device further includes:

[0501] The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

[0502] In a possible implementation, the trained target linear operation includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation. The first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter.

[0503] The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes:

[0504] Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters.

[0505] The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

[0506] In one possible implementation, the linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

[0507] In one possible implementation, if the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

[0508] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 18 , Figure 18 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1800 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The execution device 1800 may be equipped with... Figure 10 The data processing apparatus described in the corresponding embodiment is used to implement Figure 10 The corresponding embodiment describes the data processing function. Specifically, the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803, and a memory 1804 (wherein the execution device 1800 may have one or more processors 1803). Figure 11 (Taking a processor as an example), processor 1803 may include application processor 18031 and communication processor 18032. In some embodiments of this application, receiver 1801, transmitter 1802, processor 1803 and memory 1804 may be connected via a bus or other means.

[0509] Memory 1804 may include read-only memory and random access memory, and provides instructions and data to processor 1803. A portion of memory 1804 may also include non-volatile random access memory (NVRAM). Memory 1804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0510] The processor 1803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0511] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1803. The processor 1803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by the integrated logic circuitry in the hardware of the processor 1803 or by instructions in software form. The processor 1803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or microcontroller, a vision processing unit (VPU), a tensor processing unit (TPU), or other processors suitable for AI computation. It may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1804, and processor 1803 reads information from memory 1804 and, in conjunction with its hardware, completes the steps of the above method.

[0512] Receiver 1801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1802 can be used to output digital or character information through the first interface; transmitter 1802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1802 may also include a display device such as a display screen.

[0513] The execution device can obtain through Figure 5 or Figure 16b The model trained by the model training method in the corresponding embodiment is then used for model inference.

[0514] This application also provides a training device; please refer to [link / reference]. Figure 19 , Figure 19 This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 1900 is implemented by one or more servers. The training device 1900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1919 (e.g., one or more processors) and memory 1932, and one or more storage media 1930 (e.g., one or more mass storage devices) for storing application programs 1942 or data 1944. The memory 1932 and storage media 1930 can be temporary or persistent storage. The program stored in the storage media 1930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1919 may be configured to communicate with the storage media 1930 and execute the series of instruction operations in the storage media 1930 on the training device 1900.

[0515] The training device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input / output interfaces 1958; or, one or more operating systems 1941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0516] Specifically, the training equipment can perform Figure 5 or Figure 16b The model training method in the corresponding embodiment.

[0517] Figure 17 The model training device 1700 described herein can be a module in a training device, and the processor in the training device can execute the model training method executed by the model training device 1700.

[0518] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0519] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0520] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0521] For details, please refer to Figure 20 , Figure 20 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 2000. The NPU 2000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0522] The NPU 2000 can achieve the model training method provided in the embodiment described in Figure 6, or perform inference on the trained model, through the cooperation of its various internal components.

[0523] The computing circuit 2003 in the NPU 2000 can perform the steps of acquiring a first neural network model and training the first neural network model.

[0524] More specifically, in some implementations, the arithmetic circuitry 2003 within the NPU 2000 includes multiple processing engines (PEs). In some implementations, the arithmetic circuitry 2003 is a two-dimensional pulsating array. The arithmetic circuitry 2003 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 2003 is a general-purpose matrix processor.

[0525] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2008.

[0526] Unified memory 2006 is used to store input and output data. Weight data is directly transferred to weight memory 2002 via Direct Memory Access Controller (DMAC) 2005. Input data is also transferred to unified memory 2006 via DMAC.

[0527] BIU stands for Bus Interface Unit 2010, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2009.

[0528] The Bus Interface Unit (BIU) 2010 is used by the instruction fetch memory 2009 to fetch instructions from external memory, and also by the memory access controller 2005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0529] DMAC is mainly used to move input data from external memory DDR to unified memory 2006, or weight data to weight memory 2002, or input data to input memory 2001.

[0530] The vector computation unit 2007 includes multiple processing units that further process the output of the computation circuit 2003 when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0531] In some implementations, the vector computation unit 2007 can store the processed output vector in the unified memory 2006. For example, the vector computation unit 2007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 2003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 2003, for example, for use in subsequent layers of the neural network.

[0532] The instruction fetch buffer 2009 connected to the controller 2004 is used to store the instructions used by the controller 2004;

[0533] Unified Memory 2006, Input Memory 2001, Weight Memory 2002, and Instruction Fetch Memory 2009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0534] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0535] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0536] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0537] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0538] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model training method, characterized in that, The method includes: A first neural network model is obtained, the first neural network model includes a first convolutional layer, the input of the first neural network model includes at least one of the following: image, video, the first neural network model is used to achieve a target task, the target task includes at least one of the following: image recognition, image classification, image super-resolution reconstruction; Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation. The linear operation refers to an operation composed of multiple basic linear operations, and there are connections between the basic linear operations. The linear operation is equivalent to a convolutional layer, and the receptive field of the convolutional layer equivalent to the linear operation is less than or equal to the receptive field of the first convolutional layer. The plurality of second neural network models are trained to obtain a target neural network model, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of trained second neural network models.

2. The method according to claim 1, characterized in that, The linear operation includes multiple operation branches, the input of each operation branch is the input of the linear operation, each operation branch includes at least one sub-linear operation in sequence, and the equivalent receptive field of the at least one sub-linear operation in sequence is less than or equal to the receptive field of the first convolutional layer. or, The linear operation includes an operation branch for processing the input data of the linear operation. The operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer.

3. The method according to any one of claims 1 to 2, characterized in that, The linear operations in each second neural network model are different from those in the first convolutional layer, and the linear operations included in different second neural network models are different.

4. The method according to any one of claims 1 to 2, characterized in that, The convolutional layer equivalent to the linear operation and the linear operation itself produce the same processing results when processing the same data.

5. The method according to any one of claims 1 to 2, characterized in that, The second neural network model corresponding to the target neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation. The target neural network model includes the trained target linear operation. The method further includes: The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

6. The method according to claim 5, characterized in that, The size of the second convolutional layer is the same as the size of the first convolutional layer.

7. The method according to claim 5, characterized in that, The method further includes: Based on the order in which the multiple sub-linear operations included in the trained target linear operation process data, each sub-linear operation is merged into an adjacent sub-linear operation that is later in the order, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

8. The method according to claim 7, characterized in that, The target linear operation after training includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation. The first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter. The merging of each sub-linear operation into an adjacent sub-linear operation that is later in the stated sequence includes: Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters. The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

9. The method according to any one of claims 1 to 2, characterized in that, The linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

10. The method according to claim 8, characterized in that, If the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

11. A model training method, characterized in that, The method includes: A first neural network model is obtained, the first neural network model includes a first convolutional layer, the input of the first neural network model includes at least one of the following: image, video, the first neural network model is used to achieve a target task, the target task includes at least one of the following: image recognition, image classification, image super-resolution reconstruction; A target linear operation for replacing the first convolutional layer is determined based on at least one of the following information, wherein the information includes the network structure of the first neural network model, the target task, and the position of the first convolutional layer in the first neural network model, the target linear operation is equivalent to a convolutional layer, the target linear operation refers to an operation composed of multiple basic linear operations, there are connections between the basic linear operations, and the receptive field of the convolutional layer equivalent to the target linear operation is less than or equal to the receptive field of the first convolutional layer. Based on the first neural network model, a second neural network model is obtained, wherein the second neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation; The second neural network model is trained to obtain the target neural network model.

12. The method according to claim 11, characterized in that, The target linear operation includes multiple sub-linear operations, and the target linear operation includes M operation branches. The input of each operation branch is the input of the target linear operation, and the M operation branches satisfy at least one of the following conditions: The input of at least one of the multiple sub-linear operations included in the M operation branches is the output of the multiple sub-linear operations. At least two of the M operation branches have a different number of sub-linear operations. or, The sub-linear operations between at least two of the M operation branches have different operation types.

13. The method according to any one of claims 11 to 12, characterized in that, The target linear operation is different from that of the first convolutional layer.

14. The method according to any one of claims 11 to 12, characterized in that, The convolutional layer equivalent to the target linear operation and the target linear operation produce the same processing results when processing the same data.

15. The method according to any one of claims 11 to 12, characterized in that, The target neural network model includes a trained target linear operation, and the method further includes: The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

16. A model training device, characterized in that, The device includes: An acquisition module is used to acquire a first neural network model, the first neural network model including a first convolutional layer, the input of the first neural network model including at least one of the following: image, video, the first neural network model is used to achieve a target task, the target task including at least one of the following: image recognition, image classification, image super-resolution reconstruction; Based on the first neural network model, multiple second neural network models are obtained, wherein each second neural network model is obtained by replacing the first convolutional layer in the first neural network model with a linear operation. The linear operation refers to an operation composed of multiple basic linear operations, and there are connections between the basic linear operations. The linear operation is equivalent to a convolutional layer, and the receptive field of the convolutional layer equivalent to the linear operation is less than or equal to the receptive field of the first convolutional layer. The model training module is used to train the plurality of second neural network models to obtain a target neural network model, wherein the target neural network model is the neural network model with the highest model accuracy among the plurality of trained second neural network models.

17. The apparatus according to claim 16, characterized in that, The linear operation includes multiple operation branches, the input of each operation branch is the input of the linear operation, each operation branch includes at least one sub-linear operation in sequence, and the equivalent receptive field of the at least one sub-linear operation in sequence is less than or equal to the receptive field of the first convolutional layer. or, The linear operation includes an operation branch for processing the input data of the linear operation. The operation branch includes at least one sequential sub-linear operation, and the equivalent receptive field of the at least one sequential sub-linear operation is less than or equal to the receptive field of the first convolutional layer.

18. The apparatus according to any one of claims 16 to 17, characterized in that, The linear operations in each second neural network model are different from those in the first convolutional layer, and the linear operations included in different second neural network models are different.

19. The apparatus according to any one of claims 16 to 17, characterized in that, The convolutional layer equivalent to the linear operation and the linear operation itself produce the same processing results when processing the same data.

20. The apparatus according to any one of claims 16 to 17, characterized in that, The second neural network model corresponding to the target neural network model is obtained by replacing the first convolutional layer in the first neural network model with the target linear operation. The target neural network model includes the trained target linear operation. The acquisition module is used for: The trained target linear operation in the target neural network model is replaced with a second convolutional layer equivalent to the trained target linear operation to obtain a third neural network model.

21. The apparatus according to claim 20, characterized in that, The size of the second convolutional layer is the same as the size of the first convolutional layer.

22. The apparatus according to claim 21, characterized in that, The device further includes: The fusion module is used to fuse each sub-linear operation into an adjacent sub-linear operation that is located later in the order of processing data, based on the order in which the multiple sub-linear operations included in the trained target linear operation are processed, until the fusion to the last sub-linear operation in the order is completed, so as to obtain a second convolutional layer equivalent to the target linear operation.

23. The apparatus according to claim 22, characterized in that, The target linear operation after training includes an adjacent first sub-linear operation and a second sub-linear operation, and in the order, the second sub-linear operation is located after the first sub-linear operation. The first sub-linear operation includes a first operation parameter, and the second sub-linear operation includes a second operation parameter. The fusion module is used for: Obtain the fusion parameters of the first sub-linear operation, wherein if the input data of the first sub-linear operation is the input data of the trained target linear operation, then the fusion parameters of the first sub-linear operation are the first operation parameters; if the input data of the first sub-linear operation is the output data of the third sub-linear operation adjacent to the first sub-linear operation and preceding the order, then the fusion parameters of the first sub-linear operation are obtained based on the fusion parameters of the third sub-linear operation and the first operation parameters. The fusion parameters of the second sub-linear operation are obtained based on the fusion parameters of the first sub-linear operation, the second operation parameters, and the operation type of the second sub-linear operation; wherein, if the second sub-linear operation is the last sub-linear operation in the sequence, the fusion parameters of the second sub-linear operation are used as the operation parameters of the second convolutional layer.

24. The apparatus according to any one of claims 16 to 17, characterized in that, The linear operation includes multiple sub-linear operations, and the operation types of the multiple sub-linear operations include at least one of the following: addition operation, empty operation, identity operation, convolution operation, batch normalization (BN) operation, or pooling operation.

25. The apparatus according to claim 23, characterized in that, If the operation type of the second sub-linear operation is a convolution operation or a BN operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the inner product of the fusion parameter of the first sub-linear operation and the operation parameter of the second sub-linear operation; if the operation type of the second sub-linear operation is a summation operation, a pooling operation, an identity operation, or a null operation, then the fusion parameter of the second sub-linear operation is obtained by calculating the fusion parameter of the first sub-linear operation according to the operation type of the second sub-linear operation.

26. A model training device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 10, 11 to 15.

27. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method according to any one of claims 1 to 10, 11 to 15.

28. A computer product, comprising code, characterized in that, When the code is executed, it is used to implement the method as described in any one of claims 1 to 10, 11 to 15.

Images