A neural network structure determination method and apparatus thereof

Abstract

The embodiment of the application discloses a neural network structure determination method, comprising: obtaining an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight to be trained; performing model training on the initial neural network to obtain M updated target weights; and updating the connection relationship between the second block and the M first blocks in the initial neural network according to the M updated target weights, to obtain a first neural network. In the search process of the connection relationship between the blocks of the initial neural network, the target weight to be trained is added to the connection between the blocks, the size of the updated target weight is used as the importance judgment basis of the connection relationship between the blocks, and the selection and elimination of the connection relationship between the blocks are based on the size of the updated target weight, so that the search of the topology of the neural network is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a neural network structure determination method and device. BACKGROUND

[0002] Machine learning (ML) has achieved considerable success in recent years, and more and more machine learning derived products are changing people's lives. However, the current progress of machine learning relies on ML experts to manually design and debug models, which not only limits the wide application of machine learning, but also prolongs the product iteration cycle.

[0003] With the rapid development of artificial intelligence technology, a good performance neural network often has a delicate network structure, which needs a human expert with superb skills and rich experience to spend a lot of effort to build. There are many combinations of neural network structure, and when manually designed, it is generally through tedious multiple attempts to find some rules of high-precision network structure, and then design a good structure. The classic AlexNet, ResNet, DenseNet network structure is designed by artificial, and their appearance greatly improves the accuracy of various tasks.

[0004] With the progress of various technologies and the increase of computing resources, automated machine learning (AutoML) technology gradually begins to replace human beings to design new network structures. By encoding the network structure, then evaluating the performance of a large number of codes, and then learning through reinforcement learning, genetic algorithm, etc., the optimal code is finally generated. However, in the prior art, the width (the number of input feature channels and the number of output feature channels of the operation unit in the neural network) and the depth (the number of operation units included in the neural network) of the neural network can only be searched, and the topology of the neural network cannot be searched. SUMMARY

[0005] In a first aspect, the present application provides a neural network structure determination method, comprising:

[0006] An initial neural network to be trained is obtained, the initial neural network comprising M first blocks and a second block, the block can also be referred to as a network block, and the block can comprise a certain number of atomic operations, which can include but are not limited to convolution, pooling, residual connection, etc. The second block is connected to each first block. The connection relationship between the blocks can be understood as the data transmission direction between the blocks. Specifically, the block can perform the corresponding operation on the input data, and obtain the operation result. The operation result can be input to the next block connected to the block and used as the input data of the next block. That is, the connection relationship between two first blocks can represent that the output of one block is used as the input of another block. Each first block corresponds to a target weight. In the embodiment of the application, in order to determine which connection between the blocks is retained during the training of the model, a trainable parameter (also referred to as a target weight in the embodiment) can be set on the connection between the two blocks. The output of one block can be multiplied by the corresponding target weight (also referred to as a product operation in the embodiment), and then the result of the product operation is input to another block. The second block is used to perform the corresponding operation of the second block according to the M first outputs. The M first outputs are obtained by multiplying the output of each first block with the corresponding target weight. The target weight is a trainable weight. M is an integer greater than 1. The initial neural network is subjected to model training to obtain updated M target weights. In the embodiment of the application, the training device can perform model training on the initial neural network on a target task and update the M target weights. When the M target weights are stable, the updated M target weights can be obtained. The stability of the target weight can be understood as that the change of the target weight in the iteration training process is within a certain range. In some implementations, the number of iterations can be used to determine whether the M target weights are stable. For example, the training device can perform model training on the initial neural network for a first preset number of iterations to obtain updated M target weights. The first preset number of iterations can be a pre-set value, which can be determined according to the total number of iterations required. For example, when the number of iterations reaches a certain percentage of the total number of training required, it is considered that the M target weights are stable.

[0007] According to the updated M target weights, a connection relationship between the second block and the M first blocks in the initial neural network is updated to obtain a first neural network; wherein the second block in the first neural network is configured to perform corresponding operation of the second block according to the output of the first block corresponding to the maximum N target weights in the updated M target weights, and the N is less than the M. Specifically, the second block in the first neural network can be configured to perform corresponding operation of the second block according to the sum result of the output of the first block corresponding to the maximum N target weights in the updated M target weights. Wherein the size of the updated M target weights can indicate whether the connection between the blocks is important, wherein the greater the size of the updated target weight, the more important the connection between the blocks. Specifically, the connection where the maximum N target weights in the updated M target weights are located can be retained, and the connection where the target weights other than the maximum N target weights in the updated M target weights are located can be removed.

[0008] In the process of searching the connection relationship between the blocks of the initial neural network, the embodiment adds trainable target weights on the connection between the blocks, uses the size of the updated target weight as the basis for judging the importance of the connection relationship between the blocks, and selects and removes the connection relationship between the blocks based on the size of the updated target weight, thereby realizing the search of the topology of the neural network.

[0009] In a possible implementation, the M first blocks and the second block in the initial neural network are sequentially formed into a serial connection, and the second block is the terminal point of the serial connection, the M first blocks in the initial neural network include a target block, the target block is connected with the second block on the serial connection, and in the case that the updated target weight corresponding to the target block does not belong to the maximum N target weights in the updated M target weights, the second block in the first neural network is further configured to perform corresponding operation of the second block according to the output of the target block.

[0010] That is, whether the updated target weight corresponding to the target block is one of the N largest target weights in the M updated target weights, the connection between the target block and the second block is always retained, which can be referred to as a backbone connection, and the backbone connection will not be pruned, which can ensure that the backbone architecture of the entire neural network is not damaged. Specifically, in one implementation, if the updated target weight belonging to the backbone connection is one of the N largest target weights in the M target weights, the connection of the N updated target weights can be retained, and if the updated target weight belonging to the backbone connection is not one of the N largest target weights in the M target weights, the connection of the N+1 updated target weights can be retained.

[0011] In one possible implementation, the N is 1.

[0012] In one possible implementation, the model training of the initial neural network is performed to obtain the M updated target weights, including:

[0013] The model training of the initial neural network is performed for a first preset number of iterations to obtain the M updated target weights.

[0014] In one possible implementation, the method further includes:

[0015] The model training of the first neural network is performed until the data processing accuracy of the first neural network meets a preset condition or the number of iterations of the model training reaches a second preset number of iterations, to obtain a second neural network.

[0016] In the embodiments of the present application, the ratio between the first preset number of iterations and the second preset number of iterations can be preset, which is equivalent to that the obtaining of the M updated target weights is performed at a fixed percentage position of the overall training rounds, which ensures the stability of the target weights and the sufficient training of the network after the topology optimization. At the same time, the time of single topology optimization is maintained, which is basically the same as the original training time, ensuring the search efficiency.

[0017] In one possible implementation, the number of channels of the input and output of each first block of the M first blocks is consistent with the number of channels of the input and output of the second block.

[0018] In one possible implementation, the second block in the initial neural network is configured to perform the operation corresponding to the second block according to the sum result of the M first outputs.

[0019] The second block in the first neural network is configured to perform corresponding operation according to a sum of outputs of the first blocks corresponding to the maximum N target weights in the updated M target weights.

[0020] In a possible implementation, the method further includes:

[0021] The data to be trained includes at least one of image data, text data, and voice data.

[0022] The initial neural network is trained according to the data to be trained.

[0023] In a second aspect, the present application provides a neural network structure determination method, which includes:

[0024] An initial neural network to be trained is obtained, the initial neural network including M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, and each first block being configured to perform corresponding operation according to corresponding first output, wherein the first output corresponding to each first block is obtained by multiplying the target weight corresponding to the first block and the output of the second block, the target weight being a trainable weight, and M being an integer greater than 1; and different from the embodiments described in the first aspect, the output of the M first blocks is used as the input of the second block in the first aspect, and the selection of the connection relationship is also based on the size of the M updated target weights corresponding to the M first blocks in the subsequent selection process, and in this embodiment, the output of the second block is used as the input of the M first blocks.

[0025] The initial neural network is trained to obtain updated M target weights.

[0026] According to the updated M target weights, the connection relationship between the second block and the M first blocks in the initial neural network is updated to obtain a first neural network, wherein the first blocks corresponding to the maximum N target weights in the updated M target weights are configured to perform corresponding operation according to the output of the second block, and N is less than M.

[0027] In a possible implementation, the second block in the initial neural network is sequentially connected with the M first blocks in series, and the second block is the starting point of the series connection, the M first blocks in the initial neural network include a target block, the target block is connected with the second block on the series connection, and in a case where the updated target weight corresponding to the target block does not belong to the N target weights with the largest values in the updated M target weights, the target block in the first neural network is further configured to perform the operation corresponding to the target block according to the output of the second block.

[0028] In the searching process of the connection relationship between the blocks of the initial neural network, the target weight between the blocks is added, the importance of the connection relationship between the blocks is determined according to the size of the updated target weight, and the connection relationship between the blocks is selected and removed based on the size of the updated target weight, so that the searching of the topology of the neural network is realized.

[0029] In a possible implementation, the N is 1.

[0030] In a possible implementation, the model training of the initial neural network is performed to obtain the updated M target weights, including:

[0031] The model training of the initial neural network is performed for a first preset number of iterations to obtain the updated M target weights.

[0032] In a possible implementation, the method further includes:

[0033] The model training of the first neural network is performed until the data processing accuracy of the first neural network meets a preset condition or the number of iterations of the model training reaches a second preset number of iterations, to obtain a second neural network.

[0034] In a possible implementation, the number of channels of the input and the output of each first block of the M first blocks is consistent with the number of channels of the input and the output of the second block.

[0035] In a possible implementation, the method further includes:

[0036] The data to be trained is obtained, and the data to be trained includes at least one of the following: image data, text data, and voice data; and correspondingly, the model training of the initial neural network includes:

[0037] According to the data to be trained, the initial neural network is subjected to model training.

[0038] In a third aspect, the present application provides a neural network structure determination method, which comprises:

[0039] A plurality of target encodings are obtained, each target encoding being used to indicate a candidate neural network, the plurality of target encodings comprising a first target encoding and a plurality of first encodings, the first target encoding being used to indicate a first neural network; in the embodiments of the present application, when the structure of a neural network is searched, the structural features of the neural network can be written in the form of an encoding, each encoding being used to indicate at least one of the following structural features of a candidate neural network: the type of an operation unit included in the candidate neural network, the number of operation units included in the candidate neural network, and the input feature and output feature channel numbers of the operation units included in the candidate neural network. Wherein, the operation unit can refer to each atomic operation in a block, and in another expression, each encoding is used to indicate the type of an atomic operation included in a candidate neural network, the number of atomic operations included in the candidate neural network, and the input feature and output feature channel numbers of the atomic operations included in the candidate neural network. Since the input feature and output feature channel numbers of the atomic operations in each block are the same in the same stage, it is equivalent to that each encoding is used to indicate the input feature and output feature channel numbers of a block included in a candidate neural network.

[0040] The first neural network is subjected to model training to obtain the data processing precision of the first neural network; in the embodiments of the present application, the neural network data processing precision can be the value of a loss function of a training network, the test precision of a neural network, and the like, which are not limited in the embodiments of the present application.

[0041] In the embodiments of the present application, not all of the plurality of candidate neural networks indicated by the plurality of target encodings are subjected to model training, and based on the data processing precisions of the plurality of candidate neural networks, a candidate neural network with higher precision is selected as the search result of the model, but only the first neural network indicated by one target encoding (the first target encoding) is selected, and the first neural network is subjected to model training, and then based on the difference degree between the target encodings, the data processing precisions of the candidate neural networks indicated by the remaining target encodings (the plurality of first encodings) in the plurality of target encodings are determined.

[0042] According to the difference between the first target code and the plurality of first codes and the data processing accuracy of the first neural network, the data processing accuracy of the candidate neural network indicated by each first code is determined; in the embodiment of the application, the target code can include a plurality of bits, each bit indicating a structural feature of the candidate neural network, in order to eliminate the dimensional difference between the bits and the subsequent influence, each target code can be standardized. For example, the mean and standard deviation of the plurality of target codes can be calculated for each bit of the target code, and then each bit of the target code is subtracted from the mean and divided by the standard deviation. Thereafter, the dimension of the target code will no longer affect the subsequent algorithm. In this embodiment, the Gaussian process can be used to determine the data processing accuracy of the candidate neural network indicated by each first code according to the difference between the first target code and the plurality of first codes and the data processing accuracy of the first neural network, and specifically, the values of other sample points can be estimated according to the distance between two sample points and the values of a part of sample points. In this embodiment, the sample points are the target codes, and the values of the sample points are the data processing accuracy of the candidate neural network indicated by the target codes.

[0043] The first candidate neural network with the highest data processing accuracy among the candidate neural networks indicated by the plurality of target codes is obtained.

[0044] The first candidate neural network is subjected to model training to obtain a first target neural network.

[0045] In a possible implementation, the method further includes:

[0046] The data processing accuracy of the first target neural network is obtained, the plurality of target codes include a second target code, and the second target code is used to indicate the first target neural network.

[0047] According to the difference between the second target code and the codes other than the second target code among the plurality of target codes and the data processing accuracy of the first target neural network, the data processing accuracy of the candidate neural network indicated by each target code other than the second target code among the plurality of target codes is determined.

[0048] According to the data processing accuracy of the candidate neural networks indicated by the plurality of target codes, a second candidate neural network with the highest data processing accuracy is determined, and the second candidate neural network is subjected to model training to obtain a second target neural network. Thereafter, the training device can repeat the above process, and through a preset number of iterations (for example, 4 rounds), a very ideal model can be obtained as the result of neural model structure search.

[0049] In a possible implementation, each target encoding indicates at least one of the following structural features of a candidate neural network:

[0050] The type of operation unit included in the candidate neural network, the number of operation units included in the candidate neural network, and the number of input feature and output feature channels of the operation unit included in the candidate neural network.

[0051] In a possible implementation, the method further includes:

[0052] Clustering the plurality of encodings to obtain a plurality of encoding sets, each encoding set corresponding to a clustering category, the plurality of encoding sets including a target encoding set, and the target encoding set including the plurality of target encodings.

[0053] In a possible implementation, the first target encoding is a clustering center of the target encoding set.

[0054] In the embodiments of the present application, the plurality of encodings can be obtained by screening a plurality of candidate encodings. The first target encoding can be one of the target encoding set, and in one implementation, the first target encoding can be a clustering center of the target encoding set. The first target encoding is used to indicate a first neural network. It should be understood that the clustering can be a K-Means algorithm, a DBSCAN algorithm, a BIRCH algorithm, a MeanShift algorithm, or the like.

[0055] In a possible implementation, each target encoding indicates a candidate neural network satisfying at least one of the following conditions:

[0056] The amount of computation required when running the candidate neural network indicated by each target encoding is less than a first preset value;

[0057] The number of weights included in the candidate neural network indicated by each target encoding is less than a second preset value; and

[0058] The running speed when running the candidate neural network indicated by each target encoding is higher than a third preset value.

[0059] In an implementation, the training device can generate a plurality of candidate encodings, and filter the plurality of candidate encodings based on preset rules. The preset rules can be at least one of the following: selecting, from the plurality of candidate encodings, a candidate neural network whose required computation amount when running the indicated candidate neural network is less than a first preset value, a candidate neural network whose included weight amount is less than a second preset value, and a candidate neural network whose running speed when running the indicated candidate neural network is higher than a third preset value. The computation amount can be the number of floating-point multiplications required in the entire neural network, and the floating-point multiplication operation is the most time-consuming, and thus can be used to represent the computation amount of the neural network. The first preset value, the second preset value, and the third preset value can be preset.

[0060] In a possible implementation, the first candidate neural network includes M first blocks and a second block, the second block is connected with each first block, and each first block corresponds to a target weight, the first candidate neural network is configured to perform a product operation on an output of each first block and the corresponding target weight to obtain M first outputs, and the second block is configured to perform a corresponding operation of the second block according to the M first outputs; the target weight is a trainable weight, and the M is an integer greater than 1.

[0061] The model training on the first candidate neural network to obtain the first target neural network includes:

[0062] The model training on the first candidate neural network to obtain the first target neural network includes:

[0063] According to the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; the second block in the second neural network is configured to perform a corresponding operation of the second block according to the output of the first block corresponding to the maximum N target weights in the updated M target weights, and the N is less than the M.

[0064] The model training on the first candidate neural network to obtain the first target neural network includes:

[0065] In a possible implementation, the first candidate neural network comprises M first blocks and a second block, the second block is connected with each first block, and each first block corresponds to a target weight, the first candidate neural network is configured to multiply an output of the second block with each target weight to obtain M first outputs, and each first block is configured to perform an operation corresponding to the first block according to a corresponding first output; wherein the target weight is a trainable weight, and M is an integer greater than 1.

[0066] The model training on the first candidate neural network comprises:

[0067] The model training on the first candidate neural network comprises:

[0068] According to the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; wherein the first blocks corresponding to the largest N target weights in the updated M target weights are configured to perform an operation corresponding to the first block according to the output of the second block, and N is less than M.

[0069] The model training on the first candidate neural network comprises:

[0070] In a fourth aspect, the present application provides a neural network structure determination apparatus, which comprises:

[0071] The obtaining module is configured to obtain an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, the second block being configured to perform an operation corresponding to the second block according to M first outputs; wherein the M first outputs are obtained by multiplying the output of each first block with a corresponding target weight, the target weight being a trainable weight, and M being an integer greater than 1.

[0072] The model training module is configured to perform model training on the initial neural network to obtain updated M target weights.

[0073] a model updating module configured to update a connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights, to obtain a first neural network; wherein the second block in the first neural network is configured to perform corresponding operation of the second block according to an output of the first block corresponding to the maximum N target weights in the updated M target weights, and the N is less than the M.

[0074] In a possible implementation, the M first blocks and the second block in the initial neural network are sequentially formed into a serial connection, and the second block is a terminal point of the serial connection, the M first blocks in the initial neural network include a target block, the target block is connected with the second block on the serial connection, and in a case where the updated target weight corresponding to the target block does not belong to the maximum N target weights in the updated M target weights, the second block in the first neural network is further configured to perform corresponding operation of the second block according to an output of the target block.

[0075] In a possible implementation, the N is 1.

[0076] In a possible implementation, the model training module is configured to perform model training on the initial neural network for a first preset number of iterations, to obtain the updated M target weights.

[0077] In a possible implementation, the model training module is configured to perform model training on the first neural network until a data processing accuracy of the first neural network meets a preset condition or a number of iterations of model training reaches a second preset number of iterations, to obtain a second neural network.

[0078] In a possible implementation, a channel number of an input and an output of each first block of the M first blocks is consistent with a channel number of an input and an output of the second block.

[0079] In a possible implementation, the second block in the initial neural network is configured to perform corresponding operation of the second block according to a sum result of the M first outputs.

[0080] The second block in the first neural network is configured to perform corresponding operation of the second block according to a sum result of the outputs of the first blocks corresponding to the maximum N target weights in the updated M target weights.

[0081] In a possible implementation, the obtaining module is configured to obtain to-be-trained data, the to-be-trained data including at least one of image data, text data, and voice data; and the model training of the initial neural network includes:

[0082] The model training module is configured to perform model training on the initial neural network according to the to-be-trained data.

[0083] In a fifth aspect, the present application provides a neural network structure determination apparatus, the apparatus comprising:

[0084] An obtaining module is configured to obtain an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, and each first block being configured to perform corresponding operation of the first block according to a corresponding first output; wherein the corresponding first output of each first block is obtained by performing product operation on the target weight corresponding to the first block and an output of the second block, the target weight being a trainable weight, and M being an integer greater than 1.

[0085] A model training module is configured to perform model training on the initial neural network to obtain updated M target weights.

[0086] A model updating module is configured to update a connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights to obtain a first neural network; wherein first blocks corresponding to the largest N target weights in the updated M target weights are configured to perform corresponding operation of the first blocks according to the output of the second block, and N is less than M.

[0087] In a possible implementation, the second block and the M first blocks in the initial neural network form a serial connection in sequence, and the second block is a starting point of the serial connection, and the M first blocks in the initial neural network include a target block, the target block being connected with the second block on the serial connection, and the target block in the first neural network being further configured to perform corresponding operation of the target block according to the output of the second block in a case where the updated target weight corresponding to the target block does not belong to the largest N target weights in the updated M target weights.

[0088] In a possible implementation, N is 1.

[0089] In a possible implementation, the model training module is configured to perform model training on the initial neural network for a first preset number of iterations to obtain M target weights.

[0090] In a possible implementation, the model training module is configured to perform model training on the first neural network until data processing accuracy of the first neural network meets a preset condition or a number of iterations of model training reaches a second preset number of iterations, to obtain a second neural network.

[0091] In a possible implementation, a number of channels of input and output of each first block of the M first blocks is consistent with a number of channels of input and output of the second block.

[0092] In a possible implementation, the obtaining module is configured to obtain training data, the training data including at least one of image data, text data, and voice data; and correspondingly, the model training on the initial neural network includes:

[0093] The model training module is configured to perform model training on the initial neural network according to the training data.

[0094] In a sixth aspect, the present application provides a neural network structure determination apparatus, the apparatus comprising:

[0095] An obtaining module is configured to obtain a plurality of target encodings, each target encoding being used to indicate a candidate neural network, the plurality of target encodings including a first target encoding and a plurality of first encodings, the first target encoding being used to indicate a first neural network.

[0096] A model training module is configured to perform model training on the first neural network to obtain data processing accuracy of the first neural network.

[0097] An accuracy determination module is configured to determine data processing accuracy of a candidate neural network indicated by each first encoding according to a difference between the first target encoding and the plurality of first encodings and the data processing accuracy of the first neural network.

[0098] The obtaining module is configured to obtain a first candidate neural network with the highest data processing accuracy from the candidate neural networks indicated by the plurality of target encodings.

[0099] The model training module is configured to perform model training on the first candidate neural network to obtain a first target neural network.

[0100] In a possible implementation, the obtaining module is configured to obtain the data processing accuracy of the first target neural network, the plurality of target encodings include a second target encoding, and the second target encoding is used to indicate the first target neural network.

[0101] According to a difference between the second target encoding and an encoding other than the second target encoding in the plurality of target encodings and the data processing accuracy of the first target neural network, a data processing accuracy of a candidate neural network indicated by each target encoding other than the second target encoding in the plurality of target encodings is determined.

[0102] According to the data processing accuracy of the candidate neural networks indicated by the plurality of target encodings, a second candidate neural network with the highest data processing accuracy is determined, and model training is performed on the second candidate neural network to obtain a second target neural network.

[0103] In a possible implementation, each target encoding is used to indicate at least one of the following structural features of a candidate neural network:

[0104] a type of an operation unit included in the candidate neural network, a number of operation units included in the candidate neural network, and a number of input feature and output feature channels of the operation units included in the candidate neural network.

[0105] In a possible implementation, the apparatus further includes:

[0106] a clustering module configured to cluster a plurality of encodings to obtain a plurality of encoding sets, each encoding set corresponding to a clustering category, the plurality of encoding sets including a target encoding set, and the target encoding set including the plurality of target encodings.

[0107] In a possible implementation, the first target encoding is a clustering center of the target encoding set.

[0108] In a possible implementation, the candidate neural network indicated by each target encoding satisfies at least one of the following conditions:

[0109] a computation amount required when running the candidate neural network indicated by each target encoding is less than a first preset value;

[0110] a weight amount included in the candidate neural network indicated by each target encoding is less than a second preset value; and

[0111] a running speed when running the candidate neural network indicated by each target encoding is higher than a third preset value.

[0112] In a possible implementation, the first candidate neural network comprises M first blocks and a second block, the second block is connected with each first block, and each first block corresponds to a target weight, and the second block is configured to perform an operation corresponding to the second block according to M first outputs; wherein the M first outputs are obtained by performing a product operation on an output of each first block and a corresponding target weight, the target weight is a trainable weight, and M is an integer greater than 1.

[0113] The model training module is configured to perform model training on the first candidate neural network to obtain updated M target weights.

[0114] According to the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; wherein a second block in the second neural network is configured to perform an operation corresponding to the second block according to an output of a first block corresponding to the largest N target weights in the updated M target weights, and N is less than M.

[0115] The second neural network is subjected to model training to obtain the first target neural network.

[0116] In a possible implementation, the first candidate neural network comprises M first blocks and a second block, the second block is connected with each first block, and each first block corresponds to a target weight, and each first block is configured to perform an operation corresponding to the first block according to a corresponding first output; wherein the first output corresponding to each first block is obtained by performing a product operation on the output of the second block and the target weight corresponding to the first block, the target weight is a trainable weight, and M is an integer greater than 1.

[0117] The model training module is configured to perform model training on the first candidate neural network to obtain updated M target weights.

[0118] According to the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; wherein a first block corresponding to the largest N target weights in the updated M target weights is configured to perform an operation corresponding to the first block according to an output of the second block, and N is less than M.

[0119] perform model training on the second neural network to obtain the first target neural network.

[0120] In a seventh aspect, an embodiment of the present application provides a neural network structure determination apparatus, which can include a memory, a processor, and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method of the first aspect and any optional method of the first aspect.

[0121] In an eighth aspect, an embodiment of the present application provides a neural network training apparatus, which can include a memory, a processor, and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method of the second aspect and any optional method of the first aspect.

[0122] In a ninth aspect, an embodiment of the present application provides a neural network training apparatus, which can include a memory, a processor, and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method of the third aspect and any optional method of the first aspect.

[0123] In a tenth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method of the first aspect and any optional method thereof.

[0124] In an eleventh aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method of the second aspect and any optional method thereof.

[0125] In a twelfth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method of the third aspect and any optional method thereof.

[0126] In a thirteenth aspect, an embodiment of the present application provides a computer program, which, when run on a computer, causes the computer to perform the method of the first aspect and any optional method thereof.

[0127] In a fourteenth aspect, an embodiment of the present application provides a computer program, which, when run on a computer, causes the computer to perform the method of the second aspect and any optional method thereof.

[0128] In a fifteenth aspect, an embodiment of the present application provides a computer program, which, when run on a computer, causes the computer to perform the method of the third aspect and any optional method thereof.

[0129] In a sixteenth aspect, an embodiment of the present application provides a computer program product, including code, when executed, for performing the method of the first aspect and any optional method thereof.

[0130] In a seventeenth aspect, an embodiment of the present application provides a computer program product, including code, when executed, for performing the method of the second aspect and any optional method thereof.

[0131] In an eighteenth aspect, an embodiment of the present application provides a computer program product, including code, when executed, for performing the method of the third aspect and any optional method thereof.

[0132] In a nineteenth aspect, the present application provides a chip system, including a processor, for supporting the implementation of the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or, information. In a possible design, the chip system further includes a memory, the memory being used to save the necessary program instructions and data for the execution device or the training device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0133] An embodiment of the present application provides a neural network structure determination method, the method comprising: obtaining an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, the second block being used to perform a corresponding operation of the second block according to M first outputs, wherein the M first outputs are obtained by multiplying the output of each first block with the corresponding target weight respectively, the target weight being a trainable weight, and the M being an integer greater than 1; performing model training on the initial neural network to obtain updated M target weights; updating the connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights to obtain a first neural network; wherein the second block in the first neural network is used to perform a corresponding operation of the second block according to the outputs of N first blocks corresponding to the maximum N target weights in the updated M target weights, and the N is less than the M. In the above manner, in the searching process of the connection relationship between blocks of the initial neural network, the trainable target weight is added to the connection between blocks, the importance of the connection relationship between blocks is judged according to the size of the updated target weight, and the selection and elimination of the connection relationship between blocks are based on the size of the updated target weight, so that the search of the topology of the neural network is realized. BRIEF DESCRIPTION OF DRAWINGS

[0134] Figure 1 A structural schematic diagram of an artificial intelligence subject framework;

[0135] Figure 2 An application scenario of an embodiment of the present application;

[0136] Figure 3 A schematic diagram of a system architecture provided by an embodiment of the present application;

[0137] Figure 4 A flowchart of a neural network structure determination method provided by an embodiment of the present application;

[0138] Figure 5a A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0139] Figure 5b A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0140] Figure 5c A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0141] Figure 6 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0142] Figure 7 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0143] Figure 8 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0144] Figure 9 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0145] Figure 10 A flowchart of a neural network structure determination method provided by an embodiment of the present application;

[0146] Figure 11 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0147] Figure 12 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0148] Figure 13 A schematic of a neural network structure determination method provided by an embodiment of the present application;

[0149] Figure 14 A flowchart of a neural network structure determination method provided for an embodiment of the present application is shown in the figure;

[0150] Figure 15 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure;

[0151] Figure 16 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure;

[0152] Figure 17 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure;

[0153] Figure 18 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure;

[0154] Figure 19 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure;

[0155] Figure 20 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure;

[0156] Figure 21 A schematic diagram of a neural network structure determination device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0157] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0158] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0159] The terms "first", "second", etc. in the specification and claims of the present application and the above-described figures are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0160] First, the overall workflow of the artificial intelligence system is described, please refer toFigure 1 , Figure 1 A structural diagram of an artificial intelligence subject framework is shown, and the artificial intelligence subject framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0161] (1) Infrastructure

[0162] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the external world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0163] (2) Data

[0164] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0165] (3) Data processing

[0166] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0167] Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc.

[0168] Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0169] Decision-making refers to the process of decision-making after intelligent information reasoning, which usually provides functions such as classification, sorting, and prediction.

[0170] (4) General ability

[0171] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0172] (5) Smart products and industry applications

[0173] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, and safe cities.

[0174] The embodiments of this application can be applied to scenarios such as image classification, object detection, semantic segmentation, room layout, image completion, or automatic encoding.

[0175] The following section provides a brief introduction to the application scenarios of this application, using ADAS / ADS visual perception systems and mobile phone beautification as examples.

[0176] Application Scenario 1: ADAS / ADS Visual Perception System

[0177] like Figure 2As shown, in the ADAS and ADS, real-time multi-type 2D target detection is required, including: dynamic obstacles (Pedestrian, Cyclist, Tricycle, Car, Truck, Bus), static obstacles (TrafficCone, TrafficStick, FireHydrant, Motocycle, Bicycle), traffic signs (TrafficSign, GuideSign, Billboard, TrafficLight_Red / TrafficLight_Yellow / TrafficLight_Green / TrafficLight_Black, RoadSign). In addition, in order to accurately obtain the area occupied by the dynamic obstacle in the 3D space, 3D estimation of the dynamic obstacle is also required, and a 3D box is output. In order to fuse the data of the laser radar, the mask of the dynamic obstacle needs to be obtained, so as to filter out the laser point cloud hitting the dynamic obstacle; in order to accurately detect the parking space, the four key points of the parking space need to be detected; in order to perform composition positioning, the key points of the static target need to be detected. Using the technical solutions provided in the embodiments of the present application, all or part of the above functions can be completed in the neural network.

[0178] Application scenario 2: mobile phone beautification function

[0179] In a mobile phone, the mask and key points of a human body can be detected by the neural network provided in the embodiments of the present application, and the corresponding parts of the human body can be enlarged or reduced, such as waist-in and hip-up operations, so as to output a beautified image.

[0180] Application scenario 3: image classification scenario

[0181] After obtaining the image to be classified, the category of the object in the image to be classified can be obtained based on the neural network, and then the image to be classified can be classified according to the category of the object in the image to be classified. For photographers, they will take many photos every day, some of which contain animals, some contain people, and some contain plants. The method of the present application can quickly classify the photos according to the content in the photos, and the photos can be classified into photos containing animals, photos containing people, and photos containing plants.

[0182] For a large number of images, the manual classification method is inefficient, and people are prone to fatigue when dealing with the same thing for a long time, and the classification result will have a large error; and the method of the application can quickly classify the images and will not have errors.

[0183] Application scenario 4: commodity classification

[0184] After obtaining the image including the commodity, the category of the commodity in the image of the commodity can be obtained through neural network processing, and then the commodity can be classified according to the category of the commodity. For a large number of commodities in a large shopping mall or supermarket, the object recognition method of the application can quickly complete the classification of the commodities, reducing the time cost and labor cost.

[0185] The embodiment of the application can search the structure of the neural network, and train the searched neural network, so that the trained neural network can perform the task processing of the above several scenarios.

[0186] Since the embodiment of the application involves a large number of applications of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiment of the application will be introduced first.

[0187] (1) Object detection: using image processing, machine learning, computer graphics and other related methods, object detection can determine the category of the image object and determine the detection frame for positioning the object.

[0188] (2) Convolutional Neural Network (CNN): a deep neural network with convolutional structure. The convolutional neural network includes a feature extractor composed of a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter. The perception network in the embodiment can include a convolutional neural network, which is used for convolution processing of an image or convolution processing of a feature map to generate a feature map.

[0189] (3) Back propagation algorithm

[0190] The convolutional neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output will produce an error loss, and the error loss information is propagated backward to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix. In the embodiment, the perception network can be updated based on the back propagation algorithm during the training of the perception network.

[0191] 3. Feature Map: The input data, output data, and intermediate result data of a neural network can all be called feature maps. In a neural network, the data exists in a three-dimensional form (length, width, number of channels), which can be viewed as multiple two-dimensional images stacked together.

[0192] 4. Network Structure Block: The design of a neural network typically involves two steps. The first step is designing a Block, which is composed of atomic units (such as convolution operations and pooling operations). The second step is assembling these Blocks into a complete network structure.

[0193] 5. Channel: This is the third dimension in a feature map, besides length and width. It can be understood as the thickness of the feature map. Additionally, for atomic operations such as convolutional layers, the number of channels is also a dimension.

[0194] 6. Block Width: For a block, the topological relationship of its internal atomic units is fixed, but the number of input and output channels of the atomic units is not fixed. This is a variable property of a block, called the block width.

[0195] 7. Network width: The set of widths of all blocks in a neural network is called the width of the network. It is usually a set of integers.

[0196] 8. Network Depth: The number of blocks stacked when forming a neural network. It is positively correlated with the convolutional stacking depth of the network.

[0197] 9. Network Stages: In neural networks, the input feature map is gradually reduced in size through multiple downsampling operations. The interval between two downsampling operations constitutes a stage in the network. Generally, the blocks within a stage of the network have the same width.

[0198] 10. Network Structure Encoding: In this invention, the depth and width of the network constitute the network structure encoding. Once the topology is determined, the network structure encoding uniquely identifies the structure of the network. The number of bits in the network structure encoding is generally the same as the number of stages in the network.

[0199] 11. Network structure coding candidate set: In this invention, the set of network structure codes that may meet the requirements is called the candidate set.

[0200] 12. Network computational cost: i.e., FLOPs. The number of floating-point multiplications performed in the entire network. This part is the most time-consuming and is therefore used to represent the network's computational cost.

[0201] 13、Weighted sum: when the outputs of different atomic operations are aggregated, for the feature maps of the same shape, sum or stack can be performed. In the present application, sum is always used. However, in the sum, each input is multiplied by a learnable weight. This is weighted sum.

[0202] 14、Network data processing performance: an indicator of the quality of a neural network, such as the accuracy of the network on the test set, the loss function value on the training set, etc. It needs to be manually specified according to business requirements.

[0203] 15、Target task: the final task to be solved, which exists relative to the proxy task. For example, image classification on the ImageNet dataset, face recognition on the business dataset, etc.

[0204] 16、Proxy task: when AutoML optimizes the network structure, a large number of networks need to be evaluated for performance. If training and testing are directly performed on the target task, resource consumption will become unacceptable. Therefore, a smaller task is manually designed to quickly complete the training and testing of the network, which is the proxy task.

[0205] Figure 3 is a schematic diagram of a system architecture provided by an embodiment of the present application, in which an execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices, and a user can input data to the I / O interface 112 through a client device 140. Figure 3

[0206] In the process of pre-processing the input data by the execution device 110, or in the process of performing calculation by the calculation module 111 of the execution device 110 (such as the functional implementation of the neural network in the present application), the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0207] Finally, the I / O interface 112 returns the processing result to the client device 140, thereby providing the user.

[0208] Optionally, the client device 140, for example, can be a control unit in an autonomous driving system, a functional algorithm module in a mobile terminal, etc. For example, the functional algorithm module can be used to implement related tasks.

[0209] ​It is worth mentioning that the training device 120 can generate corresponding target models / rules based on different training data for different targets or different tasks, and the corresponding target models / rules can be used to achieve the above targets or complete the above tasks, thereby providing the required results for the user.

[0210] In Figure 3 the case shown in the figure, the user can manually give the input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send the input data to the I / O interface 112, and if the user's authorization is required to automatically send the input data to the client device 140, the user can set the corresponding permission in the client device 140. The user can view the results output by the execution device 110 in the client device 140, and the specific forms can be specific ways such as display, sound, action, etc. The client device 140 can also act as a data acquisition end to collect the input data input into the I / O interface 112 and the output results output from the I / O interface 112 as new sample data and store them into the database 130. Of course, the collection can also be performed without the client device 140, and the I / O interface 112 directly stores the input data input into the I / O interface 112 and the output results output from the I / O interface 112 as new sample data into the database 130.

[0211] It is worth noting that Figure 3 The system architecture provided by the embodiment of the present application is only a schematic diagram, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 , the data storage system 150 is an external memory relative to the execution device 110, and in other cases, the data storage system 150 can also be placed in the execution device 110.

[0212] Referring to Figure 4 , Figure 4 The flowchart of the neural network structure determination method provided by the embodiment of the present application is shown in Figure 4 , the neural network structure determination method provided by the embodiment of the present application comprises:

[0213] 401、obtain an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, the second block being configured to perform an operation corresponding to the second block according to M first outputs; wherein the M first outputs are obtained by multiplying the output of each first block with a corresponding target weight, the target weight being a trainable weight, and M being an integer greater than 1.

[0214] The embodiment of the present application can search the topology of the neural network. The topology of the neural network in the embodiment can refer to the connection relationship between the blocks in the neural network.

[0215] The block can also be referred to as a network block. The block can include a certain number of atomic operations. The atomic operations can include, but are not limited to, convolution, pooling, residual connection, etc. For example, the block can include the following operation types: 1x3 and 3x1 convolution, 1x7 and 7x1 convolution, 3x3 dilated convolution, 3x3 average pooling, 3x3 max pooling, 5x5 max pooling, 7x7 max pooling, 1x1 convolution, 3x3 convolution, 3x3 separable conv, 5x5 separable conv, 7x7 separable conv, skip connection operation, zero operation (Zero, all neurons in the corresponding position are set to zero), etc. For example, 3x3 average pooling represents average pooling with a kernel size of 3x3; 3x3 max pooling represents max pooling with a kernel size of 3x3; 3x3 dilated convolution represents dilated convolution with a kernel size of 3x3 and a dilation rate of 2; 3x3 separable conv represents separable convolution with a kernel size of 3x3; and 5x5 separable conv represents separable convolution with a kernel size of 5x5.

[0216] In the design of the neural network, two steps are often involved. The first step is to design the block, and the second step is to connect the blocks to combine a complete network structure.

[0217] In the embodiments of the present application, the connection relationship between blocks can be understood as the data transmission direction between blocks. Specifically, a block can perform a corresponding operation on input data and obtain an operation result, and the operation result can be input to the next block connected to the block and serve as the input data of the next block. That is, the connection relationship between two first blocks can represent that the output of one block is the input of the other block.

[0218] In the embodiments of the present application, in order to search the topology of the neural network, the blocks in the neural network to be searched are first connected in a large number, and it is determined in the training process of the model which connections can be retained and which connections can be discarded.

[0219] Next, how to connect the blocks in the neural network to be searched is first described.

[0220] In some scenarios, after the specific types of blocks in the neural network, or the types of atomic operations included in the blocks, the width of the network, or the number of input and output channels of each atomic operation in the block, and the depth, or the number of blocks included in the neural network, and the equal weight are determined, all or part of the blocks in the neural network can be connected two by two.

[0221] In one implementation, all or part of the blocks in the same stage of the neural network can be connected two by two. In the neural network, the input feature map is gradually reduced through multiple downsampling. Between two times of downsampling, a stage of the neural network is formed. Generally, the width (the number of input and output channels of each atomic operation in the block) of the blocks in one stage of the neural network is the same.

[0222] In one implementation, all blocks in the same stage of the neural network can be connected two by two. For example, as shown in FIG. 1, block1, block2 and block3 are blocks in the same stage of the neural network, block1 is connected to block2, block2 is connected to block3, and block1 is connected to block3. Figure 5a Figure 5b ​As shown, block1, block2, block3 and block4 are blocks in the same stage of the neural network, block1 is connected with block2, block2 is connected with block3, block1 is connected with block3, block2 is connected with block4, block1 is connected with block4, and block3 is connected with block4.

[0223] In an implementation, connections can be made between some blocks in the same stage of the neural network. For example, as shown in FIG. 4, block1, block2, block3 and block4 are blocks in the same stage of the neural network, block1 is connected with block2, block2 is connected with block3, block1 is connected with block3, block2 is connected with block4, and block3 is connected with block4, but block1 is not connected with block4. Figure 5c As shown, block1, block2, block3 and block4 are blocks in the same stage of the neural network, block1 is connected with block2, block2 is connected with block3, block1 is connected with block3, block2 is connected with block4, block3 is connected with block4, and block1 is not connected with block4. It should be noted that although it can be considered that there is no connection between block1 and block4, there can be other data paths between block1 and block4, for example, the output of block1 can be used as the input of block2, and the output of block2 can be used as the input of block4, even if there is a data path of block1-block1-block1 between block1 and block4, the present embodiment still considers that there is no connection between block1 and block4.

[0224] In the embodiments of the present application, in order to determine which connections between blocks are retained during the training of the model, a trainable weight parameter (also referred to as a target weight in the embodiments) can be set on the connection between two blocks, and the output of one block can be multiplied by the corresponding target weight (also referred to as a product operation in the embodiments), and then the result of the product operation is input into another block. More specifically, taking setting a target weight 1 between block1 and block2 as an example, in the case where the target weight 1 is not set, the output of block1 is directly used as the input of block2, and in the case where the target weight 1 is set, the output of block1 is multiplied by the target weight 1 first, and then the result of the product operation is used as the input of block2. During the training of the model, as the training iteration proceeds, each target weight is updated, and the size of the updated target weight can represent whether the connection is important.

[0225] It should be understood that when multiple blocks have a connection relationship with the same block, for example, the outputs of multiple blocks are simultaneously used as the inputs of a block, the outputs of the multiple blocks can be multiplied by the corresponding target weights, and then the results of the multiplication operation can be added and used as the inputs of the block connected with the multiple blocks. For example, refer to Figure 6 The first block 1 and the first block 2 are both connected with the second block, the output of the first block 1 is multiplied by the target weight 1, the output of the first block 2 is multiplied by the target weight 2, and the results of the two multiplications can be added, and the results of the two additions can be used as the inputs of the second block.

[0226] In the embodiment of the application, the training device can obtain an initial neural network to be trained. The initial neural network to be trained can be obtained by connecting all or part of the blocks in the neural network two by two after determining the specific types of the blocks, the width and depth of the network, and other weights in the neural network.

[0227] The initial neural network can include M first blocks and a second block, the second block is connected with each first block, and each first block corresponds to a target weight, the initial neural network is configured to multiply the output of each first block by the corresponding target weight to obtain M first outputs, and the second block is configured to perform a corresponding operation of the second block according to the M first outputs. Specifically, the second block in the initial neural network can be configured to perform a corresponding operation of the second block according to the sum of the M first outputs.

[0228] The channel numbers of the inputs and outputs of each first block of the M first blocks are consistent with the channel numbers of the inputs and outputs of the second block, that is, the M first blocks and the second block are blocks in the same stage in the initial neural network.

[0229] Taking M as 3 as an example, refer to Figure 7The initial neural network can include three first blocks (including a first block 1, a first block 2, and a first block 3) and a second block. The second block is connected to the first block 1, the first block 2, and the first block 3. The first block 1 corresponds to a target weight 1, the first block 2 corresponds to a target weight 2, and the first block 3 corresponds to a target weight 3. The initial neural network is configured to perform a multiplication operation on an output of the first block 1 and the target weight 1 to obtain a first output 1, perform a multiplication operation on an output of the first block 2 and the target weight 2 to obtain a first output 2, and perform a multiplication operation on an output of the first block 3 and the target weight 3 to obtain a first output 3. The second block is configured to perform an operation corresponding to the second block according to the first output 1, the first output 2, and the first output 3. Specifically, the second block is configured to perform an operation corresponding to the second block according to a sum of the first output 1, the first output 2, and the first output 3.

[0230] It should be understood that, in addition to the target weights, the blocks having a connection relationship in the initial neural network can also have other operation units, such as an operation unit for adjusting the size of a feature map, and the like. The present application is not limited in this regard.

[0231] 402. Model training is performed on the initial neural network to obtain updated M target weights.

[0232] In the embodiments of the present application, after obtaining the initial neural network to be trained, the training device can perform model training on the initial neural network to obtain updated M target weights.

[0233] In the embodiments of the present application, the training device can perform model training on the initial neural network on a target task and update the M target weights. When the M target weights are stable, the updated M target weights can be obtained. The so-called stable target weights can be understood as the change of the target weights within a certain range during the iterative training process. In some implementations, whether the M target weights are stable can be determined by the number of iterations. For example, the training device can perform model training on the initial neural network for a first preset number of iterations to obtain updated M target weights. The first preset number of iterations can be a pre-set value, which can be determined according to the total number of iterations required. For example, when the number of iterations reaches a certain percentage of the total number of training required, it is considered that the M target weights are stable.

[0234] In the embodiments of the present application, the updated M target weights are obtained at a fixed percentage position of the total training rounds, which ensures the stability of the target weights and the sufficient training of the network after topology optimization. Meanwhile, the time for single topology optimization is maintained, which is basically the same as the original training time, ensuring the search efficiency.

[0235] It should be understood that, in the process of model training on the initial neural network, the network general weights (that is, the weights to be trained in the atomic operation included in the block) and the M target weights can be updated simultaneously, or the network general weights and the M target weights can be updated alternately, and the present application is not limited.

[0236] 403、According to the updated M target weights, update the connection relationship between the second block and the M first blocks in the initial neural network to obtain a first neural network; wherein the second block in the first neural network is used to perform the operation corresponding to the second block according to the output of the first block corresponding to the largest N target weights in the updated M target weights, and the N is less than the M.

[0237] In the embodiments of the present application, after the model training on the initial neural network to obtain the updated M target weights, the connection relationship between the second block and the M first blocks in the initial neural network can be updated according to the updated M target weights to obtain a first neural network; wherein the second block in the first neural network is used to perform the operation corresponding to the second block according to the output of the first block corresponding to the largest N target weights in the updated M target weights. Specifically, the second block in the first neural network can be used to perform the operation corresponding to the second block according to the sum result of the output of the first block corresponding to the largest N target weights in the updated M target weights.

[0238] The size of the updated M target weights can indicate whether the connection between the blocks is important, and the larger the size of the updated target weight, the more important the connection between the blocks. Specifically, the connection where the largest N target weights in the updated M target weights are located can be retained, and the connection where the target weights other than the largest N target weights in the updated M target weights are located can be eliminated.

[0239] In one implementation, the M first blocks and a second block in the initial neural network are sequentially connected in series, and the second block is a terminal point of the series connection, the M first blocks in the initial neural network include a target block, the target block is connected with the second block on the series connection, and in a case where the updated target weight corresponding to the target block does not belong to the N largest target weights in the updated M target weights, the second block in the first neural network is further configured to perform a calculation corresponding to the second block according to an output of the target block.

[0240] That is, regardless of whether the updated target weight corresponding to the target block is one of the N largest target weights in the updated M target weights, the connection between the target block and the second block is always retained, the connection between the target block and the second block can be referred to as a backbone connection, and the backbone connection cannot be removed to ensure that the backbone architecture of the entire neural network is not damaged.

[0241] Specifically, in one implementation, if the updated target weight belonging to the backbone connection is one of the N largest target weights in the M target weights, the connection of the N updated target weights can be retained, and if the updated target weight belonging to the backbone connection is not one of the N largest target weights in the M target weights, the connection of the N+1 updated target weights can be retained.

[0242] For example, refer to Figure 8 For example, taking the number of M as 3 and N as 1, if the updated target weight 1 is greater than the updated target weight 2 and the updated target weight 3, the connection between the first block 1 and the second block and the connection between the first block 3 and the second block (backbone connection) can be retained, as shown in Figure 8 The second block in the first neural network is configured to perform a calculation corresponding to the second block according to the outputs of the first block 1 and the first block 3, and specifically, the second block in the first neural network is configured to perform a calculation corresponding to the second block according to a sum of the outputs of the first block 1 and the first block 3.

[0243] For example, refer to Figure 9 For example, taking the number of M as 3 and N as 1, if the updated target weight 3 is greater than the updated target weight 1 and the updated target weight 2, only the connection between the first block 3 and the second block can be retained, as shown in Figure 8As shown, the second block in the first neural network is used to perform the operation corresponding to the second block according to the output of the first block 3.

[0244] In the embodiments of the present application, each block in the initial neural network can be regarded as the second block in the above embodiments, the block outputting as the input of the second block can be regarded as the first block, and the elimination and selection of the above connection can be performed to obtain the first neural network.

[0245] In the embodiments of the present application, after obtaining the first neural network, the model training of the first neural network can be performed until the data processing accuracy of the first neural network meets a preset condition or the iteration number of the model training reaches a second preset iteration number, to obtain a second neural network.

[0246] Specifically, the training device can obtain training data, and the training data includes at least one of image data, text data, and voice data; accordingly, the training device can perform model training on the initial neural network according to the training data.

[0247] The embodiments of the present application provide a neural network structure determination method, which comprises: obtaining an initial neural network to be trained, the initial neural network comprising M first structure blocks block and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, the second block being used to perform the operation corresponding to the second block according to M first outputs; wherein the M first outputs are obtained by multiplying the output of each first block with the corresponding target weight, the target weight being a trainable weight, and M being an integer greater than 1; performing model training on the initial neural network to obtain updated M target weights; updating the connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights to obtain a first neural network; wherein the second block in the first neural network is used to perform the operation corresponding to the second block according to the output of the N first blocks corresponding to the N target weights with the largest value in the updated M target weights, and N is less than M. In the above manner, in the search process of the connection relationship between the blocks of the initial neural network, the trainable target weight is added to the connection between the blocks, the importance of the connection between the blocks is judged according to the size of the updated target weight, and the selection and elimination of the connection relationship between the blocks are based on the size of the updated target weight, so as to realize the search of the topology of the neural network.

[0248] Referring to Figure 10 , Figure 10 A flowchart of a neural network structure determination method provided by an embodiment of the present application is shown in FIG. 1. The neural network structure determination method provided by the embodiment of the present application includes the following steps. Figure 10

[0249] 1001. An initial neural network to be trained is acquired, the initial neural network including M first blocks and a second block, the second block being connected with each first block, and each first block corresponding to a target weight, each first block being configured to perform a corresponding operation of the first block according to a corresponding first output, wherein the corresponding first output of each first block is obtained by multiplying the target weight corresponding to the first block and an output of the second block, the target weight being a trainable weight, and M being an integer greater than 1.

[0250] and Figure 4 The difference between the corresponding embodiments is that, Figure 4 In the embodiment, the outputs of the M first blocks are used as the input of the second block, and in the subsequent selection process of the connection relationship, the selection is also based on the sizes of the M updated target weights corresponding to the M first blocks. In the embodiment, the output of the second block is used as the input of the M first blocks. For example, as shown in FIG. 2, taking M as 3, the M first blocks include a first block 1, a first block 2 and a first block 3, and the output of the second block can be used as the input of the first block 1, the input of the first block 2 and the input of the first block 3. Specifically, the output of the second block can be multiplied by the target weight 1, and the multiplication result is used as the input of the first block 1. The output of the second block can be multiplied by the target weight 2, and the multiplication result is used as the input of the first block 2. The output of the second block can be multiplied by the target weight 3, and the multiplication result is used as the input of the first block 3. Figure 11

[0251] In a possible implementation, the number of channels of the input and output of each first block of the M first blocks is consistent with the number of channels of the input and output of the second block.

[0252] It should be understood that other specific descriptions of step 1001 can refer to similar descriptions in the corresponding embodiments of step 401, which will not be described here again.

[0253] 1002. Model training is performed on the initial neural network to obtain M updated target weights. ​​

[0254] In a possible implementation, the training device can perform model training on the initial neural network for a first preset number of iterations to obtain updated M target weights.

[0255] The specific description of step 1002 can refer to the description in the corresponding embodiment of step 402, which will not be repeated here.

[0256] 1003, updating a connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights to obtain a first neural network; wherein the first blocks corresponding to the largest N target weights in the updated M target weights are used to perform operations corresponding to the first blocks according to the output of the second block, and the N is less than the M.

[0257] Similar to the corresponding embodiment of step 403, in this embodiment, the first blocks corresponding to the largest N target weights in the updated M target weights are used to perform operations corresponding to the first blocks according to the output of the second block.

[0258] In order to preserve the backbone connection, in a possible implementation, the second block and the M first blocks in the initial neural network form a serial connection in turn, and the second block is the starting point of the serial connection, and the M first blocks in the initial neural network include a target block, the target block is connected with the second block on the serial path, and in the case that the updated target weight corresponding to the target block does not belong to the largest N target weights in the updated M target weights, the target block in the first neural network is also used to perform operations corresponding to the target block according to the output of the second block.

[0259] In a possible implementation, the N is 1.

[0260] For example, refer to Figure 12 Taking the number of M as 3 and N as 1 as an example, if the updated target weight 1 is greater than the updated target weight 2 and the updated target weight 3, the connection between the first block 1 and the second block and the connection between the first block 3 and the second block (the backbone connection) can be preserved, as shown in Figure 12As shown, the output of the second block in the first neural network is used as the input of the first block 1 and the first block 3, respectively, the first block 1 is used to perform the operation corresponding to the first block 1 according to the output of the second block, and the first block 3 is used to perform the operation corresponding to the first block 3 according to the output of the second block, and specifically, the first block 3 in the first neural network is used to perform the operation corresponding to the first block 3 according to the sum of the output of the second block and the output of the first block 2.

[0261] For example, refer to Figure 13 For example, when the number of M is 3 and the number of N is 1, if the updated target weight 2 is greater than the updated target weight 1 and the updated target weight 3, only the connection between the first block 1 and the second block can be reserved, as shown in Figure 13 As shown, a block 1 in the first neural network is used to perform the operation corresponding to the first block 1 according to the output of the second block.

[0262] In the embodiments of the present application, each block in the initial neural network can be regarded as the second block in the above embodiments, the block with the output of the second block as the input can be regarded as the first block, and the elimination and selection of the connection can be performed to obtain the first neural network.

[0263] In a possible implementation, the training device can perform model training on the first neural network until the data processing accuracy of the first neural network meets a preset condition or the number of iterations of model training reaches a second preset number of iterations, to obtain the second neural network.

[0264] In a possible implementation, the data to be trained includes at least one of the following: image data, text data, and voice data, and correspondingly, the training device can perform model training on the initial neural network according to the data to be trained.

[0265] The application provides a neural network structure determination method, which comprises the following steps: obtaining an initial neural network to be trained, wherein the initial neural network comprises M first blocks and a second block, the second block is connected with each first block, each first block corresponds to a target weight, and each first block is used for performing corresponding operation of the first block according to a corresponding first output; wherein the first output corresponding to each first block is obtained by performing product operation on the target weight corresponding to the first block and the output of the second block, the target weight is a trainable weight, and M is an integer greater than 1; the target weight is a trainable weight, and M is an integer greater than 1; performing model training on the initial neural network to obtain updated M target weights; updating the connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights to obtain a first neural network; wherein the first blocks corresponding to the largest N target weights in the updated M target weights are used for performing corresponding operation of the first blocks according to the output of the second block, and N is less than M. In the above manner, in the searching process of the connection relationship between blocks of the initial neural network, the trainable target weight is added to the connection between blocks, the importance of the connection relationship between blocks is judged according to the size of the updated target weight, and the connection relationship between blocks is selected and removed based on the size of the updated target weight, so that the search of the topology of the neural network is realized.

[0266] Reference Figure 14 , Figure 14 The neural network structure determination method provided by the embodiment of the application is shown in the flowchart as shown in Figure 14 The neural network structure determination method provided by the embodiment of the application comprises the following steps:

[0267] 1401, obtaining a plurality of target codes, each target code being used for indicating a candidate neural network, the plurality of target codes comprising a first target code and a plurality of first codes, and the first target code being used for indicating a first neural network.

[0268] In an embodiment of the present application, when performing structure search of a neural network, the structural features of the neural network can be written in the form of codes, each code being used to indicate at least one of the following structural features of a candidate neural network: the type of operation unit included in the candidate neural network, the number of operation units included in the candidate neural network, and the input feature and output feature channel number of the operation units included in the candidate neural network. Wherein, the operation unit can refer to each atomic operation in a block, and in another expression, each code is used to indicate the type of atomic operation included in a candidate neural network, the number of atomic operations included in the candidate neural network, and the input feature and output feature channel number of the atomic operations included in the candidate neural network. Since the input feature and output feature channel number of the atomic operations in each block are the same in the same stage, it is equivalent to each code being used to indicate the input feature and output feature channel number of the block included in a candidate neural network.

[0269] In an implementation, the training device can generate a plurality of candidate codes, and screen the plurality of candidate codes based on a preset rule, wherein the preset rule can be at least one of the following: selecting, from the plurality of candidate codes, a candidate neural network whose required calculation amount when running the indicated candidate neural network is less than a first preset value, whose included weight amount is less than a second preset value, and whose running speed when running the indicated candidate neural network is higher than a third preset value. Wherein, the calculation amount can be the number of floating-point multiplications required in the entire neural network, and the floating-point multiplication operation is the most time-consuming, and thus can be used to represent the calculation amount of the neural network. The first preset value, the second preset value, and the third preset value can be preset.

[0270] In an embodiment of the present application, a plurality of codes can be clustered to obtain a plurality of code sets, each code set corresponding to a clustering category, and the plurality of code sets include a target code set, and the target code set includes the plurality of target codes. Wherein, the plurality of codes can be obtained after screening a plurality of candidate codes. The first target code can be one code in the target code set, and in an implementation, the first target code can be the clustering center of the target code set. The first target code is used to indicate a first neural network.

[0271] It should be understood that the above clustering can be K-Means algorithm, DBSCAN algorithm, BIRCH algorithm, and MeanShift algorithm, etc.

[0272] 1402, model training is performed on the first neural network to obtain the data processing accuracy of the first neural network.

[0273] In the embodiments of the present application, the training device can select a first neural network indicated by one of the plurality of target encodings (a first target encoding) to perform model training, so as to obtain the data processing accuracy of the first neural network.

[0274] It should be understood that after the first neural network is obtained, the first neural network can be optimized in network topology, for example, the first neural network can be optimized by the neural network structure determination method described in the corresponding embodiments. Figures 4 to 13 The first neural network can be optimized according to the neural network structure determination method described in the corresponding embodiments, and in this case, the optimized first neural network can be subjected to model training to obtain the data processing accuracy of the first neural network.

[0275] In the embodiments of the present application, the neural network data processing accuracy can be the value of the loss function of the training network, the test accuracy of the neural network, and the like, and the present application is not limited thereto.

[0276] In the embodiments of the present application, not all of the plurality of candidate neural networks indicated by the plurality of target encodings are subjected to model training, and based on the data processing accuracy of the plurality of candidate neural networks, a candidate neural network with higher accuracy is selected as the search result of the model, but only one of the target encodings (a first target encoding) indicates a first neural network, and the first neural network is subjected to model training, and then based on the difference degree between the target encodings, the data processing accuracy of the candidate neural networks indicated by the remaining target encodings (a plurality of first encodings) in the plurality of target encodings is determined.

[0277] 1403、According to the difference degree between the first target encoding and the plurality of first encodings and the data processing accuracy of the first neural network, the data processing accuracy of the candidate neural network indicated by each first encoding is determined.

[0278] In the embodiments of the present application, after the first neural network is subjected to model training to obtain the data processing accuracy of the first neural network, the data processing accuracy of the candidate neural network indicated by each first encoding can be determined according to the difference degree between the first target encoding and the plurality of first encodings and the data processing accuracy of the first neural network.

[0279] In the embodiments of the present application, the target encoding can include a plurality of bits, and each bit indicates a structural feature of the candidate neural network. In order to eliminate the dimensional difference between the bits and the subsequent influence, each target encoding can be standardized. For example, the mean and standard deviation of the plurality of target encodings can be calculated for each bit of the target encoding, and then each bit of the target encoding is subtracted by the mean and divided by the standard deviation. Thereafter, the dimension of the target encoding will no longer affect the subsequent algorithm.

[0280] Next, it is described how to determine the data processing accuracy of the candidate neural network indicated by each first encoding according to the difference between the first target encoding and the plurality of first encodings and the data processing accuracy of the first neural network.

[0281] Exemplarily, a Gaussian process can be utilized. The Gaussian process is a very classic and mature machine learning algorithm, which can estimate the value of other sample points according to the distance between two sample points and the value of a part of sample points. In the embodiment, the sample points are the target encodings, and the value of the sample points is the data processing accuracy of the candidate neural network indicated by the target encoding. A specific Gaussian process is uniquely determined by its mean function and covariance function. Using the Gaussian process for modeling is actually learning the mean function and the covariance function. In the embodiment, the Gaussian process can be learned in the following manner:

[0282] Firstly, the covariance function can be learned, wherein the covariance function can be Formula 1 as follows:

[0283] ……(1)

[0284] In the formula, the target encoding is , and the standard deviation to be learned is . The calculation method of the standard deviation is: all the target encodings that have been completed are grouped in pairs, the encoding distance between them is calculated , these distances are in the order of , and then is taken as the estimated value of . In this way, the learning of the covariance function is completed.

[0285] Then, the performance mean and standard deviation of all untrained encodings can be calculated. Assuming that n target encodings have been completed and trained, and the data processing accuracy of the candidate neural network indicated by the target encodings is , the following variables can be defined:

[0286] ……(2)

[0287] In the formula, the kernel is calculated by Formula 1, , and the i-th completed training target encoding is

[0288] After obtaining the above two matrices, for any untrained target encoding x, the value of the covariance function between it and all the completed training encodings is calculated according to Formula 3:

[0289] ……(3)

[0290] According to the formulas 1, 2, 3, 4, the mean value of it can be calculated:

[0291] ……(4)

[0292] Wherein , I is a unit matrix.

[0293] According to the formulas 1, 2, 3, 5, the covariance function of it can be calculated:

[0294] ……(5)

[0295] Then the data processing accuracy of the candidate neural network indicated by each target code can be calculated, and the mean value function and the covariance function of the target code have been obtained, at this time, the data processing accuracy of the candidate neural network indicated by the target code can be predicted according to the formula 6:

[0296] ……(6)

[0297] Wherein, The meaning of x is that the target code x has the highest data processing accuracy relative to the candidate neural network currently indicated, and how much data processing accuracy is expected to be improved, that is, Expected Improvement (EI). The larger the value is, the more priority it should be trained in the next round. In the formula, f(x) is a Gaussian process function:

[0298] In the formula, f(x) is a Gaussian process function:

[0299] f(x) is obtained by the formula 4, is obtained by the formula 5.

[0300] In the above manner, the data processing accuracy of the candidate neural network indicated by each target code can be predicted by the above process, and the data processing accuracy of the candidate neural network indicated by each first code is obtained.

[0301] 1404, obtaining a first candidate neural network with the highest data processing accuracy in the candidate neural networks indicated by the plurality of target codes.

[0302] In the embodiment of the application, after obtaining the data processing accuracy of each candidate neural network in the candidate neural networks indicated by the plurality of target codes, a first candidate neural network with the highest data processing accuracy in the candidate neural networks indicated by the plurality of target codes can be selected.

[0303] 1405, model training is performed on the first candidate neural network to obtain a first target neural network.​

[0304] Since the data processing accuracy of the candidate neural network other than the first target neural network at this time is determined based on the difference degree between the target codes, it is not very accurate, and therefore the first candidate neural network with the highest data processing accuracy among the candidate neural networks can be subjected to model training to obtain the first target neural network.

[0305] It should be understood that after the first candidate neural network is obtained, the network topology of the first candidate neural network can be optimized, for example, the first candidate neural network can be optimized by the neural network structure determination method described in the corresponding embodiment, in which case the training device can perform model training on the optimized first candidate neural network to obtain the first target neural network. Figures 4 to 13 The first candidate neural network can be optimized by the neural network structure determination method described in the corresponding embodiment, in which case the training device can perform model training on the optimized first candidate neural network to obtain the first target neural network.

[0306] Specifically, in one implementation, the first candidate neural network includes M first blocks and a second block, the second block is connected to each first block, and each first block corresponds to a target weight, the first candidate neural network is configured to multiply the output of each first block with the corresponding target weight to obtain M first outputs, and the second block is configured to perform the operation corresponding to the second block according to the M first outputs; wherein the target weight is a trainable weight, and M is an integer greater than 1.

[0307] The training device can perform model training on the first candidate neural network to obtain updated M target weights, and update the connection relationship between the second block and the M first blocks in the first candidate neural network according to the updated M target weights to obtain a second neural network; wherein the second block in the second neural network is configured to perform the operation corresponding to the second block according to the output of the first block corresponding to the largest N target weights in the updated M target weights, and N is less than M, and the second neural network is subjected to model training to obtain the first target neural network.

[0308] In one implementation, the first candidate neural network includes M first blocks and a second block, the second block is connected to each first block, and each first block corresponds to a target weight, the first candidate neural network is configured to multiply the output of each first block with the corresponding target weight to obtain M first outputs, and the second block is configured to perform the operation corresponding to the second block according to the M first outputs; wherein the target weight is a trainable weight, and M is an integer greater than 1.

[0309] The training device can perform model training on the first candidate neural network to obtain updated M target weights, and update a connection relationship between the second block and the M first blocks in the first candidate neural network according to the updated M target weights to obtain a second neural network, wherein the first blocks corresponding to the largest N target weights in the updated M target weights are used to perform the operation corresponding to the first blocks according to the output of the second block, and the N is less than the M. The first target neural network is obtained by performing model training on the second neural network.

[0310] In the embodiments of the present application, the training device can also obtain the data processing accuracy of the first target neural network, the plurality of target encodings include a second target encoding, the second target encoding is used to indicate the first target neural network, and the data processing accuracy of the candidate neural network indicated by each target encoding in the plurality of target encodings except the second target encoding is determined according to the difference between the second target encoding and the encoding in the plurality of target encodings except the second target encoding and the data processing accuracy of the first target neural network. Then, the training device can determine the second candidate neural network with the highest data processing accuracy according to the data processing accuracy of the candidate neural networks indicated by the plurality of target encodings, and perform model training on the second candidate neural network to obtain a second target neural network.

[0311] Then the training device can repeat the above process, and through a preset number of iterations (for example, 4 rounds), a very ideal model can be obtained as the result of neural model structure search.

[0312] For example, if the clustering result obtains 10 clustering categories, the candidate neural network model with the highest data processing accuracy in the encoding set corresponding to the 10 clustering categories is obtained each time to determine the data processing accuracy of the candidate neural networks indicated by the remaining target encodings in the encoding set, and after 4 rounds of iteration processing, 40 candidate neural networks can be trained. Then, the one with the highest data processing accuracy among the 40 candidate neural networks can be selected as the result of neural model structure search.

[0313] In the embodiments of the present application, model training is not performed on all candidate neural networks indicated by the plurality of target encoding indications, and based on the data processing accuracy of the plurality of candidate neural networks, a candidate neural network with higher accuracy is selected as the search result of the model. Instead, only a first neural network indicated by one target encoding (first target encoding) is selected, and model training is performed on the first neural network. Then, based on the difference degree between the target encodings, the data processing accuracy of the candidate neural networks indicated by the remaining target encodings (a plurality of first encodings) in the plurality of target encodings is determined. Compared with various topology search algorithms, the number of model training is greatly reduced, and the search efficiency is greatly improved.

[0314] Next, taking the candidate neural network as the MobileNetV2 network, the topology optimization of the neural network on the ImageNet image classification task is taken as an example for description.

[0315] First, the encoding candidate set can be generated. The MobileNetV2 network itself can be divided into 7 stages in series, and the depth (number of operation units included in the candidate neural network) of the neural network of each stage, the output feature channel number of the operation units included in the candidate neural network and other structural characteristics can be encoded. For example: [1, 2, 3, 4, 3, 3, 1, 16, 24, 32, 48, 64, 192, 376] represents that the 7 stages are to repeat the basic network structure (i.e. the operation unit mentioned above) 1, 2, 3, 4, 3, 3, and 1 times, and the output channel numbers are 16, 24, 32, 48, 64, 192, and 376 respectively. Each encoding length is 14 bits, and the encoding can uniquely determine a candidate neural network. After the encoding method is determined, the upper and lower limits of the search can also be set for each bit of the encoding. For example, the search upper and lower limits of a 14-bit search are limited to 3 and 1, 4 and 1, 5 and 2, 6 and 2, 5 and 1, 5 and 1, 3 and 1, 48 and 16, 48 and 16, 64 and 24, 96 and 32, 96 and 32, 256 and 112, and 512 and 256.

[0316] Then, the training device can uniformly generate a large number of encodings according to the number of encoding bits and the search upper and lower limits of each bit. The calculation amount of the neural network indicated by each generated encoding is calculated. According to the specified limit, the encodings that meet the requirements are retained to form the encoding candidate set. For example, under the 300M limit, about 20,000 candidate encodings can be obtained.

[0317] Afterwards, the training device can model the candidate encoding set, specifically by standardizing each encoding. Specifically, for each bit of the encoding, the mean and standard deviation are calculated across the entire candidate set, then the mean is subtracted from each bit of the encoding, and the result is divided by the standard deviation. After this, the dimensions of the encoding will no longer affect subsequent algorithms. The training device can then perform K-Means clustering on the standardized encodings. The resulting cluster centers (the first target encoding) can be considered the most representative structures in the current encoding space, and their data processing accuracy can represent the performance of the entire class. Evaluating their performance allows for more efficient modeling of the performance of all network structures in the entire search space. In this way, 10 network encodings are generated in the first round, and training begins for each, with topology optimization and data processing accuracy obtained. Next, the training device can train a fully connected network. Specifically, for each encoding to be trained, the training device converts it into a specific neural network using a network parser D. Before topology optimization, a fully connected network is generated. The specific fully connected rule is to connect all network blocks within each stage, forming a very dense network structure. The network's ordinary weights and target weights (see [reference] for details) are then processed. Figures 4 to 13 (As described in the corresponding embodiment) Simultaneously, optimization is performed. When the number of training rounds reaches 40% of the total number of rounds, the updated target weights are considered stable. Based on the updated target weights, the topology of the neural network is optimized, and the optimized neural network is trained until convergence. The candidate encoding model is then updated. Specifically, after all 10 network encodings have completed model training and data processing performance is obtained, the modeling of all candidate encodings can be updated. Based on the updated cluster centers, 10 new encodings are generated, and the above process is repeated (e.g., four rounds of repetition). A total of 40 models are trained. As the search progresses, it can be found that the performance of the later trained networks is better than that of the earlier trained ones. Among these 40 models, the best one is selected as the result of the neural model structure search and output.

[0318] In terms of specific product form, the embodiments of this application can be provided to users as a component of an AutoML system. Users provide the platform with a dataset, network size requirements (weight requirements / speed requirements, etc.), and a basic network structure to be adjusted. Figures 4 to 14 The neural network structure determination method described in the corresponding embodiment can obtain the optimized network structure. This embodiment can be provided to users as a component of an AutoML system via cloud services.

[0319] Alternatively, this embodiment can also be provided to users as a standalone algorithm package, which users can then follow. Figures 4 to 14 The neural network structure determination method described in the corresponding embodiment can obtain an optimized network structure.

[0320] Referring to Figure 15 , the neural network structure determination apparatus provided by the embodiments of the present application can include a network code generator A, a network size determinator B, a network code modeler C, a network parser D, a trainer E, and an edge selector F, and the mutual relationship therebetween can be seen from Figure 15 .

[0321] Specifically, the network code generator A can generate multiple codes as evenly as possible according to a possible code space. These codes define the structural features of the neural network. During the generation process, it is determined whether to add the code to the code candidate set according to the result of the network size determinator B.

[0322] The network size determinator B can evaluate the calculation amount, weight amount, running speed, etc. of the candidate neural network indicated by each code, and determine whether to meet the user's limit.

[0323] The network code modeler C can model the code, evaluate the possible data processing accuracy of the candidate neural network indicated by each code, and send the code to be trained to the network parser D, and at the same time receive the data processing accuracy of the candidate neural network indicated by the code returned by the trainer E. During the search process, the modeling result is updated according to the received data processing accuracy, so that the evaluation of itself becomes more and more accurate. Until the search is completed, the target code with the optimal performance is given.

[0324] The network parser D can convert the code into a specific neural network.

[0325] The trainer E can train a specific neural network according to the training data provided by the user, and output the data processing accuracy (such as test accuracy, training loss function value, etc.) and the trained neural network.

[0326] The edge selector F can optimize the topology of the converted neural network.

[0327] Referring to Figure 16 , Figure 16 The structure of the neural network structure determination apparatus 1600 provided by the embodiments of the present application is shown in Figure 16 , the neural network structure determination apparatus 1600 provided by the embodiments of the present application can include:

[0328] The obtaining module 1601 is configured to obtain an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each of the first blocks, and each of the first blocks corresponding to a target weight, and the second block being configured to perform an operation corresponding to the second block according to M first outputs; wherein the M first outputs are obtained by multiplying an output of each of the first blocks with a corresponding target weight, the target weight being a trainable weight, and the M being an integer greater than 1.

[0329] The specific description of the obtaining module 1601 can refer to the description in step 401 and the corresponding embodiments, which will not be repeated here.

[0330] The model training module 1602 is configured to perform model training on the initial neural network to obtain updated M target weights.

[0331] The specific description of the model training module 1602 can refer to the description in step 402 and the corresponding embodiments, which will not be repeated here.

[0332] The model updating module 1603 is configured to update a connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights, to obtain a first neural network; wherein the second block in the first neural network is configured to perform an operation corresponding to the second block according to outputs of first blocks corresponding to the maximum N target weights in the updated M target weights, and the N being less than the M.

[0333] The specific description of the model updating module 1603 can refer to the description in step 403 and the corresponding embodiments, which will not be repeated here.

[0334] In a possible implementation, the M first blocks and the second block in the initial neural network are sequentially formed into a serial connection, and the second block is a terminal point of the serial connection, the M first blocks in the initial neural network comprising a target block, the target block being connected with the second block on the serial connection, and in a case where an updated target weight corresponding to the target block does not belong to the maximum N target weights in the updated M target weights, the second block in the first neural network is further configured to perform an operation corresponding to the second block according to an output of the target block.

[0335] In a possible implementation, the N is 1.

[0336] In a possible implementation, the model training module is configured to perform model training on the initial neural network for a first preset number of iterations to obtain updated M target weights.

[0337] In a possible implementation, the model training module is configured to perform model training on the first neural network until data processing accuracy of the first neural network meets a preset condition or a number of iterations of model training reaches a second preset number of iterations, to obtain a second neural network.

[0338] In a possible implementation, a number of input and output channels of each first block of the M first blocks is consistent with a number of input and output channels of the second block.

[0339] In a possible implementation, the second block in the initial neural network is configured to perform corresponding operation of the second block according to a sum result of the M first outputs.

[0340] The second block in the first neural network is configured to perform corresponding operation of the second block according to a sum result of outputs of the first blocks corresponding to the maximum N target weights in the updated M target weights.

[0341] In a possible implementation, the obtaining module is configured to obtain training data, and the training data includes at least one of the following: image data, text data and voice data; correspondingly, the model training on the initial neural network includes:

[0342] The model training module is configured to perform model training on the initial neural network according to the training data.

[0343] Reference Figure 17 , Figure 17 The neural network structure determination apparatus 1700 provided by the embodiments of the present application can include: Figure 17

[0344] ​The obtaining module 1701 is configured to obtain an initial neural network to be trained, the initial neural network comprising M first blocks and a second block, the second block being connected with each of the first blocks, and each of the first blocks corresponding to a target weight, and each of the first blocks being configured to perform a corresponding operation of the first block according to a corresponding first output; wherein the corresponding first output of each of the first blocks is obtained by performing a multiplication operation on the target weight corresponding to the first block and an output of the second block, the target weight being a trainable weight, and M being an integer greater than 1.

[0345] The specific description of the obtaining module 1701 can refer to the description in step 1001 and the corresponding embodiments, which will not be repeated here.

[0346] The model training module 1702 is configured to perform model training on the initial neural network to obtain updated M target weights.

[0347] The specific description of the model training module 1702 can refer to the description in step 1002 and the corresponding embodiments, which will not be repeated here.

[0348] The model updating module 1703 is configured to update a connection relationship between the second block and the M first blocks in the initial neural network according to the updated M target weights to obtain a first neural network; wherein the first blocks corresponding to the largest N target weights in the updated M target weights are configured to perform a corresponding operation of the first block according to the output of the second block, and N is less than M.

[0349] The specific description of the model updating module 1703 can refer to the description in step 1003 and the corresponding embodiments, which will not be repeated here.

[0350] In a possible implementation, the second block and the M first blocks in the initial neural network form a serial connection in sequence, and the second block is the starting point of the serial connection, and the M first blocks in the initial neural network comprise a target block, the target block being connected with the second block on the serial connection, and in a case where the updated target weight corresponding to the target block does not belong to the largest N target weights in the updated M target weights, the target block in the first neural network is further configured to perform a corresponding operation of the target block according to the output of the second block.

[0351] In a possible implementation, N is 1.

[0352] In a possible implementation, the model training module is configured to perform model training on the initial neural network for a first preset number of iterations to obtain M updated target weights.

[0353] In a possible implementation, the model training module is configured to perform model training on the first neural network until data processing accuracy of the first neural network meets a preset condition or a number of iterations of model training reaches a second preset number of iterations, to obtain a second neural network.

[0354] In a possible implementation, a number of input and output channels of each first block of the M first blocks is consistent with a number of input and output channels of the second block.

[0355] In a possible implementation, the obtaining module is configured to obtain training data, the training data including at least one of image data, text data, and voice data; and correspondingly, the model training on the initial neural network includes:

[0356] The model training module is configured to perform model training on the initial neural network according to the training data.

[0357] Reference Figure 18 , Figure 18 The neural network structure determination apparatus 1800 provided by the embodiments of the present application is shown in FIG. 18, which can include: Figure 18

[0358] The obtaining module 1801 is configured to obtain a plurality of target encodings, each target encoding being used to indicate a candidate neural network, the plurality of target encodings including a first target encoding and a plurality of first encodings, the first target encoding being used to indicate a first neural network.

[0359] The specific description of the obtaining module 1801 can be referred to the description in step 1401 and the corresponding embodiments, which will not be repeated here.

[0360] The model training module 1802 is configured to perform model training on the first neural network to obtain data processing accuracy of the first neural network.

[0361] The specific description of the model training module 1802 can be referred to the description in step 1402 and the corresponding embodiments, which will not be repeated here.

[0362] ​The precision determination module 1803 is configured to determine, according to the difference degree between the first target code and the plurality of first codes and the data processing precision of the first neural network, the data processing precision of the candidate neural network indicated by each first code.

[0363] The specific description of the precision determination module 1803 can be referred to the description in step 1403 and corresponding embodiments, which will not be repeated here.

[0364] The acquisition module 1801 is configured to acquire a first candidate neural network with the highest data processing precision from the plurality of candidate neural networks indicated by the plurality of target codes.

[0365] The specific description of the acquisition module 1801 can be referred to the description in step 1401 and corresponding embodiments, which will not be repeated here.

[0366] The model training module 1802 is configured to perform model training on the first candidate neural network to obtain a first target neural network.

[0367] The specific description of the model training module 1802 can be referred to the description in step 1405 and corresponding embodiments, which will not be repeated here.

[0368] In a possible implementation, the acquisition module is configured to acquire the data processing precision of the first target neural network, the plurality of target codes include a second target code, and the second target code is used to indicate the first target neural network.

[0369] According to the difference degree between the second target code and the codes other than the second target code in the plurality of target codes and the data processing precision of the first target neural network, the data processing precision of the candidate neural network indicated by each target code other than the second target code in the plurality of target codes is determined.

[0370] According to the data processing precisions of the candidate neural networks indicated by the plurality of target codes, a second candidate neural network with the highest data processing precision is determined, and model training is performed on the second candidate neural network to obtain a second target neural network.

[0371] In a possible implementation, each target code is used to indicate at least one of the following structural features of a candidate neural network:

[0372] The type of the operation unit included in the candidate neural network, the number of operation units included in the candidate neural network, and the input feature and output feature channel numbers of the operation units included in the candidate neural network.

[0373] In a possible implementation, the apparatus further includes:

[0374] a clustering module, configured to cluster the plurality of encodings to obtain a plurality of encoding sets, each encoding set corresponding to a cluster category, the plurality of encoding sets comprising a target encoding set, the target encoding set comprising the plurality of target encodings.

[0375] In a possible implementation, the first target encoding is a cluster center of the target encoding set.

[0376] In a possible implementation, each target encoding indicates a candidate neural network that satisfies at least one of the following conditions:

[0377] A computation amount required when running the candidate neural network indicated by each target encoding is less than a first preset value;

[0378] A weight amount included in the candidate neural network indicated by each target encoding is less than a second preset value; and

[0379] A running speed when running the candidate neural network indicated by each target encoding is higher than a third preset value.

[0380] In a possible implementation, the first candidate neural network comprises M first blocks and a second block, the second block is connected with each first block, and each first block corresponds to a target weight, and the second block is configured to perform an operation corresponding to the second block according to M first outputs; wherein the M first outputs are obtained by performing a multiplication operation on an output of each first block and a corresponding target weight, the target weight is a trainable weight, and the M is an integer greater than 1.

[0381] The model training module is configured to perform model training on the first candidate neural network to obtain updated M target weights;

[0382] According to the updated M target weights, update a connection relationship between the second block and the M first blocks in the first candidate neural network to obtain a second neural network; wherein the second block in the second neural network is configured to perform an operation corresponding to the second block according to outputs of first blocks corresponding to the largest N target weights in the updated M target weights, and the N is less than the M.

[0383] Perform model training on the second neural network to obtain the first target neural network.

[0384] In one possible implementation, the first candidate neural network includes M first structural blocks and a second block, the second block being connected to each first block, and each first block corresponding to a target weight. Each first block is used to perform the operation corresponding to the first block based on the corresponding first output. The first output corresponding to each first block is obtained by multiplying the target weight corresponding to the first block with the output of the second block. The target weight is a trainable weight, and M is an integer greater than 1.

[0385] The model training module is used to train the first candidate neural network to obtain the updated M target weights.

[0386] Based on the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; wherein, the first block corresponding to the largest N target weights among the updated M target weights is used to perform the operation corresponding to the first block based on the output of the second block, and N is less than M;

[0387] The second neural network is trained to obtain the first target neural network.

[0388] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 19 , Figure 19 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1900 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, a monitoring data processing device, etc., and is not limited thereto. The execution device 1900 can be used to implement... Figures 4 to 14 The corresponding embodiment describes a method for determining the neural network structure. Specifically, the execution device 1900 includes: a receiver 1901, a transmitter 1902, a processor 1903, and a memory 1904 (wherein the execution device 1900 may have one or more processors 1903). Figure 19 (Taking a processor as an example), processor 1903 may include application processor 19031 and communication processor 19032. In some embodiments of this application, receiver 1901, transmitter 1902, processor 1903 and memory 1904 may be connected via a bus or other means.

[0389] The memory 1904 can include read-only memory and random access memory, and provides instructions and data to the processor 1903. A portion of the memory 1904 can also include non-volatile random access memory (NVRAM). The memory 1904 stores processor and operating instructions, executable modules, or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions can include various operating instructions for implementing various operations.

[0390] The processor 1903 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, all the buses are referred to as a bus system in the figure.

[0391] The method disclosed in the above embodiments of the present application can be applied in the processor 1903 or implemented by the processor 1903. The processor 1903 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 1903. The processor 1903 mentioned above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1903 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1904, and the processor 1903 reads the information in the memory 1904, and combines the hardware to complete the steps of the above method.

[0392] The receiver 1901 can be used to receive input digital or character information, and generate signal input related to relevant settings and function control of the execution device. The transmitter 1902 can be used to output digital or character information through the first interface; the transmitter 1902 can also be used to send instructions to the disk group through the first interface to modify data in the disk group; the transmitter 1902 can also include a display device such as a display screen.

[0393] The embodiment of the present application also provides a training device, please refer to Figure 20 , Figure 20 is a structural schematic diagram of the training device provided by the embodiment of the present application, and the training device 2000 can be deployed with Figures 15 to 17 The neural network structure determination apparatus described in the corresponding embodiment is used to realize Figures 15 to 17 The function of the neural network structure determination apparatus described in the corresponding embodiment, specifically, the training device 2000 is realized by one or more servers, and the training device 2000 can be greatly different due to different configurations or performances, and can include one or more central processing units (central processing units, CPU) 2020 (for example, one or more processors) and a memory 2032, one or more storage media 2030 (for example, one or more mass storage devices) for storing application programs 2042 or data 2044. Wherein, the memory 2032 and the storage medium 2030 can be temporary storage or persistent storage. The program stored in the storage medium 2030 can include one or more modules (not shown in the figure), each module can include a series of instruction operations in the training device. Further, the central processing unit 2020 can be configured to communicate with the storage medium 2030, and execute a series of instruction operations in the storage medium 2030 on the training device 2000.

[0394] The training device 2000 can also include one or more power supplies 2026, one or more wired or wireless network interfaces 2050, one or more input and output interfaces 2058; or one or more operating systems 2041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0395] In the embodiment of the present application, the central processing unit 2020 is used to execute the steps related to the neural network structure determination method described in the above embodiment.

[0396] The embodiment of the present application also provides a computer program product including code, when the code runs on a computer, so that the computer executes the steps performed by the execution device as described above, or so that the computer executes the steps performed by the training device as described above.

[0397] The embodiment of the present application also provides a computer readable storage medium, which stores a program for signal processing, and when the program runs on a computer, the computer executes the steps performed by the foregoing execution device, or the computer executes the steps performed by the foregoing training device.

[0398] The execution device, the training device or the terminal device provided by the embodiment of the present application can be a chip, which includes a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the foregoing embodiments, or the chip in the training device executes the data processing method described in the foregoing embodiments. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0399] Specifically, refer to Figure 21 , Figure 21 A structural diagram of the chip provided by the embodiment of the present application is shown in FIG. 2. The chip can be a neural network processor NPU 2100, which is mounted on a host CPU as a coprocessor and is assigned tasks by the host CPU. The core part of the NPU is an operation circuit 2103, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 2104.

[0400] In some implementations, the operation circuit 2103 internally includes a plurality of processing units (Process Engine, PE). In some implementations, the operation circuit 2103 is a two-dimensional systolic array. The operation circuit 2103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2103 is a general matrix processor.

[0401] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the data of matrix B from the weight memory 2102 and caches it on each PE of the operation circuit. The operation circuit takes the data of matrix A from the input memory 2101 and performs matrix operation with matrix B to obtain a partial result or a final result of the matrix, which is saved in the accumulator 2108.

[0402] The unified memory 2106 is used to store input data and output data. The weight data is transferred to the weight memory 2102 through the DMAC 2105. The input data is also transferred to the unified memory 2106 through the DMAC.

[0403] The BIU (Bus Interface Unit) 2110 is used for the interaction between the AXI bus and the DMAC and the IFB (Instruction Fetch Buffer) 2109.

[0404] The BIU (Bus Interface Unit) 2110 is used for the interaction between the AXI bus and the DMAC and the IFB (Instruction Fetch Buffer) 2109.

[0405] The DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2106, or to transfer weight data to the weight memory 2102, or to transfer input data to the input memory 2101.

[0406] The vector calculation unit 2107 includes a plurality of operation processing units, which further process the output of the operation circuit 2103 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0407] In some implementations, the vector computation unit 2107 can store the processed output vector to the unified memory 2106. For example, the vector computation unit 2107 can apply a linear function; or, a non-linear function to the output of the arithmetic circuit 2103, such as linear interpolation to the feature planes extracted by a convolution layer, and further such as vector accumulation to generate activation values. In some implementations, the vector computation unit 2107 generates normalized values, pixel-wise summed values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 2103, such as for use in a subsequent layer in a neural network.

[0408] The controller 2104 is connected to an instruction fetch buffer 2109 for storing instructions used by the controller 2104;

[0409] The unified memory 2106, the input memory 2101, the weight memory 2102, and the instruction fetch buffer 2109 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0410] Any of the processors mentioned above can be a general purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.

[0411] It should be noted that the apparatus examples described above are merely illustrative, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0412] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, of course, also can be implemented by special hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, the functions completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, the software program implementation is the better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the application.

[0413] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product.

[0414] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A method for determining the structure of a neural network, characterized in that, The method includes: An initial neural network to be trained is obtained. The initial neural network includes M first structural blocks and a second block. The second block is connected to each first block, and each first block corresponds to a target weight. The second block is used to perform the corresponding operation based on the M first outputs. The M first outputs are obtained by multiplying the output of each first block with the corresponding target weight. The target weight is a trainable weight, and M is an integer greater than 1. Acquire training data, which includes one of the following: image data, text data, and speech data; Based on the data to be trained, the initial neural network is trained to obtain the updated M target weights; Based on the updated M target weights, the connection relationship between the second block and the M first blocks in the initial neural network is updated to obtain a first neural network; wherein, the second block in the first neural network is used to perform the operation corresponding to the second block based on the outputs of the N first blocks corresponding to the largest N target weights among the updated M target weights, where N is less than M; The first neural network is used for image processing, text processing, or speech processing.

2. The method according to claim 1, characterized in that, The M first blocks and the second block in the initial neural network are sequentially connected, and the second block is the endpoint of the sequential connection. The M first blocks in the initial neural network include a target block, which is connected to the second block in the sequential connection. If the updated target weight corresponding to the target block does not belong to the largest N target weights among the updated M target weights, the second block in the first neural network is also used to perform the operation corresponding to the second block based on the output of the target block.

3. The method according to claim 2, characterized in that, The value of N is 1.

4. The method according to any one of claims 1 to 3, characterized in that, The step of training the initial neural network to obtain the updated M target weights includes: The initial neural network is trained for a first preset number of iterations to obtain the updated M target weights.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The first neural network is trained until the data processing accuracy of the first neural network meets the preset conditions or the number of iterations of model training reaches the second preset number of iterations, so as to obtain the second neural network.

6. The method according to any one of claims 1 to 3, characterized in that, The number of input and output channels of each of the M first blocks is the same as the number of input and output channels of the second block.

7. The method according to any one of claims 1 to 3, characterized in that, The second block in the initial neural network is used to perform the operation corresponding to the second block based on the summation of the M first outputs; The second block in the first neural network is used to perform the operation corresponding to the second block based on the sum of the outputs of the first block corresponding to the largest N target weights among the updated M target weights.

8. A method for determining the structure of a neural network, characterized in that, The method includes: An initial neural network to be trained is obtained. The initial neural network includes M first structural blocks and a second block. The second block is connected to each first block, and each first block corresponds to a target weight. Each first block is used to perform the operation corresponding to the first block according to the corresponding first output. The first output corresponding to each first block is obtained by multiplying the target weight corresponding to the first block with the output of the second block. The target weight is a trainable weight, and M is an integer greater than 1. Acquire training data, which includes one of the following: image data, text data, and speech data; Based on the data to be trained, the initial neural network is trained to obtain the updated M target weights; Based on the updated M target weights, the connection relationship between the second block and the M first blocks in the initial neural network is updated to obtain the first neural network; wherein, the first block corresponding to the largest N target weights among the updated M target weights is used to perform the operation corresponding to the first block based on the output of the second block, and N is less than M; The first neural network is used for image processing, text processing, or speech processing.

9. The method according to claim 8, characterized in that, The second block in the initial neural network is sequentially connected to the M first blocks, and the second block is the starting point of the sequential connection. The M first blocks in the initial neural network include a target block, which is connected to the second block in the sequential connection. If the updated target weight corresponding to the target block does not belong to the largest N target weights among the updated M target weights, the target block in the first neural network is also used to perform the operation corresponding to the target block based on the output of the second block.

10. The method according to claim 9, characterized in that, The value of N is 1.

11. The method according to any one of claims 8 to 10, characterized in that, The step of training the initial neural network to obtain the updated M target weights includes: The initial neural network is trained for a first preset number of iterations to obtain the updated M target weights.

12. The method according to any one of claims 8 to 10, characterized in that, The method further includes: The first neural network is trained until the data processing accuracy of the first neural network meets the preset conditions or the number of iterations of model training reaches the second preset number of iterations, so as to obtain the second neural network.

13. The method according to any one of claims 8 to 10, characterized in that, The number of input and output channels of each of the M first blocks is the same as the number of input and output channels of the second block.

14. A method for determining the structure of a neural network, characterized in that, The method includes: Multiple target codes are obtained, each target code is used to indicate a candidate neural network, the multiple target codes include a first target code and multiple first codes, the first code is used to indicate a first neural network; The first neural network is trained to obtain the data processing accuracy of the first neural network; Based on the difference between the first target code and the plurality of first codes and the data processing accuracy of the first neural network, the data processing accuracy of the candidate neural network indicated by each first code is determined; Obtain the first candidate neural network with the highest data processing accuracy among the candidate neural networks of the plurality of target encoding indicators; Acquire training data, which includes one of the following: image data, text data, and speech data; Based on the data to be trained, the first candidate neural network is trained to obtain a first target neural network, which is used for image processing, text processing, or speech processing.

15. The method according to claim 14, characterized in that, The method further includes: The data processing accuracy of the first target neural network is obtained, and the plurality of target codes include a second target code, which is used to indicate the first target neural network; Based on the difference between the second target code and the codes other than the second target code among the plurality of target codes, and the data processing accuracy of the first target neural network, the data processing accuracy of the candidate neural network indicated by each of the plurality of target codes other than the second target code is determined; Based on the data processing accuracy of the candidate neural networks indicated by the multiple target codes, the second candidate neural network with the highest data processing accuracy is determined, and the second candidate neural network is trained to obtain the second target neural network.

16. The method according to claim 14, characterized in that, Each target is encoded to indicate at least one of the following structural features of a candidate neural network: The types of computational units included in the candidate neural network, the number of computational units included in the candidate neural network, and the number of input and output feature channels of the computational units included in the candidate neural network.

17. The method according to any one of claims 14 to 16, characterized in that, The method further includes: Multiple codes are clustered to obtain multiple code sets, each code set corresponding to a cluster category, the multiple code sets including a target code set, and the target code set including the multiple target codes.

18. The method according to claim 17, characterized in that, The first target code is the cluster center of the target code set.

19. The method according to any one of claims 14 to 16, characterized in that, Each target encoding indicates a candidate neural network that satisfies at least one of the following conditions: The computational cost required to run each candidate neural network for a target encoding indication is less than a first preset value; The number of weights included in the candidate neural network indicated by each target encoding is less than a second preset value; and, The running speed is higher than the third preset value when running the candidate neural network for each target encoding indication.

20. The method according to any one of claims 14 to 16, characterized in that, The first candidate neural network includes M first structural blocks and a second block. The second block is connected to each first block, and each first block corresponds to a target weight. The second block is used to perform the corresponding operation based on the M first outputs. The M first outputs are obtained by multiplying the output of each first block with the corresponding target weight. The target weight is a trainable weight, and M is an integer greater than 1. The step of training the first candidate neural network to obtain the first target neural network includes: The first candidate neural network is trained to obtain the updated M target weights; Based on the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; wherein, the second block in the second neural network is used to perform the operation corresponding to the second block based on the output of the first block corresponding to the largest N target weights among the updated M target weights, where N is less than M; The second neural network is trained to obtain the first target neural network.

21. The method according to any one of claims 14 to 16, characterized in that, The first candidate neural network includes M first structural blocks and a second block. The second block is connected to each first block, and each first block corresponds to a target weight. Each first block is used to perform the operation corresponding to the first block according to the corresponding first output. The first output corresponding to each first block is obtained by multiplying the target weight corresponding to the first block with the output of the second block. The target weight is a trainable weight, and M is an integer greater than 1. The step of training the first candidate neural network to obtain the first target neural network includes: The first candidate neural network is trained to obtain the updated M target weights; Based on the updated M target weights, the connection relationship between the second block and the M first blocks in the first candidate neural network is updated to obtain a second neural network; wherein, the first block corresponding to the largest N target weights among the updated M target weights is used to perform the operation corresponding to the first block based on the output of the second block, and N is less than M; The second neural network is trained to obtain the first target neural network.

22. A neural network structure determination device, characterized in that, It includes a storage medium, a processing circuit, and a bus system; wherein the storage medium is used to store instructions, and the processing circuit is used to execute the instructions in the memory to perform the steps of the method according to any one of claims 1 to 21.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 21.

24. A computer program product, characterized in that, The computer program product includes code that, when executed, performs the steps of the method according to any one of claims 1 to 21.

Images