Neural network optimization method, device, electronic device and storage medium

Abstract

Disclosed are a neural network optimization method, device, electronic device and storage medium, the method comprising mapping transformation operations in an initial neural network to nodes; connecting each pair of nodes in each node corresponding to the initial neural network through an edge to obtain a complete graph, wherein the edge has an edge weight; optimizing the edge weight of each edge of the complete graph for a target task; and obtaining a target neural network structure corresponding to the target task based on the optimized edge weight.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and in particular to a neural network optimization method, device, electronic device and storage medium. Background Art

[0002] Deep learning has successfully transformed feature engineering from manual design to automatic design, which means that the features of extracted samples can be obtained in an optimized way. Finding more effective neural network structures has gradually become an important and practical direction.

[0003] Currently, most networks are constructed by manually designing modules and stacking them, which limits the space for optimization and makes it difficult to determine structures suitable for different tasks. How to efficiently obtain network structures suitable for different tasks remains a problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure provides a method for optimizing a neural network.

[0005] According to one aspect of the present disclosure, a method for optimizing a neural network is proposed, the method comprising: mapping transformation operations in an initial neural network to nodes; connecting each pair of nodes in each node corresponding to the initial neural network through edges to obtain a complete graph, wherein the edges have edge weights; optimizing the edge weights of each edge of the complete graph for a target task; and obtaining a target neural network structure corresponding to the target task based on the optimized edge weights.

[0006] In combination with any implementation manner provided in the present disclosure, the transformation operations of each network layer include one or more of summation, convolution, normalization, and activation.

[0007] In combination with any embodiment provided by the present disclosure, for each node, the features of the preceding node are aggregated and input to the node through at least one input edge, and the input edge is an edge connected to the input end of the node; the features generated by the node are output to the succeeding node through at least one output edge, and the output edge is an edge connected to the output end of the node.

[0008] In combination with any embodiment provided in the present disclosure, the input node in the node is obtained by mapping the input in the initial neural network, and the input node is a node without a preceding node; the output node in the node is obtained by mapping the output in the initial neural network, and the output node is a node without a succeeding node.

[0009] In combination with any embodiment provided in the present disclosure, the weights corresponding to the nodes of each order in the initial neural network are stored in an adjacency matrix, each row of the adjacency matrix represents the weight of the input edge of the node, and each column of the adjacency matrix represents the weight of the output edge of the node.

[0010] In combination with any embodiment provided by the present disclosure, the optimizing the edge weights of each edge in the complete graph includes: inputting an input sample in a training set into a neural network corresponding to the complete graph to obtain a first output, wherein the training set also includes an output sample corresponding to the input sample; obtaining a gradient of a network loss function with respect to an output feature of the node, wherein the network loss function indicates a loss between the first output of the complete graph and the output sample; obtaining a gradient of a transformation operation of the node with respect to an aggregate feature, wherein the aggregate feature is a result of a weighted summation of multiple input features of the node; and updating the edge weights of each edge corresponding to the node according to the gradient of the network loss function with respect to the output feature of the node and the gradient of the transformation operation of the node with respect to the aggregate feature.

[0011] In combination with any embodiment provided by the present disclosure, the network loss also includes a regularization loss; the optimization of the edge weights of each edge in the complete graph includes: obtaining the gradient of the regularization loss with respect to the output of the node; and updating the edge weights of each edge corresponding to the node according to the gradient of the regularization loss with respect to the output of the node.

[0012] In combination with any implementation manner provided in the present disclosure, the method further includes: updating the edge weights of each edge corresponding to the node according to the in-degree of the node, where the in-degree is the number of input edges of the node.

[0013] In combination with any implementation manner provided in the present disclosure, the method further includes: optimizing the network parameters of the node while optimizing the edge weights of each edge.

[0014] According to one aspect of the present disclosure, a neural network optimization device is proposed, which includes: a mapping unit for mapping transformation operations in an initial neural network to nodes; a connecting unit for connecting each pair of nodes corresponding to the initial neural network through edges to obtain a complete graph, wherein the edges have edge weights; an optimizing unit for optimizing the edge weights of each edge of the complete graph for a target task; and an obtaining unit for obtaining a target neural network structure corresponding to the target task based on the optimized edge weights.

[0015] In combination with any implementation manner provided in the present disclosure, the transformation operations of each network layer include one or more of summation, convolution, normalization, and activation.

[0016] In combination with any embodiment provided by the present disclosure, for each node, the features of the preceding node are aggregated and input to the node through at least one input edge, and the input edge is an edge connected to the input end of the node; the features generated by the node are output to the succeeding node through at least one output edge, and the output edge is an edge connected to the output end of the node.

[0017] In combination with any embodiment provided in the present disclosure, the input node in the node is obtained by mapping the input in the initial neural network, and the input node is a node without a preceding node; the output node in the node is obtained by mapping the output in the initial neural network, and the output node is a node without a succeeding node.

[0018] In combination with any embodiment provided in the present disclosure, the weights corresponding to the nodes of each order in the initial neural network are stored in an adjacency matrix, each row of the adjacency matrix represents the weight of the input edge of the node, and each column of the adjacency matrix represents the weight of the output edge of the node.

[0019] In combination with any embodiment provided by the present disclosure, the optimization unit is specifically used to: input an input sample in a training set into the complete graph to obtain a first output, wherein the training set also includes an output sample corresponding to the input sample; obtain a gradient of a network loss function for an output feature of the node, wherein the network loss function indicates a loss between the first output of the complete graph and the output sample; obtain a gradient of a transformation operation of the node for an aggregate feature, wherein the aggregate feature is a result of a weighted summation of multiple input features of the node; and update the edge weights of each edge corresponding to the node according to the gradient of the network loss function for the output feature of the node and the gradient of the transformation operation of the node for the aggregate feature.

[0020] In combination with any embodiment provided by the present disclosure, the network loss also includes regularization, and the optimization unit is specifically used to: obtain the gradient of the regularization loss with respect to the output of the node; and update the edge weights of each edge corresponding to the node according to the gradient of the regularization loss with respect to the output of the node.

[0021] In combination with any embodiment provided in the present disclosure, the device also includes an adaptive constraint unit, which is used to update the edge weights of each edge corresponding to the node according to the in-degree of the node, where the in-degree is the number of input edges of the node.

[0022] In combination with any of the embodiments provided in the present disclosure, the device further includes a network parameter optimization unit, which is used to optimize the network parameters of the node while optimizing the edge weights of each edge.

[0023] According to one aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions to implement a method of any embodiment provided by the present disclosure.

[0024] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by an electronic device, a method of any embodiment provided by the present disclosure is implemented.

[0025] The neural network optimization method, device and electronic device proposed in any embodiment of the present disclosure maps the transformation operations of each network layer in the initial neural network to nodes, and connects each pair of nodes in each node corresponding to the initial neural network through edges to obtain a complete graph, and optimizes the edge weights of each edge in the complete graph for a target task, and obtains a target neural network structure corresponding to the target task according to the optimized edge weights. The neural network is represented from the perspective of topological connection, and an optimization space with edge connections between any pair of nodes is constructed, and the edge weights of each edge are optimized in the search space. The optimization of the neural network structure is converted into a process of finding an optimal subgraph under the complete graph, and a topological connection suitable for the target task can be obtained without increasing the amount of additional calculations and parameters, thereby improving the efficiency and effect of the neural network structure design. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A flowchart of a neural network optimization method provided by at least one embodiment of the present disclosure;

[0028] Figure 2 A schematic diagram of a node transformation operation in a neural network optimization method provided by at least one embodiment of the present disclosure;

[0029] Figure 3 An example diagram of an adjacency matrix in a neural network optimization method provided by at least one embodiment of the present disclosure;

[0030] Figure 4A A schematic diagram of converting an initial neural network of a residual structure provided by at least one embodiment of the present disclosure into a topological graph, Figure 4Bis an example graph of the complete graph obtained based on the topological graph;

[0031] Figure 5 A schematic diagram of the structure of a neural network optimization device provided by at least one embodiment of the present disclosure;

[0032] Figure 6 A structural diagram of an electronic device provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this disclosure.

[0034] Figure 1 A flowchart of a neural network optimization method provided by at least one embodiment of the present disclosure. Figure 1 As shown, the method includes steps 101 to 104.

[0035] In step 101, transformations in the initial neural network are mapped to nodes.

[0036] The initial neural network may be an existing neural network or a newly constructed neural network; the structure of the initial neural network may include multiple directly stacked convolutional layers to obtain multiple parallel modules, and may also include multiple residual modules, which is not limited in the present disclosure.

[0037] The transformation operation in the initial neural network includes one or more of combination (e.g., summation), convolution, normalization, and activation. The transformation operation is mapped to a node, that is, the transformation operation is represented by a node. Those skilled in the art should understand that the node can also represent other transformation operations, not limited to the above.

[0038] In step 102, each pair of nodes corresponding to the initial neural network is connected through an edge to obtain a complete graph.

[0039] In the embodiment of the present disclosure, the connection between nodes can be mapped to an edge, that is, the transmission of the signal is represented by an edge, wherein the edge has an edge weight, and the edge weight represents the importance of the connection corresponding to the edge.

[0040] By mapping the transformation operations in the initial neural network to nodes and mapping the connections between nodes to edges, the neural network can be represented as a directed acyclic graph to display the neural network structure from a topological perspective.

[0041] For the directed acyclic graph of the initial neural network, by directly connecting all nodes and directly connecting all nodes to input nodes and output nodes, a neural network structure in which any pair of nodes has a connection relationship can be obtained, and the neural network structure represented by a directed acyclic graph can be called a complete graph. Since all nodes in the complete graph are directly connected to input nodes and output nodes, it is guaranteed that the original input features can be obtained in the forward calculation process, and the original input features are used for gradient update in the reverse calculation process, ensuring that the entire optimization process is more stable.

[0042] Since any pair of nodes in the complete graph is connected by edges, all connection possibilities are provided, which is suitable as the search space (optimization space) of the network structure.

[0043] In step 103, the edge weights of the edges of the complete graph are optimized according to the target task.

[0044] In the search space, by learning the weights of the edges, the connections can be optimized. By learning for different target tasks, the edge weights suitable for the target task can be obtained, that is, the complete graph optimization results suitable for the target task can be obtained.

[0045] In step 104, a target neural network structure corresponding to the target task is obtained according to the optimized edge weights.

[0046] After optimizing the edge weights of each edge according to the target task, an edge weight of zero indicates that there is no connection between the corresponding nodes, and a non-zero edge weight indicates that there is a connection between the corresponding nodes, and the larger the edge weight, the more important the corresponding connection. The connection result of the neural network is determined based on the complete graph optimization result applicable to the target task.

[0047] In the embodiments of the present disclosure, by mapping the transformation operations of each network layer in the initial neural network to nodes, and connecting each pair of nodes in each node corresponding to the initial neural network through edges, a complete graph is obtained, and the edge weights of each edge in the complete graph are optimized for the target task. According to the optimized edge weights, the target neural network structure corresponding to the target task is obtained. By representing the neural network from the perspective of topological connection, and by constructing an optimization space where there are edge connections between any pair of nodes, and optimizing the edge weights of each edge in the search space, the optimization of the neural network structure is transformed into a process of finding the optimal subgraph under the complete graph, which can obtain the topological connection suitable for the target task without increasing the additional computational amount and the number of parameters, and improves the efficiency and effect of the neural network structure design.

[0048] In the embodiments of the present disclosure, the network structure represented from the perspective of topological connection can be represented as ɡ=(Ν,ε), where Ν represents the set of nodes, and ε represents the set of edges. Each node n in the set of nodes i ∈Ν implements the transformation operation o i , and the parameter of the transformation operation is w i , where i represents the topological serial number of the node. The edge e ji =(j,i,α ji )∈ε represents the transfer feature from node j to node i, and the importance degree of the connection is determined by the edge weight a ji . Where j < i, that is, the feature flows from the node with a smaller serial number to the node with a larger serial number.

[0049] In one example, the input of the initial neural network is mapped to the first node in the set of nodes Ν; the output of the initial neural network is mapped to the last node in the set of nodes.

[0050] In some embodiments, for each node, the features of the pre-order nodes are aggregated through at least one input edge and input to the node, and the input edge is the edge connected to the input end of the node; the features generated by the node are output to the post-order nodes through at least one output edge, and the output edge is the edge connected to the output end of the node. Wherein, the pre-order node refers to the node with a serial number smaller than the serial number of this node, and the post-order node refers to the node with a serial number larger than the serial number of this node.

[0051] Figure 2 Shows a schematic diagram of the node transformation operation proposed in at least one embodiment of the present disclosure. As Figure 2 shown, the features x j ,x k ,x m ,xo transferred by the pre-order nodes through the edges are aggregated respectively to obtain the aggregated feature x' iAmong them, the edge weights corresponding to each edge are a ji ,a ki ,a mi ,a oi Node n i For the aggregate feature x' i Perform feature transformation to get output tensor x i ; The output tensor x i Transmit to the next node n through the outgoing edge q ,n p .

[0052] Output tensor x i It can be expressed by formula (1):

[0053] x i =o i (x′ i ;w i ),in

[0054] In formula (1), o i represents the transformation operation, x' i represents the aggregate feature, w i represents the parameters of the transform unit that performs the transform operation, x j is the feature passed by the previous node, a ji represents the edge e connecting the jth node and the ith node ji The corresponding edge weight.

[0055] In one embodiment, the weights corresponding to the nodes of each order in the initial neural network are stored in an adjacency matrix. Figure 3 As shown, each row of the adjacency matrix represents the edge weight of the input edge of the node, and each column of the adjacency matrix represents the edge weight of the output edge of the node. For nodes without edge connections, the corresponding edge weight is zero. Among them, the size of the row (with non-zero edge weight) is called the in-degree of the node, and the size of the column (with non-zero edge weight) is called the out-degree of the node.

[0056] For a k-order neural network, k initialized directed acyclic graphs can be cascaded, that is, each directed acyclic graph is linked to the previous or next directed acyclic graph through the input or output of the node.

[0057] The following describes in detail the process of determining the optimization space of the initial neural network by transforming the initial neural network of the residual structure.

[0058] Figure 4ASchematic diagram of the initial neural network conversion topology of the residual structure, which contains three residual connection units, each of which can be expressed as Among them, x represents the identity mapping direct connection, Represents the residual map. Usually, the residual connection unit is composed of several repeated network layers, and the number of repetitions can be called the interval. Figure 4A The spacing of the residual structure in is 2.

[0059] In one example, you can Figure 4A The input x of the initial neural network in the left part is mapped to the input node 210 shown in the right part; the output y of the initial neural network is mapped to the output node 211 shown in the right part.

[0060] For the transformation operation of the initial neural network, the summation operation performed by the summation unit 201 in the initial neural network and the convolution operation performed by the network layer 202 can be mapped to a node, such as node 213; and the convolution operation performed by the network layer 203 is mapped to node 214. Similar transformations are also performed on other residual connection units in the initial neural network, so that Figure 4A The residual structure shown on the right side is represented from a topological perspective.

[0061] exist Figure 4A In the initial neural network shown on the left, the features input to the network layer 202 actually include the output features 1 and 2 of other network layers, as well as the input feature 3. However, from the structure diagram shown on the left, it is not possible to directly obtain the association between the network layer 202 and the network layers or inputs corresponding to these features; while in the directed acyclic graph shown on the right, it can be seen that features 1, 2, and 3 are transmitted to node 213 through corresponding edges, which is equivalent to expanding the structure diagram on the left for representation, that is, showing the structure of the original neural network from the perspective of topological connection.

[0062] In one example, by directly connecting all nodes to inputs and outputs, and connecting any pair of nodes, a complete graph is obtained, that is, a complete graph corresponding to the initial neural network is obtained, such as Figure 4B As shown. For a complete graph with N nodes, the search space consists of 2 N(N-1) / 2 possible topological structures. For an initial neural network with order k, the total search space can be expressed as Compared with neuron-based and block-based methods, the neural network optimization method proposed in the embodiment of the present disclosure provides a broader search space.

[0063] In at least one embodiment of the present disclosure, a differentiable method is proposed to optimize topological connections by learning edge weights.

[0064] In one example, the edge weight of each edge in the complete graph may be optimized by the following method.

[0065] First, input the input sample x in the training set into the complete graph corresponding to the initial neural network to obtain the first output Based on the difference between the first output and the output sample, the network loss function for mapping the input sample x can be determined

[0066] In the embodiment of the present disclosure, the optimization objective can be expressed as:

[0067]

[0068] In formula (2), (x, y) is a sample pair in the training set, x is the input sample, y is the output sample, W is the network parameter, a is the edge weight, is the mapping for the input sample x, is the network loss function for mapping the input sample x.

[0069] Next, the gradient of the network loss function with respect to the output feature of the node is obtained, and this gradient can be referred to as the first gradient. The output feature of the node is, for example, Figure 2 x in i , the gradient of the network loss function with respect to xi can be obtained by taking the partial derivative of the network loss function.

[0070] Afterwards, the gradient of the transformation operation of the node for the aggregate feature is obtained, and this gradient can be called the second gradient. The aggregate feature is the result of the weighted summation of multiple input features of the node, for example Figure 2 x' i , the transformation operation of the node is, for example, Figure 2 o in i .

[0071] Finally, the edge weights of each edge corresponding to the node are updated according to the gradient of the network loss function for the output feature of the node and the gradient of the node's transformation operation for the aggregate feature, that is, the edge weights of each edge are updated according to the first gradient and the second gradient of each node.

[0072] For edge weight a in the complete graph ji The update of can be expressed by formula (3):

[0073]

[0074] While optimizing the edge weights of each edge, the network parameters of the node can also be optimized. For example, in the network parameter W, the network parameter w of the i-th node i The optimization can be expressed by formula (4):

[0075]

[0076] In formulas (3) and (4), η represents the learning rate and ⊙ represents the entrywise product.

[0077] Since the features generated by different layers show different semantics and contribute differently to subsequent layers, the importance of connections is different. In order to focus on important connections, the embodiment of the present disclosure proposes a method for adding sparse constraints.

[0078] First, the gradient of the regularization loss with respect to the output of the node is obtained.

[0079] The regularization loss can be expressed as L1=||α||1, where the regularization loss L1 is the output x of the node i The gradient of can be obtained by taking the partial derivative of the regularization loss, and this gradient can be called the third gradient.

[0080] When the regularization loss is added, the total loss function of the optimization method proposed in the embodiment of the present disclosure is It can be expressed as:

[0081]

[0082] In formula (5), λ is a hyperparameter that balances the sparsity.

[0083] Afterwards, the edge weights of each edge corresponding to the node can be updated according to the gradient of the regularization loss with respect to the output of the node, that is, the edge weights can be updated according to the third gradient.

[0084] In an example, the edge weight may be updated jointly according to the first gradient, the second gradient, and the third gradient.

[0085] In the embodiments of the present disclosure, by utilizing the distribution of sparsely constrained topological connections, the network structure pays more attention to important connections, has better generalization capabilities, and can achieve better results in image classification and target detection tasks.

[0086] Under the condition that all edge weights in the complete graph are uniformly sparsely constrained, the edge weight a ji The update of can be expressed by formula (6):

[0087]

[0088] In one example, the weights in the complete graph may be constrained in relation to the in-degree of the node, for example, a logarithmically related constraint may be imposed. The in-degree refers to the number of input edges connected to the node. For example, a larger sparse constraint may be imposed on a node with a larger in-degree, and / or a smaller sparse constraint may be imposed on a node with a smaller in-degree. For nodes with relatively few input edges, by imposing a relatively small sparse constraint, the smoothness of the signal flow may be ensured to avoid obstruction. Those skilled in the art will appreciate that other related forms of constraints may also be imposed, and the present disclosure does not limit this.

[0089] In the case of logarithmic constraints, the edge weight a ji The update of can be expressed by formula (7):

[0090]

[0091] In the disclosed embodiment, the edge weight may be a continuous parameter, thereby ensuring the continuity between training and testing.

[0092] Figure 5 A schematic diagram of a neural network optimization device provided by at least one embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the device includes a mapping unit 51, which is used to map the transformation operations in the initial neural network to nodes; a connecting unit 502, which is used to connect each pair of nodes in each node corresponding to the initial neural network through an edge to obtain a complete graph, and the edge has an edge weight; an optimization unit 503, which is used to optimize the edge weight of each edge of the complete graph according to the target task; and an obtaining unit 504, which is used to obtain the target neural network structure corresponding to the target task according to the optimized edge weight.

[0093] In some embodiments, the transformation operations of each network layer include one or more of summation, convolution, normalization, and activation.

[0094] In some embodiments, for each node, features of the preceding node are aggregated and input to the node through at least one input edge, where the input edge is an edge connected to the input end of the node; features generated by the node are output to the succeeding node through at least one output edge, where the output edge is an edge connected to the output end of the node.

[0095] In some embodiments, the input nodes in the nodes are obtained by mapping the inputs in the initial neural network, and the input nodes are nodes without preceding nodes; the output nodes in the nodes are obtained by mapping the outputs in the initial neural network, and the output nodes are nodes without succeeding nodes.

[0096] In some embodiments, the weights corresponding to nodes of each order in the initial neural network are stored in an adjacency matrix, each row of the adjacency matrix represents the weight of the input edge of the node, and each column of the adjacency matrix represents the weight of the output edge of the node.

[0097] In some embodiments, the optimization unit is specifically used to: input an input sample in a training set into the complete graph to obtain a first output, wherein the training set also includes an output sample corresponding to the input sample; obtain a gradient of a network loss function with respect to an output feature of the node, wherein the network loss function indicates a loss between the first output of the complete graph and the output sample; obtain a gradient of a transformation operation of the node with respect to an aggregate feature, wherein the aggregate feature is a result of a weighted summation of multiple input features of the node; and update the edge weights of each edge corresponding to the node according to the gradient of the network loss function with respect to the output feature of the node and the gradient of the transformation operation of the node with respect to the aggregate feature.

[0098] In some embodiments, the network loss also includes regularization, and the optimization unit is specifically used to: obtain the gradient of the regularization loss with respect to the output of the node; and update the edge weights of each edge corresponding to the node according to the gradient of the regularization loss with respect to the output of the node.

[0099] In some embodiments, the device further comprises an adaptive constraint unit for updating edge weights of each edge corresponding to the node according to the in-degree of the node, where the in-degree is the number of input edges of the node.

[0100] In some embodiments, the device further comprises a network parameter optimization unit for optimizing the network parameters of the node while optimizing the edge weights of the respective edges.

[0101] Figure 6 An electronic device provided for at least one embodiment of the present disclosure includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to execute the computer instructions to implement the neural network optimization method described in any embodiment of the present specification.

[0102] At least one embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the neural network optimization method described in any embodiment of the present specification.

[0103] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0105] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0107] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0108] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0109] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0110] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0111] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0112] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0113] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for optimizing a neural network, characterized in that: The method comprises: Mapping transformation operations in the initial neural network to nodes; Connecting each pair of nodes in the respective nodes corresponding to the initial neural network through edges to obtain a complete graph, wherein the edges have edge weights; For the target task, input the input samples in the training set to the neural network corresponding to the complete graph to obtain the first output, wherein the training set also includes the output samples corresponding to the input samples; obtain the gradient of the network loss function for the output feature of the node, wherein the network loss function indicates the loss between the first output of the complete graph and the output sample; obtain the gradient of the transformation operation of the node for the aggregate feature, wherein the aggregate feature is the result of the weighted summation of multiple input features of the node; update the edge weights of each edge corresponding to the node according to the gradient of the network loss function for the output feature of the node and the gradient of the transformation operation of the node for the aggregate feature; the target task includes an image classification task; The target neural network structure corresponding to the target task is obtained according to the optimized edge weights.

2. The method according to claim 1, characterized in that The transformation operation includes one or more of summation, convolution, normalization, and activation.

3. The method according to claim 1, characterized in that: For each node, the features of the preceding node are aggregated and input to the node through at least one input edge, where the input edge is an edge connected to the input end of the node; the features generated by the node are output to the succeeding node through at least one output edge, where the output edge is an edge connected to the output end of the node.

4. The method according to claim 3, characterized in that The mapping of the transformation operations in the initial neural network to the nodes includes: Mapping an input of the initial neural network to a first node among the nodes; The output of the initial neural network is mapped to a last node in the nodes.

5. The method according to claim 1, characterized in that The weights corresponding to the nodes of each order in the initial neural network are stored in an adjacency matrix, each row of the adjacency matrix represents the weight of the input edge of the node, and each column of the adjacency matrix represents the weight of the output edge of the node.

6. The method according to claim 1, characterized in that The network loss also includes regularization loss; The step of optimizing the edge weight of each edge in the complete graph includes: Obtaining the gradient of the regularization loss with respect to the output of the node; The edge weights of each edge corresponding to the node are updated according to the gradient of the regularization loss with respect to the output of the node.

7. The method according to claim 6, characterized in that The method further comprises: The edge weights of the edges corresponding to the node are updated according to the in-degree of the node, where the in-degree is the number of input edges of the node.

8. The method according to claim 1, characterized in that The method further includes: optimizing the network parameters of the nodes while optimizing the edge weights of the edges.

9. A neural network optimization device, characterized in that: The device comprises: A mapping unit, used for mapping the transformation operations in the initial neural network to nodes; A connection unit, used for connecting each pair of nodes corresponding to the initial neural network through an edge to obtain a complete graph, wherein the edge has an edge weight; An optimization unit is used to input the input samples in the training set to the neural network corresponding to the complete graph for a target task, and obtain a first output, wherein the training set also includes an output sample corresponding to the input sample; obtain a gradient of a network loss function for an output feature of the node, wherein the network loss function indicates a loss between the first output of the complete graph and the output sample; obtain a gradient of a transformation operation of the node for an aggregate feature, wherein the aggregate feature is a result of a weighted summation of multiple input features of the node; and update an edge weight of each edge corresponding to the node according to the gradient of the network loss function for the output feature of the node and the gradient of the transformation operation of the node for the aggregate feature, wherein the target task includes an image classification task; The obtaining unit is used to obtain the target neural network structure corresponding to the target task according to the optimized edge weights.

10. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the method according to any one of claims 1 to 8 is implemented.

Images