Training method and device of convolutional neural network, storage medium and program product

Abstract

The present disclosure relates to the technical field of computer, and particularly relates to a training method and device of convolutional neural network, a storage medium and a program product. The training method of convolutional neural network comprises the following steps: removing a first connection according to weights of connections between nodes in the convolutional neural network, to obtain a convolutional neural network after connection removal, the first connection corresponding to a first element in a convolution kernel of the convolutional neural network; determining a first score of a node pair in the convolutional neural network after connection removal, in a case where the node pair does not exist connection, wherein the node pair comprises an output layer node and an input layer node located in a first receptive field of the output layer node, and the first receptive field comprises a coverage range of the input layer when the convolution kernel is in a first sliding position; and generating a second connection according to the first scores of each node pair in the convolutional neural network, the second connection corresponding to a second element in the convolution kernel of the convolutional neural network.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method and apparatus, storage medium, and program product for a convolutional neural network. Background Technology

[0002] Convolutional Neural Networks (CNNs) are deep learning models that simulate biological visual perception mechanisms. They achieve hierarchical extraction of spatial features and data modeling through multi-layer convolution and pooling operations. With the development of artificial intelligence technology, CNNs can be applied to an increasing number of fields, such as image recognition, object detection, semantic segmentation, and video analysis.

[0003] Compared to fully connected convolutional neural networks, sparsely connected convolutional neural networks can reduce the risk of overfitting by decreasing the number of neuron connections in the neural network, thereby reducing the computational cost and improving computational efficiency, and thus enabling efficient training and deployment.

[0004] In related technologies, it is impossible to train sparsely connected convolutional neural networks in a targeted manner based on their characteristics, resulting in poor training performance. Summary of the Invention

[0005] In view of this, this disclosure provides a training method, apparatus, storage medium, and program product for a convolutional neural network. During the training process of the neural network, by removing and generating connections corresponding to elements in the convolutional kernel, the network can maintain weight sharing constraints during dynamic sparse training, thereby improving the training effect of the neural network.

[0006] According to a first aspect of this disclosure, a method for training a convolutional neural network is provided, comprising: removing a first connection based on the weights of connections between nodes in the convolutional neural network to obtain a convolutional neural network after connection removal, wherein the first connection corresponds to a first element in the convolutional kernel of the convolutional neural network; for node pairs in the convolutional neural network after connection removal, determining a first score for the node pair when there is no connection in the node pair, wherein the node pair includes an output layer node and an input layer node located within a first receptive field of the output layer node, the first receptive field including the coverage area of ​​the convolutional kernel in the input layer at a first sliding position; and generating a second connection based on the first score of each node pair in the convolutional neural network, wherein the second connection corresponds to a second element in the convolutional kernel of the convolutional neural network.

[0007] In some embodiments, generating a second connection based on a first score of each node pair in the convolutional neural network includes: for each element in the convolutional kernel, determining a probability of generating a connection corresponding to that element based on the sum of the first scores of all node pairs corresponding to that element; determining the second element based on the probabilities corresponding to each element in the convolutional kernel; and generating a second connection corresponding to the second element, the second connection including connections between all node pairs corresponding to the second element.

[0008] In some embodiments, generating a second connection based on a first score of each node pair in the convolutional neural network further includes: for each node pair, determining the receptive field center of the output layer node in the node pair; and determining the element in the convolutional kernel corresponding to the node pair based on the position of the input layer node in the node pair relative to the center.

[0009] In some embodiments, removing a first connection based on the weights of the connections between nodes in the convolutional neural network includes: for each element in the convolutional kernel, determining a probability of removing a connection corresponding to that element based on at least one of the sum of the weights of all connections corresponding to that element and the sum of the relative importance of the weights; determining the first element based on the probabilities corresponding to each element in the convolutional kernel; and removing a first connection corresponding to the first element, wherein the first connection includes connections between all node pairs corresponding to the first element.

[0010] In some embodiments, when the node pair is not connected, determining a first score for the node pair includes: determining a set of paths between the node pairs whose path length is equal to a threshold; and determining a first score for the node pair based on the set of paths.

[0011] In some embodiments, determining the set of paths whose path length between the node pairs is equal to a threshold includes: determining the convolutional kernel sliding range of the path between the node pairs according to the threshold; determining a second receptive field according to the convolutional kernel sliding range, wherein the distance between the center of the second receptive field and the center of the first receptive field is within the convolutional kernel sliding range; and determining the set of paths whose path length between the node pairs is equal to the threshold according to the connection corresponding to the first receptive field and the connection corresponding to the second receptive field.

[0012] In some embodiments, determining the convolution kernel sliding range of the path between the node pairs based on the threshold includes: determining the number of receptive field switching of the path between the node pairs based on the threshold; and determining the convolution kernel sliding range based on the number of receptive field switching and the size of the convolution kernel.

[0013] In some embodiments, determining a first score for the node pair based on the path set includes: determining common neighbor nodes of the node pair based on the path set, the common neighbor nodes including pathway nodes located on any path in the path set; and determining a first score for the node pair based on the number of connections of the common neighbor nodes.

[0014] In some embodiments, determining a first score for a node pair based on the number of connections between the common neighbor nodes includes: determining a first score for the node pair based on the number of internal connections and the number of external connections between the common neighbor nodes, wherein the external connections include connections between the common neighbor nodes and non-common neighbor nodes, and the internal connections include connections between the common neighbor nodes.

[0015] In some embodiments, determining a first score for a node pair based on the number of connections between the common neighbor nodes includes: for each path in the path set, determining a second score for each path based on the number of external connections between the common neighbor nodes on each path, wherein the external connections include connections between the common neighbor nodes and non-common neighbor nodes; and determining a first score for the node pair based on the sum of the second scores of each path in the path set.

[0016] In some embodiments, the training method further includes training the weights of each connection in the convolutional neural network before each removal of a portion of the connections in the convolutional kernel of the convolutional neural network.

[0017] In some embodiments, the input of the convolutional neural network is a first image, and the output is at least one of the following: the category of the first image, an object in the first image, a second image generated from the first image, and text generated from the first image; or the input of the convolutional neural network is first text, and the output is at least one of the following: the category of the first text, an element in the first text, second text generated from the first text, and an image generated from the first text.

[0018] According to a second aspect of this disclosure, a training apparatus for a convolutional neural network is provided, comprising: a connection removal module configured to remove a first connection based on the weights of connections between nodes in the convolutional neural network to obtain a connection-removed convolutional neural network, wherein the first connection corresponds to a first element in the convolutional kernel of the convolutional neural network; a first score determination module configured to determine a first score for a node pair in the connection-removed convolutional neural network when the node pair has no connection, wherein the node pair includes an output layer node and an input layer node located within a first receptive field of the output layer node, the first receptive field including the coverage area of ​​the convolutional kernel in the input layer at a first sliding position; and a connection regeneration module configured to generate a second connection based on the first score of each node pair in the convolutional neural network, wherein the second connection corresponds to a second element in the convolutional kernel of the convolutional neural network.

[0019] According to a third aspect of this disclosure, a training apparatus for a convolutional neural network is provided, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to execute a training method for a convolutional neural network as described in any embodiment of this disclosure based on instructions stored in the at least one memory.

[0020] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement a training method for a convolutional neural network as described in any embodiment of this disclosure.

[0021] According to a fifth aspect of this disclosure, a computer program product is provided that, when run on a computer, causes the computer to implement a training method for a convolutional neural network as described in any embodiment of this disclosure. Attached Figure Description

[0022] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0023] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart illustrating a training method for a convolutional neural network according to some embodiments of the present disclosure; Figure 2 A flowchart illustrating connection removal according to some embodiments of this disclosure is shown; Figure 3 A schematic diagram illustrating connection removal according to some embodiments of the present disclosure is shown; Figure 4A flowchart illustrating the determination of a first score according to some embodiments of the present disclosure is shown; Figure 5 A flowchart illustrating connection regeneration according to some embodiments of the present disclosure is shown; Figure 6 A schematic diagram illustrating the training of a convolutional neural network according to some embodiments of the present disclosure is shown; Figure 7 A block diagram of a training apparatus according to some embodiments of the present disclosure is shown; Figure 8 Block diagrams of training apparatus according to other embodiments of the present disclosure are shown; Figure 9 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0024] It should be understood that the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. Furthermore, the same or similar reference numerals denote the same or similar components. Detailed Implementation

[0025] Various embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The descriptions of the embodiments are merely illustrative and are in no way intended to limit the scope of the disclosure or its application or use. The present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps set forth in these embodiments should be interpreted as merely illustrative and not as limiting.

[0026] The terms “first,” “second,” and similar words used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different parts. Words such as “including” mean that the element preceding the word covers the element listed after the word, and do not exclude the possibility of covering other elements as well.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0028] All terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in a general dictionary, such as a dictionary, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0029] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0030] Traditional training methods cannot effectively train sparsely connected convolutional neural networks based on their characteristics, resulting in poor training outcomes.

[0031] In view of this, this disclosure proposes a training method for convolutional neural networks. During the training process of the neural network, by removing and generating connections corresponding to elements in the convolutional kernel, the network can maintain weight sharing constraints during dynamic sparse training, thereby improving the training effect of the neural network.

[0032] First, combined Figure 1 The training method of the convolutional neural network in this disclosure is described. Figure 1 A flowchart illustrating a training method for a convolutional neural network according to some embodiments of the present disclosure is shown.

[0033] like Figure 1 As shown, the training method for a convolutional neural network may include: Step S1, removing a first connection according to the weights of the connections between nodes in the convolutional neural network to obtain a convolutional neural network after connection removal, wherein the first connection corresponds to a first element in the convolutional kernel of the convolutional neural network; Step S2, for node pairs in the convolutional neural network after connection removal, determining a first score for the node pair when there is no connection in the node pair, wherein the node pair includes an output layer node and an input layer node located within the first receptive field of the output layer node, and the first receptive field includes the coverage area of ​​the convolutional kernel in the input layer at the first sliding position; Step S3, generating a second connection according to the first score of each node pair in the convolutional neural network, wherein the second connection corresponds to a second element in the convolutional kernel of the convolutional neural network.

[0034] Dynamic Sparse Training (DST) is an algorithm that has emerged in recent years. It can be used to train non-fully connected convolutional neural networks. It achieves the target sparsity by dynamically removing and generating links, thereby reducing inference overhead during training.

[0035] In step S1, a portion of the connections in the neural network can be removed based on their weights in order to update the connections in the neural network.

[0036] Removing connections in a neural network refers to eliminating the relationship between two nodes in the neural network. For example, removing the correspondence between an input layer node and an output layer node so that the signal is no longer transmitted to the output layer node through that input layer node.

[0037] Furthermore, in a neural network's weight matrix, nodes corresponding to non-zero elements are considered to be connected, while nodes corresponding to zero elements are considered to be disconnected. Removing a connection can also be understood as resetting the original non-zero weights between nodes to zero.

[0038] The first connection removed in step S1 above can be the connection corresponding to the first element in the convolution kernel. The convolution kernel refers, for example, to the parameter matrix in a convolutional neural network used to slide across the input layer and perform convolution operations. The elements in the convolution kernel can be understood, for example, as the elements in the parameter matrix, corresponding to different positions in the matrix.

[0039] Taking a 3x3 convolution kernel in a single-layer convolutional neural network as an example, the first element mentioned above can be the element at the top left corner of the matrix, i.e., the element at position (1, 1). The first connection corresponding to this first element refers to the connection between each output layer node and the input layer node at position (1, 1) in its receptive field.

[0040] It should be understood that the position and number of the first elements described above are merely exemplary, and elements at other positions in the convolution kernel can also be determined as the first elements, or multiple first elements can be determined. Furthermore, the training method described above is not only applicable to single-layer convolutional neural networks, but also to multi-layer convolutional neural networks.

[0041] Figure 2 A flowchart illustrating connection removal according to some embodiments of this disclosure is shown. Figure 2 As shown, step S1, removing the first connection based on the weights of the connections between nodes in the convolutional neural network, may include: step S11, for each element in the convolutional kernel, determining the probability of removing the connection corresponding to that element based on at least one of the sum of the weights of all connections corresponding to that element and the sum of the relative importance of the weights; step S12, determining the first element based on the probabilities corresponding to each element in the convolutional kernel; step S13, removing the first connection corresponding to the first element, wherein the first connection includes the connections between all node pairs corresponding to the first element.

[0042] In step S11, the weight magnitude or relative importance of the connections corresponding to each element in the convolution kernel can be determined first. The relative importance of the connection weights can be characterized, for example, by the ratio between the weight of that connection and the weights of other surrounding connections.

[0043] As an example, for the element at position (1, 2) in the convolution kernel, the magnitude of the connection weights between each output layer node and the input layer node at position (1, 2) in its receptive field can be summed, or the relative importance of the connection weights can be summed.

[0044] After determining the sum of connection weights or the sum of relative importance of connections for each element in the convolution kernel, the probability of removing the connection corresponding to the element can be determined by comparing the summation results.

[0045] For example, the reciprocal of the sum of the connection weights or the reciprocal of the sum of the relative importance of the connections can be normalized, and the connection removal probability for each element can be determined by means of a multinomial distribution.

[0046] By using the soft sampling method described above, it is possible to avoid the erroneous removal of important connections in the neural network during the early stages of training, thereby improving the training effect of the neural network.

[0047] In step S12, the first element that needs to be removed from the connection can be determined based on the above probability.

[0048] For example, the first element can be determined by random selection based on the probabilities mentioned above. Since these probabilities are determined by the reciprocal of the connection weight or the reciprocal of the connection's relative importance, the higher the weight of a connection, the lower the probability of it being removed.

[0049] By using the above method, unimportant connections in the neural network can be identified and removed, so that important connections can be regenerated in subsequent training processes.

[0050] It should be understood that the above method is merely exemplary. Alternatively, for example, the elements with the lowest sum of connection weights or the lowest sum of relative connection importance can be identified as the first elements that need to be removed from the connection.

[0051] In step S13, all connections corresponding to the determined first element can be removed, thereby ensuring the weight sharing constraint in the convolutional neural network.

[0052] For example, if the first element is determined to be the element at position (1, 2) in the convolution kernel, the connection between all output layer nodes and the input layer nodes at position (1, 2) in their receptive fields can be removed, thereby ensuring the weight sharing constraint in the convolutional neural network, that is, the connection between the input layer nodes and the output layer nodes corresponds to the weight matrix of the convolution kernel.

[0053] In other words, the above-mentioned connection removal can also be understood as resetting the weights at the corresponding positions in the convolution kernel to zero and re-performing the convolution operation to determine the connection between the input layer nodes and the output layer nodes.

[0054] Figure 3 A schematic diagram illustrating connection removal according to some embodiments of the present disclosure is shown. Figure 3 An example is shown where a 2x2 convolution kernel performs a convolution operation on a 3x3 input layer, including input layer nodes 311 to 319 and output layer nodes 321 to 324.

[0055] like Figure 3 As shown, in a convolutional neural network, each output layer node can only establish connections with input layer nodes within its receptive field. In the bipartite diagram corresponding to the above convolution operation, each output layer node has connections with input layer nodes at positions (1, 1) and (2, 1). For example, input layer nodes 311 and 314 are connected to output layer node 321, and input layer nodes 312 and 315 are connected to output layer node 322.

[0056] After weight calculation, for example, if it is determined that the connection at position (1, 1) will be removed, then during the connection removal process, the connection between each output layer node and its input layer node at position (1, 1) in its receptive field can be removed. This includes, for example, the connections between nodes 311 and 321, nodes 312 and 322, nodes 314 and 323, and nodes 315 and 324. Figure 3 As shown.

[0057] In some embodiments, the training method may further include: training the weights of each connection in the convolutional neural network before each removal of a portion of the connections in the convolutional kernel of the convolutional neural network.

[0058] In the above embodiments, before removing connections, the weights of each connection in the convolutional neural network can be trained, i.e., the weights of each connection in the neural network can be updated. For example, sample data can be input into the neural network to generate prediction results, and a loss function can be used to calculate the prediction error based on the difference between the prediction results and the true values ​​corresponding to the sample data. An optimization algorithm can then be used to update the weights of the connections in the neural network to minimize the prediction error.

[0059] The above process can increase the absolute value of the weights of correct connections and decrease the absolute value of the weights of incorrect connections in a neural network, so that incorrect connections are removed and correct connections are retained during the connection removal process, thereby training the neural network.

[0060] The above training can be performed once before each connection removal, or it can be performed multiple times. In other words, dynamic sparse training can be performed after the neural network weights have been trained for a certain number of rounds, thereby avoiding poor training results caused by frequent training.

[0061] The above text combines Figure 2and Figure 3 This section introduced how to perform connection removal during dynamic sparse training. The following section will further explain how to determine the first score of node pairs in step S2 for subsequent connection regeneration.

[0062] The node pair in the above embodiments includes an output layer node and an input layer node located within the first receptive field of the output layer node. In other words, the node pair in the above technical solution refers to an input layer node and an output layer node that can establish a connection in a convolutional neural network.

[0063] refer to Figure 3 As an example, due to Figure 3 The convolutional neural network shown is a single-layer convolutional neural network, where the size of the convolutional kernel is 2 by 2, and the receptive field corresponding to each output layer node is also 2 by 2.

[0064] For example, the receptive field corresponding to output layer node 321 is the coverage area of ​​the input layer when the convolutional kernel is at the top left corner, which is the input layer nodes at positions (1, 1), (1, 2), (2, 1), and (2, 2). Therefore, the node pairs including output layer node 321 are (311, 321), (312, 321), (314, 321), and (315, 321). Other output layer nodes can be determined in a similar way.

[0065] In other words, nodes in a neural network that cannot form a connection do not constitute the node pair described above in this disclosure. For example, between input layer node 319 and output layer node 321, since input layer node 319 is not within the receptive field of output layer node 321, a connection cannot be established between node 319 and node 321, and these two nodes do not belong to the node pair described above.

[0066] Figure 4 A flowchart illustrating the determination of a first score according to some embodiments of the present disclosure is shown. Figure 4 As shown, step S2, in the case that the node pair has no connection, determining the first score of the node pair may include: step S21, determining the set of paths between the node pairs whose path length is equal to a threshold; step S22, determining the first score of the node pair based on the set of paths.

[0067] In step S21, the set of paths between node pairs can be determined first.

[0068] A path between node pairs refers to a transmission route in a neural network that connects the two nodes in a pair, such as the adjacent nodes and their corresponding connections. In other words, each path between node pairs includes at least one connection present in the neural network.

[0069] The aforementioned path set could be, for example, the set of shortest paths between nodes. For instance, since the shortest path length between unconnected output layer nodes and input layer nodes in a convolutional neural network is 3, the aforementioned path set is the set of shortest paths between unconnected node pairs when the path threshold is set to 3.

[0070] It should be understood that the above threshold settings are merely illustrative, and other path thresholds, such as 5 or 7, can also be used to determine the set of paths between node pairs.

[0071] In some embodiments, determining the set of paths whose path length between the node pairs is equal to a threshold may include: determining the convolutional kernel sliding range of the path between the node pairs according to the threshold; determining a second receptive field according to the convolutional kernel sliding range, wherein the distance between the center of the second receptive field and the center of the first receptive field is within the convolutional kernel sliding range; and determining the set of paths whose path length between the node pairs is equal to the threshold according to the connection corresponding to the first receptive field and the connection corresponding to the second receptive field.

[0072] In the above embodiments, since the nodes that each output layer node in the convolutional neural network can establish connections are limited by the corresponding receptive field, the sliding range of the convolutional kernel between the node pairs can be determined first based on the path threshold.

[0073] Since non-directly connected node paths will correspond to multiple output layer nodes, the convolution kernel sliding range refers, for example, the range of differences between the convolution kernel positions corresponding to the receptive fields of multiple output layer nodes in all paths of a threshold length.

[0074] By using the aforementioned convolution kernel sliding range, other receptive fields that the path may pass through can be determined. Thus, when determining the path between nodes, the receptive fields of the output layer nodes can be used for filtering, thereby reducing the amount of computation required to determine the path.

[0075] As an example, determining the convolution kernel sliding range of the path between the node pairs based on the threshold may include: determining the number of receptive field switching of the path between the node pairs based on the threshold; and determining the convolution kernel sliding range based on the number of receptive field switching and the size of the convolution kernel.

[0076] Taking a path threshold of 3 as an example, each path of length 3 will involve two output layer nodes, which means there will be two receptive fields, i.e., the number of receptive field switching is 1.

[0077] Assuming the receptive field switches only once, if the convolution kernel size is 2x2, the furthest distance between receptive fields that can form a path is the distance between diagonally adjacent receptive fields, for example... Figure 3The receptive fields corresponding to output layer node 321 and output layer node 324 are defined. At this point, it can be determined that if the distance between the centers of the receptive fields is greater than or equal to 2, there will be no common input layer nodes between the receptive fields, and a path cannot be formed. Therefore, the sliding range of the convolutional kernel is determined to be 1 to 2.

[0078] It should be understood that the above parameters are merely illustrative. For example, with a path threshold of 5, the number of receptive field switching times can be determined to be 2, and the corresponding range can be determined based on the size of the convolution kernel.

[0079] For any node pair in a neural network, when determining the path set, the receptive field corresponding to the output layer node in the node pair can be used as the first receptive field. Then, based on the determined sliding range of the convolution kernel, the second receptive field within the range of the first receptive field is determined, thereby determining the path set of the node pair within these receptive fields.

[0080] The above method can determine the path between nodes only in a portion of the receptive field, without traversing all input layer nodes, thus reducing the amount of computation required to determine the path.

[0081] The previous section explained how to determine the set of paths between node pairs. Now, we will return to... Figure 4 Next, we will introduce how to determine the first score of a node pair based on the path set in step S22.

[0082] In some embodiments, determining a first score for the node pair based on the path set may include: determining common neighbor nodes of the node pair based on the path set, the common neighbor nodes including pathway nodes located on any path in the path set; and determining a first score for the node pair based on the number of connections of the common neighbor nodes.

[0083] After determining the set of paths, the common neighbor nodes can be identified through each path in the set. For example, if the set of paths for node pair AB includes AEBB and ADCB, then C, D, and E can be identified as common neighbor nodes for node pair AB.

[0084] The number of connections between common neighbor nodes can be used to determine whether a connection should be formed between node pairs. When there are many connections between common neighbor nodes, the node group is considered relatively closely related, and a connection should be formed between these node pairs to strengthen the connections between nodes. Conversely, when there are few connections between common neighbor nodes, the node group is considered poorly related, and a connection should not be formed between these node pairs.

[0085] As an example, the methods for determining the first score based on the connections of common neighbor nodes can include two types: node-based and path-based. These two methods will be described below.

[0086] In the first approach, the first score of a node pair can be determined by summing individual common neighbor nodes.

[0087] For example, determining a first score for a node pair based on the number of connections between the common neighbor nodes may include: determining a first score for the node pair based on the number of internal connections and the number of external connections between the common neighbor nodes, wherein the external connections include connections between the common neighbor nodes and non-common neighbor nodes, and the internal connections include connections between the common neighbor nodes.

[0088] An expression for determining the first score using common neighbor nodes could be, for example:

[0089] In the above formula, x is the path length threshold used to determine common neighbor nodes. The first score, Let z be the set of all common neighbor nodes, and z be any node belonging to the set of common neighbor nodes. The node, Let z be the number of internal connections of node z. Let z be the number of external connections of node z.

[0090] In the above formula, adding one to the number of inner connections as the numerator avoids the situation where the number of outer connections of a node cannot be considered when the number of inner connections is 0. Adding one to the number of outer connections avoids the situation where the reciprocal is undefined when the number of outer connections of a node is 0.

[0091] The above formula allows for the determination of a first score for a node pair based solely on the number of internal and external connections for each common neighbor node. A higher first score indicates a higher number of internal connections among common neighbors, while a lower score indicates a higher number of external connections. Therefore, the first score effectively reflects the characteristics of a node group and can be used to determine whether to establish a connection between node pairs.

[0092] In the second approach, the first score of a node pair can be determined by summing each path between the node pairs.

[0093] For example, determining the first score of the node pair based on the number of connections of the common neighbor nodes may include: for each path in the path set, determining the second score of each path based on the number of external connections of the common neighbor nodes on each path, wherein the external connections include connections between the common neighbor nodes and non-common neighbor nodes; and determining the first score of the node pair based on the sum of the second scores of each path in the path set.

[0094] Taking a path threshold of 3 as an example, the expression for determining the first score by path could be:

[0095] In the above formula, The first score for the node pair. P Let n be the set of all paths of length 3 between pairs of nodes, where n is a path from this set. The second fraction corresponding to path n. These are nodes on a path of length 3 between node pairs. and They are nodes The number of external connections.

[0096] For each path with a length equal to the threshold, its corresponding second score can be determined based on the number of external connections for each node on the path. By summing the second scores of all paths in the path set, the final first score of the node pair can be determined. The first score determined in this way can also fully reflect the characteristics of the node group and can be used to determine whether to generate a connection between the node pairs.

[0097] It should be understood that the formulas in the two methods described above are merely exemplary and not restrictive, and other formulas can also be used to calculate the first score of node pairs based on the number of connections.

[0098] The above text combines Figure 4 This disclosure describes how the first score of a node pair is determined. Below, we will combine... Figure 5 Next, we will explain how to perform connection regeneration.

[0099] Figure 5 A flowchart illustrating connection regeneration according to some embodiments of this disclosure is shown. For example... Figure 5As shown, step S3, generating the second connection based on the first score of each node pair in the convolutional neural network, may include: step S31, for each element in the convolutional kernel, determining the probability of generating a connection corresponding to that element based on the sum of the first scores of all node pairs corresponding to that element; step S32, determining the second element based on the probability corresponding to each element in the convolutional kernel; step S33, generating the second connection corresponding to the second element, wherein the second connection includes connections between all node pairs corresponding to the second element.

[0100] In step S31, for each element within the convolution kernel, that is, each position in the matrix, the first scores of all node pairs corresponding to that element can be summed.

[0101] Similar to connections in a neural network, each node pair in a neural network can also correspond to an element in the convolutional kernel. (See reference) Figure 3 The example shown includes, for example, nodes 311 and 321, 312 and 322, 314 and 323, and 315 and 324, corresponding to the element at position (1, 1) in the convolution kernel.

[0102] In some embodiments, generating a second connection based on a first score of each node pair in the convolutional neural network may further include: for each node pair, determining the receptive field center of the output layer node in the node pair; and determining the element in the convolutional kernel corresponding to the node pair based on the position of the input layer node in the node pair relative to the center.

[0103] In the above embodiments, for any unconnected node pair in the convolutional neural network, since there is currently no connection between the node pairs generated by the convolution kernel, it is not possible to directly map the node pairs to the elements in the convolution kernel. Therefore, before summing the first score of the node pair, the element corresponding to the node pair can be determined based on the position of the receptive field center of the input layer node relative to the output layer node in the node pair.

[0104] Taking a 3x3 convolution kernel as an example, for any node pair, the position of the input layer node in the receptive field can be determined based on the position of the input layer node relative to the center of the receptive field of the output layer node, thereby determining the element corresponding to the node pair.

[0105] In other words, the correspondence between node pairs and convolutional kernels can also be understood as follows: if there is a connection between node pairs, then the connection is generated by the element at the corresponding position of the convolutional kernel.

[0106] After summing the first fractions, for example, the reciprocal of the sum of the first fractions corresponding to each element can be normalized, and the connection regeneration probability corresponding to each element can be determined by means of a multinomial distribution.

[0107] By using the soft sampling method described above, it is possible to avoid the neural network structure getting stuck in local optima during the early stages of training, thereby improving the training effect of the neural network.

[0108] In step S32, the second element that needs to be regenerated can be determined based on the connection regeneration probability corresponding to each element.

[0109] Similar to the first element that needs to be removed as described above, the second element can be determined by random selection based on the connection regeneration probability mentioned above.

[0110] By using the above method, important connections in the neural network can be identified and regenerated in order to train the neural network.

[0111] It should be understood that the above method is merely exemplary, and it is also possible to directly determine the elements with the highest sum of the first fractions as the second elements that need to be connected and regenerated.

[0112] In step S33, all connections corresponding to the determined second element can be generated, thereby ensuring the weight sharing constraint in the convolutional neural network.

[0113] For example, if the second element is determined to be the element at position (2, 1) in the convolution kernel, the connection between each output layer node and the input layer node at position (2, 1) in its receptive field can be generated respectively, thereby ensuring the weight sharing constraint in the convolutional neural network, that is, the connection between the input layer node and the output layer node corresponds to the weight matrix of the convolution kernel.

[0114] In other words, the above-mentioned connection regeneration can also be understood as resetting the weights at the corresponding positions in the convolution kernel to non-zero, and then re-performing the convolution operation to determine the connection between the input layer nodes and the output layer nodes.

[0115] The above text combines Figures 1 to 5 The various steps in the training method have been introduced. Below, we will combine... Figure 6 This section introduces a specific implementation of the training method.

[0116] Figure 6 A schematic diagram illustrating the training of a convolutional neural network according to some embodiments of the present disclosure is shown. Figure 6 As shown, the training method may include steps (a) to (c). Figure 6The bipartite graph shown represents a convolutional neural network in which a 2x2 convolutional kernel performs convolution operations on a 3x3 input layer. This convolutional neural network may include input layer nodes 611 to 619 and output layer nodes 621 to 624.

[0117] In step (a), connections can be removed first based on the weights of the connections in the neural network.

[0118] The implementation method of step (a) has been described above in conjunction with... Figure 2 and Figure 3 As previously explained, this will not be repeated here. In step (a), after calculating the weights, for example, it is determined that the connection at position (1, 1) of the convolution kernel will be removed.

[0119] In step (b), the first score, i.e., the connection regeneration score, of each node pair in the neural network that has no connection can be determined based on the path set of the node pairs. An example of the first score of node pairs including output layer node 621 is shown below. Figure 6 As shown, the score between nodes 611 and 621 is, for example, 0.07, the score between nodes 612 and 621 is, for example, 0.15, and the score between nodes 615 and 621 is, for example, 0.78.

[0120] In step (c), connection regeneration can be performed based on the determined first fraction. Figure 6 In the example shown, the sum of the first scores corresponding to the (2, 2) element in the convolution kernel is the highest, and the probability of generating the connection corresponding to the (2, 2) element is also the highest.

[0121] If it is determined that a connection corresponding to the element (2, 2) will be generated, then step (c) will generate connections between all node pairs corresponding to the element (2, 2), including connections between nodes 615 and 621, nodes 616 and 622, nodes 618 and 623, and nodes 619 and 624, as follows: Figure 6 As shown.

[0122] The number of connections generated during the connection regeneration process can be the same as the number of connections removed during the connection removal process, thus ensuring that the sparsity of the network remains unchanged.

[0123] In the training method, connection removal and connection regeneration can be performed until the number of training iterations reaches a threshold or the neural network converges, meaning that the removed and regenerated connections in the neural network are the same.

[0124] In addition, the neural network can be initialized before the training method begins, such as by small-world initialization or scale-free initialization, so that the initialized neural network has certain characteristics and improves the effect of subsequent training.

[0125] The above describes the training method provided by embodiments of this disclosure. By removing and generating connections corresponding to elements in the convolutional kernel, the training method of this disclosure enables the network to maintain weight sharing constraints during dynamic sparse training, thereby improving the training effect of the neural network. The application scenarios of the above-described neural network will be described below.

[0126] In some embodiments, a convolutional neural network can be used for image processing, wherein the input of the convolutional neural network is a first image, and the output is at least one of the following: the category of the first image, an object in the first image, a second image generated from the first image, and text generated from the first image.

[0127] In other words, the convolutional neural network trained in this application can be used for tasks such as image classification, object recognition in images, image enhancement, and text generation from images. Text generation from images includes tasks such as providing textual descriptions of images or writing based on images; the input of the convolutional neural network is an image, and the output is the corresponding text.

[0128] In tests using a 10-class small image dataset and a 100-class fine-grained dataset as data samples to train a convolutional neural network, the training method proposed in this disclosure achieves comparable results to the ensemble (SET) training method in related technologies, while reducing the computational load required for training and improving training efficiency.

[0129] In other embodiments, a convolutional neural network can be used for text processing, wherein the input of the convolutional neural network is a first text, and the output is at least one of the following: the category of the first text, elements in the first text, a second text generated based on the first text, and an image generated based on the first text.

[0130] In other words, the training obtained in this application

[0131] It can be used for tasks such as text classification, text element recognition, machine translation, and image generation from text. For example, in intelligent painting, the input of a convolutional neural network is text, and the output is the corresponding image.

[0132] Furthermore, the training method proposed in this disclosure can be used to train neural networks such as Convolutional Neural Networks (CNN), Deep Convolutional Networks, and Lightweight Convolutional Models, thereby reducing training complexity and training time.

[0133] The above application scenarios are merely illustrative and not restrictive. The training method proposed in this disclosure can reduce the training complexity of convolutional neural networks in various scenarios, reduce the training time required, and improve the training effect.

[0134] The following is for reference. Figure 7 and Figure 8 The present disclosure describes a training apparatus for performing any of the above-described training methods. Figure 7 A block diagram of a training apparatus according to some embodiments of the present disclosure is shown.

[0135] like Figure 7 As shown, the training device 7 for the convolutional neural network includes: a connection removal module 71, configured to remove a first connection according to the weight of the connection between each node in the convolutional neural network, to obtain a convolutional neural network after connection removal, wherein the first connection corresponds to a first element in the convolutional kernel of the convolutional neural network; a first score determination module 72, configured to determine a first score of a node pair in the convolutional neural network after connection removal, wherein the node pair includes an output layer node and an input layer node located within a first receptive field of the output layer node, wherein the first receptive field includes the coverage area of ​​the convolutional kernel in the input layer at a first sliding position; and a connection regeneration module 73, configured to generate a second connection according to the first score of each node pair in the convolutional neural network, wherein the second connection corresponds to a second element in the convolutional kernel of the convolutional neural network.

[0136] The connection removal module 71 of the training device 7 can be used, for example, to perform... Figure 1 Step S1. The first score determination module 72 of the training device 7 can be used, for example, to perform... Figure 1 Step S2. The connection regeneration module 73 of the training device 7 can be used, for example, to perform... Figure 1 Step S3.

[0137] It should be understood that the training apparatus may also include other modules for performing other steps in the training method of the embodiments of this disclosure, such as an initialization module, a weight training module, etc.

[0138] Figure 8 A block diagram of a training apparatus according to other embodiments of the present disclosure is shown.

[0139] like Figure 8 As shown, the training apparatus 8 includes: at least one memory 81; and at least one processor 82 coupled to the at least one memory 81, the at least one processor 82 being configured to execute the training method as described in any of the foregoing embodiments based on instructions stored in the at least one memory 81.

[0140] Memory 81 is used to store one or more computer-readable instructions. Memory 81 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory, dynamic random access memory, static random access memory, read-only memory, and flash memory. Memory 81 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0141] Processor 82 is configured to execute computer-readable instructions to implement the training method described in any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments, for example... Figure 1 The steps involved are repeated here, so the details will not be repeated.

[0142] The processor 82 can be manifested as various processing devices, such as a central processing unit (CPU), a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The CPU can be based on x86 or ARM architectures, etc.

[0143] The processor 82 and the memory 81 can communicate with each other directly or indirectly. For example, the processor 82 and the memory 81 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. The processor 82 and the memory 81 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0144] It should be noted that Figure 8 The components of the training device 8 shown are merely exemplary and not limiting; the training device 8 may have other components depending on the actual application requirements. The processor 82 can control other components in the training device 8 to perform desired functions. The training device 8 can be implemented by software, firmware, and / or hardware and can be integrated into a device with the relevant application installed.

[0145] Figure 9 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0146] like Figure 9 As shown, computer system 9 can be represented in the form of a general computing device. Computer system 9 includes memory 91, processor 92, and bus 90 connecting different system components.

[0147] The memory 91 can be various forms of computer-readable storage media, such as system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, application programs, a boot loader, and other programs. System memory may include volatile storage media, such as random access memory and / or cache memory. Non-volatile storage media may store, for example, instructions for executing corresponding embodiments of the training method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0148] The processor 92 can be implemented using discrete hardware components such as general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gates or transistors. Accordingly, each module can be implemented by executing instructions in the central processing unit's running memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.

[0149] Bus 90 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, industry standard architecture buses, microchannel architecture buses, and peripheral component interconnect buses.

[0150] The computer system 9 may also include an input / output interface 93, a network interface 94, and a storage interface 95. These interfaces 93, 94, and 95, as well as the memory 91 and the processor 92, can be connected via a bus 90. The input / output interface 93 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 94 provides a connection interface for various networked devices. The storage interface 95 provides a connection interface for external storage devices such as floppy disks and USB flash drives.

[0151] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to implement the training method described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, the computer instructions containing program code for performing the methods shown in the flowcharts.

[0152] Various embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0153] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for training a convolutional neural network, comprising: Based on the weights of the connections between nodes in the convolutional neural network, the first connection is removed to obtain the convolutional neural network after connection removal. The first connection corresponds to the first element in the convolutional kernel of the convolutional neural network. For a node pair in the convolutional neural network after the connection is removed, in the case that there is no connection in the node pair, a first score of the node pair is determined, wherein the node pair includes an output layer node and an input layer node located within a first receptive field of the output layer node, and the first receptive field includes the coverage area of ​​the convolutional kernel in the input layer at the first sliding position; A second connection is generated based on the first score of each node pair in the convolutional neural network, and the second connection corresponds to the second element in the convolutional kernel of the convolutional neural network.

2. The training method according to claim 1, wherein, The generation of the second connection based on the first score of each node pair in the convolutional neural network includes: For each element within the convolutional kernel, the probability of generating a connection corresponding to that element is determined based on the sum of the first scores of all node pairs corresponding to that element. The second element is determined based on the probability corresponding to each element in the convolution kernel; Generate a second connection corresponding to the second element, the second connection including connections between all node pairs corresponding to the second element.

3. The training method according to claim 2, wherein, Generating the second connection based on the first score of each node pair in the convolutional neural network further includes: For each node pair, determine the receptive field center of the output layer node in the node pair; Based on the position of the input layer node in the node pair relative to the center, determine the element in the convolution kernel corresponding to the node pair.

4. The training method according to claim 1, wherein, Removing the first connection based on the weights of the connections between nodes in the convolutional neural network includes: For each element within the convolutional kernel, the probability of removing the connection corresponding to that element is determined based on at least one of the sum of the weights of all connections corresponding to that element and the sum of the relative importance of the weights. The first element is determined based on the probability corresponding to each element in the convolution kernel; Remove the first connection corresponding to the first element, the first connection including the connections between all node pairs corresponding to the first element.

5. The training method according to claim 1, wherein, In the case where the node pair is not connected, determining the first score of the node pair includes: Determine the set of paths between the node pairs whose path length is equal to a threshold; Based on the set of paths, determine the first score of the node pair.

6. The training method according to claim 5, wherein, The set of paths whose path length between the node pairs is equal to the threshold includes: Based on the threshold, determine the convolution kernel sliding range of the path between the node pairs; The second receptive field is determined based on the sliding range of the convolution kernel, and the distance between the center of the second receptive field and the center of the first receptive field is within the sliding range of the convolution kernel. Based on the connections corresponding to the first receptive field and the connections corresponding to the second receptive field, determine the set of paths between the node pairs whose path length is equal to a threshold.

7. The training method according to claim 6, wherein, Determining the convolution kernel sliding range of the path between the node pairs based on the threshold includes: Based on the threshold, determine the number of receptive field switching of the path between the node pairs; The sliding range of the convolution kernel is determined based on the number of receptive field switching and the size of the convolution kernel.

8. The training method according to claim 5, wherein, Determining the first score of the node pair based on the set of paths includes: Based on the path set, determine the common neighbor nodes of the node pair, where the common neighbor nodes include path nodes located on any path in the path set; The first score of the node pair is determined based on the number of connections between the common neighbor nodes.

9. The training method according to claim 8, wherein, Determining the first score of the node pair based on the number of connections to the common neighbor nodes includes: A first score for the node pair is determined based on the number of internal connections and the number of external connections of the common neighbor node, wherein the external connections include connections between the common neighbor node and non-common neighbor nodes, and the internal connections include connections between the common neighbor nodes.

10. The training method according to claim 8, wherein, Determining the first score of the node pair based on the number of connections to the common neighbor nodes includes: For each path in the path set, a second score is determined based on the number of external connections of the common neighbor nodes on each path, wherein the external connections include connections between the common neighbor nodes and non-common neighbor nodes; The first score of the node pair is determined by summing the second scores of each path in the path set.

11. The training method according to claim 1, further comprising: Before removing a portion of the connections in the convolutional kernel of the convolutional neural network each time, the weights of each connection in the convolutional neural network are trained.

12. The training method according to claim 1, wherein: The input to the convolutional neural network is a first image, and the output is at least one of the following: the category of the first image, an object in the first image, a second image generated from the first image, and text generated from the first image; or The input to the convolutional neural network is a first text, and the output is at least one of the following: the category of the first text, the elements in the first text, the second text generated based on the first text, and the image generated based on the first text.

13. A training device for a convolutional neural network, comprising: The connection removal module is configured to remove a first connection based on the weight of the connection between each node in the convolutional neural network, thereby obtaining a convolutional neural network after connection removal, wherein the first connection corresponds to the first element in the convolutional kernel of the convolutional neural network. The first score determination module is configured to determine a first score for a node pair in the convolutional neural network after the connection is removed, in the case that the node pair has no connection, wherein the node pair includes an output layer node and an input layer node located within a first receptive field of the output layer node, and the first receptive field includes the coverage area of ​​the convolutional kernel in the input layer at the first sliding position. The connection regeneration module is configured to generate a second connection based on a first score of each node pair in the convolutional neural network, the second connection corresponding to a second element in the convolutional kernel of the convolutional neural network.

14. A training device for a convolutional neural network, comprising: At least one memory; as well as At least one processor coupled to the at least one memory, the at least one processor being configured to execute a training method for a convolutional neural network as described in any one of claims 1 to 12 based on instructions stored in the at least one memory.

15. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the training method of a convolutional neural network as described in any one of claims 1 to 12.

16. A computer program product, when run on a computer, causes the computer to implement the training method of a convolutional neural network as described in any one of claims 1 to 12.

Images