An automated structure learning method for brain-like binary neural networks

Abstract

The present invention relates to a brain-like binary neural network automatic structure learning method, belongs to the field of neural networks, and comprises the following steps: S1: binarizing a benchmark model, including binarization of weights and activations; S2: establishing multiple parallel feedforward subnets based on the binarized benchmark model, connecting different subnets through weighted lateral paths at the channel level, thereby obtaining an initial CBiNet model structure; S3: establishing a total loss function based on the loss functions of different application tasks and combining sparse regularization terms based on lateral path connection grouping; S4: training CBiNet on a training set after setting a learning rate, terminating the training when the loss function performance converges, and obtaining a CBiNet with sparse lateral path connections; S5: testing CBiNet on a test set. The CBiNet constructed by the present invention can improve the feature expression ability of the binarized neural network, and can make the image processing task more precise and more accurate.

Description

Technical Field

[0001] The present invention belongs to the field of deep neural networks and relates to an automatic structure learning method of a brain-like binary neural network. Background Art

[0002] As an extreme version of the compression model, the Binary Neural Network (BNN) constrains the full-precision weights and activations of the Deep Neural Network (DNN) to two discrete values, namely {-1, +1}. Therefore, in theory, the memory capacity required for DNN reasoning can be reduced by up to 32 times. By utilizing BNN, energy-intensive operations such as floating-point multiplication can be replaced by efficient bit operations, which reduces computational latency and power consumption by several orders of magnitude compared to similar high-precision models. However, BNNs usually cause severe performance degradation.

[0003] Recent research suggests using multiple binarized subnetworks in BNNs to better approximate the output of their corresponding high-precision models, rather than using multi-bit data to represent DNN parameters and activations. Experimental results show that by using 4 to 5 times more parameters, the expanded binarized ResNet-18 achieves performance comparable to the original ResNet-18 model represented in floating points. The latest research suggests that similar to the human brain, which is roughly a complex binary neural network, brain-like BNNs with multiple subnetwork expansions and integrations are still more efficient than their corresponding quantized networks with multi-bit representations. However, for models with an increased number of subnetworks and parameters, there is significant room for improvement in computational efficiency.

[0004] Early brain-like binary neural networks with multiple binary sub-networks or modules usually adopted a predefined model structure, which is also an important reason for the large computational overhead. At the same time, the increase in the number of sub-networks also hindered the improvement of model performance. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a method for automatic structural learning of brain-like binary neural networks, and to solve the problems of low precision and low accuracy in image processing through a cellular binary neural network (CBiNet). CBiNet consists of multiple parallel feedforward subnetworks, which are connected by channel-level weighted lateral paths to improve the pattern recognition ability of the neural network model. In the training phase, the L21 norm of the lateral weight is used as a regularization term for group sparsity, and the lateral connection is optimized in combination with the training of a specific task loss function, which effectively reduces the computational and storage overhead while solving the problem of decreased accuracy caused by low-precision representation.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for automatic structure learning of a brain-like binary neural network comprises the following steps:

[0008] S1: Binarize the baseline model, including weights and activations;

[0009] S2: Based on the binary benchmark model, multiple parallel feedforward subnetworks are established, and different subnetworks are connected through channel-level weighted lateral paths, thereby obtaining the initial honeycomb binary neural network CBiNet model structure;

[0010] S3: Based on the loss functions of different application tasks, the total loss function is established in combination with the sparse regularization term based on the horizontal path connection grouping;

[0011] S4: After setting the learning rate, train CBiNet on the training set. When the loss function converges, the training ends and a CBiNet with sparse lateral path connections is obtained.

[0012] S5: Test CBiNet on the test set.

[0013] Furthermore, the benchmark model in step S1 is a feedforward deep neural network, including but not limited to a ResNet model, a VGG model, and a GoogLeNet model.

[0014] Further, the CBiNet model structure in step S2 is:

[0015] CBiNet consists of multiple binary sub-networks. For the i-th sub-network, the input and output of the k-th layer are represented as X (i,k) and Y (i,k) ; For the kth layer, the channel-by-channel weights of the lateral connections from subnetwork i to subnetwork j are expressed as where c out is the number of output channels; in the jth subnetwork, the feature map of the tth channel of the input of the k+1th layer The feature map of the tth channel output of the kth layer in different sub-networks may depend on:

[0016]

[0017] Where i and j are the indices of the binarized subnetwork, k is the number of binarized convolutional layers, t is the number of feature channels, and Y t (j,k) Represents the feature map of the tth channel of the output of the kth layer of the jth subnetwork; Y represents the weight of the t-th channel of the lateral connection from subnetwork i to subnetwork j in the k-th layer;t (i,k) Represents the feature map of the tth channel of the output of the kth layer of the i-th sub-network.

[0018] Furthermore, in step S2, a learning optimization method is used to reduce the additional computational complexity and storage overhead caused by dense lateral connections, in view of the density of lateral connections between different subnetworks in the CBiNet model structure and the different groupings of different convolutional layers for feature representation:

[0019]

[0020] Where Φ represents the weight set of lateral connections, ||Φ|| 2,1 represents the L21 norm, which plays a role in grouping and sparseness. N represents the total number of binary convolutional layers. The importance of the connection is characterized by the absolute size of the weight.

[0021] Further, step S3 includes: according to the loss function of different application tasks Combined with the regularization term proposed in formula (2), the final loss function for training is obtained:

[0022]

[0023] Where γ is a hyperparameter used to control the sparsity of lateral connection grouping, and the application task is an image processing task, including but not limited to image classification, image segmentation and object detection.

[0024] Furthermore, in step S4, the stochastic gradient descent SGD method with momentum is used as the optimizer for model learning during the training process.

[0025] The beneficial effect of the present invention is that: on the basis of adding lateral connections to the feedforward neural network, the present invention enables the network model to learn the sparsely grouped lateral connection structure, reduces the computational and storage overhead, improves the pattern recognition ability of the brain-like binary neural network model, and obtains the optimal result of the model structure. By using the present invention, the precision of image processing (including image classification, target detection, image segmentation, etc.) tasks can be made higher and the accuracy rate stronger.

[0026] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 This is a structural diagram of the binary neural subnetwork described in the present invention;

[0029] Figure 2 It is a structural diagram of a honeycomb binary neural network according to the present invention;

[0030] Figure 3 Schematic diagram of grouping coefficient regularization between sub-networks. DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0032] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0033] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0034] See also Figure 1 to Figure 3The brain-like honeycomb binary neural network used in the present invention is composed of multiple parallel feedforward subnets, which are connected by channel-level weighted lateral paths, whose lateral weights are sparse and grouped according to different source layers, and the L21 norm of the lateral weights is used as a group sparse regularization term for training and learning the optimized structure of lateral connections. The implementation scheme of the present invention is based on two image classification datasets, CIFAR10 and ImageNet, and implements CBiNet technology on the basis of the ResNet full-precision benchmark model.

[0035] 1) Establishment of CBiNet model

[0036] First, the ResNet full-precision benchmark model is binarized, including the binary representation of weights and activations. Then, multiple parallel feedforward subnets are established based on the binarized ResNet model, and different subnets are connected through channel-level weighted lateral paths, thus obtaining the initial CBiNet model structure.

[0037] The CBiNet proposed in the present invention adopts a learnable binary multi-subnet structure with global lateral path connections, and its lateral connections are sparse and grouped according to different source layers. This architecture is similar to the mode of information transmission between different neurons in the human brain through synapses, and the connections between synapses are different in strength. In addition, studies have shown that the transition of neurons from the inhibitory state to the activated state in the human brain is achieved by pulse signals breaking through the membrane potential. This mechanism is similar to the binarization of weights and activations in CBiNet, so CBiNet can be called a brain-like model architecture.

[0038] like Figure 1-2 As shown in Figure 2, different from the binary neural network BNN, CBiNet consists of multiple binary sub-networks. For the i-th sub-network, the input and output of the k-th layer are represented as X (i,k) and Y (i,k) At the same time, for the kth layer, the channel-by-channel weight of the lateral connection from subnetwork i to subnetwork j can be expressed as where c out is the number of output channels. Due to its channel-by-channel nature, the lateral connection weights are represented in vector form in this chapter. In the jth subnetwork, the feature map X of the tth channel of the input of the k+1th layer t (j,k+1) The feature map of the tth channel output of the kth layer in different sub-networks may depend on:

[0039]

[0040] where Y t (j,k) Represents the feature map of the tth channel of the output of the kth layer of the jth subnetwork; Y represents the weight of the t-th channel of the lateral connection from subnetwork i to subnetwork j in the k-th layer; t (i,k) Represents the feature map of the tth channel of the output of the kth layer of the i-th sub-network.

[0041] like Figure 3 As shown, in view of the density of lateral connections between different sub-networks and the different grouping effects of different convolutional layers on the performance of the final image classification, this patent adopts a learning optimization method to greatly reduce the additional computational complexity and storage overhead brought by dense lateral connections:

[0042]

[0043] Where i and j are the indices of the binarized subnetworks, respectively, while N represents the total number of binarized convolutional layers, Φ represents the weight set of the lateral connections, and ||Φ|| 2,1 represents the L21 norm, which plays a role in grouping sparseness, while the importance of the connection is characterized by the absolute size of the weight.

[0044] 2) CBiNet model training and testing

[0045] For classification tasks, the objective function is basically defined based on cross-entropy CE (Cross-Entropy). In particular, in the training process of binary neural network classification models, full-precision models are often combined for knowledge distillation, that is, a pre-trained full-precision benchmark model is used as a teacher model to help train a compact binary student model (Student Model) to reduce the output distribution offset caused by the binarization of the full-precision neural network.

[0046] The output distribution difference between the teacher model and the student model is used as the objective function to optimize the lateral connections between subnetworks:

[0047]

[0048] in, and The student models are θ and the teacher model t θ The soft labels generated are is the output of the student model, Y g are the ground truth labels, and ρ is the adjustment coefficient used to control the above two losses. Specifically, KL divergence (Kullback-Leibler divergence) is used to define the first loss in (3): The cross-entropy CE (Cross-Entropy) is used to define the second loss in (3)

[0049] Combining the regularization term proposed in (2) with the loss function in (3) yields the final loss function for training:

[0050]

[0051] where γ is a hyperparameter used to control the sparsity of the lateral connection grouping.

[0052] In this embodiment, the adjustment coefficient ρ and the regularization term coefficient γ are set to 0.99 and 0.0001 respectively to achieve the best model effect. After setting the learning rate η, the student model s is trained on the training set. θ During the training process, the stochastic gradient descent SGD method with momentum was selected as the optimizer for model learning. When the loss function converges, the training ends. Those lateral path connections that are less than a certain threshold can be set to 0 and deleted to perform network structure pruning. The final image classification model is obtained by testing on the corresponding test set to classify the image.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A brain-like binary neural network automated structure learning method, characterized by: Applied to image processing tasks, it includes the following steps: S1: Binarize the baseline model, including weights and activations; S2: Based on the binary benchmark model, multiple parallel feedforward subnetworks are established, and different subnetworks are connected through channel-level weighted lateral paths, thereby obtaining an initial honeycomb binary neural network CBiNet model structure; the CBiNet model structure is: CBiNet consists of multiple binary sub-networks. For the i-th sub-network, the input and output of the k-th layer are represented as X (i,k) and Y (i,k) ; For the kth layer, the channel-by-channel weights of the lateral connections from subnetwork i to subnetwork j are expressed as where c out is the number of output channels; in the jth subnetwork, the feature map of the tth channel of the input of the k+1th layer Depends on the feature map of the t-th channel of the k-th layer output in other different sub-networks: Where i and j are the indexes of the binarized sub-network, k represents the number of binarized convolutional layers, and t represents the number of feature channels. Represents the feature map of the tth channel of the output of the kth layer of the jth subnetwork; represents the weight of the t-th channel of the lateral connection from subnetwork i to subnetwork j in the k-th layer; Represents the feature map of the tth channel of the output of the kth layer of the i-th subnetwork; In view of the density of lateral connections between different sub-networks in the CBiNet model structure, and the different grouping of different convolutional layers for feature representation, a learning optimization method is used to reduce the additional computational complexity and storage overhead caused by dense lateral connections: Where Φ represents the weight set of lateral connections, ||Φ|| 2，1 represents the L21 norm, which plays a role in grouping sparseness. N represents the total number of binary convolutional layers. The importance of the connection is represented by the absolute size of the weight. S3: Based on the loss functions of different application tasks, the total loss function is established in combination with the sparse regularization term based on the horizontal path connection grouping; S4: After setting the learning rate, train CBiNet on the training set. When the loss function converges, the training ends and a CBiNet with sparse lateral path connections is obtained. S5: Test CBiNet on the test set.

2. The method for automatic structure learning of a brain-like binary neural network according to claim 1, characterized in that: The benchmark model in step S1 is a feedforward deep neural network, including but not limited to a ResNet model, a VGG model, and a GoogLeNet model.

3. The method for automatic structure learning of a brain-like binary neural network according to claim 1, characterized in that: The step S3 includes: according to the loss function of different application tasks Combined with the regularization term proposed in formula (2), the final loss function for training is obtained: Where γ is a hyperparameter used to control the sparsity of lateral connection grouping, and the application task is an image processing task, including but not limited to image classification, image segmentation and object detection.

4. The method for automatic structure learning of a brain-like binary neural network according to claim 1, characterized in that: In step S4, the stochastic gradient descent SGD method with momentum is used as the optimizer for model learning during the training process.

Images