Neural network affine connection method and system

Abstract

The invention discloses a neural network affine connection method and system. The neural network affine connection method comprises the following steps: S1, storing a sampling matrix B.sw, and calculating an actual matrix B.aw in each forward reasoning of a neural network; s2, stretching or compressing each column of the sampling matrix B.sw, converting the shape of the sampling matrix B.sw from (B.sdim, B.dim) to (B.indim, B.dim), performing longitudinal affine transformation on the sampling matrix B.sw, and mapping the sampling matrix B.sw to an actual matrix B.aw with different longitudinal dimensions; and S3, carrying out forward propagation. The neural network affine connection method and system disclosed by the invention have the beneficial effects that the traditional feedforward neural network can be compatible and improved. A trained neural network with excellent functions can be transformed into a neural network capable of accepting input of any dimension under the condition that less extra training is needed.

Description

Technical Field

[0001] The present invention belongs to the field of neural network affine connection, and in particular relates to a neural network affine connection method and system. Background Art

[0002] In the field of artificial intelligence, the affine connection of neural networks is used to solve the problem of limited input format of traditional feedforward neural networks. In other words, this algorithm solves the problem that the input dimension of the network is inconsistent with the dimension of the actual data.

[0003] On this basis, under normal circumstances, if you encounter data that does not match the input dimension of the existing (feedforward) neural network, the methods usually used include but are not limited to: compressing or padding the data to match the input of the neural network, training a new neural network that matches the input data format. Existing solutions may cause interference or loss of information contained in the data, or require a lot of cost to adjust or even rebuild the neural network, and need to be further improved. Summary of the invention

[0004] The present invention aims at the state of the prior art, overcomes the above defects, and provides a neural network affine connection method and system.

[0005] The present invention adopts the following technical scheme, a neural network affine connection method, wherein the actual weight matrix B.aw is generated by the sampling weight matrix B.sw through affine transformation during inference, and the neural network affine connection method comprises the following steps:

[0006] Step S1: store the sampling matrix B.sw, and calculate the actual matrix B.aw in each forward inference of the neural network;

[0007] Step S2: stretch or compress each column of the sampling matrix B.sw so that the shape of the sampling matrix B.sw is converted from (B.sdim, B.dim) to (B.in_dim, B.dim), perform an affine transformation on the sampling matrix B.sw in the longitudinal direction, and map the sampling matrix B.sw to an actual matrix B.aw with a different longitudinal dimension;

[0008] Step S3: Perform forward propagation.

[0009] As a preferred technical solution of the above solution, step S3 is specifically implemented as the following steps:

[0010] Step S31: B accepts the feature value transmitted by A and sets B.in_dim according to the dimension of A;

[0011] Step S32: B.sw generates B.aw by executing steps S1 and S2;

[0012] Step S33: B uses B.aw to process the eigenvalues ​​of A;

[0013] Step S34: B outputs its own feature value B.feature.

[0014] As a preferred technical solution of the above scheme, in step S3, the eigenvalue calculation formula of the forward propagation is: B.feature=A.feature×AF(B.sw); wherein AF() is an affine transformation function, A.feature is the input eigenvalue in_feature, and B.feature is the output eigenvalue out_feature.

[0015] As a preferred technical solution of the above solution, the neural network affine connection method further includes step P1, which is located after step S3:

[0016] Step P1: The AC layer acts as a plug-in for other neural networks, replacing or overwriting the original input layer, and accepting input of any dimension using the AC method.

[0017] As a preferred technical solution of the above solution, the neural network affine connection method further includes step Q1, which is located after step S3:

[0018] Step Q1: Each AC layer is extracted separately after training, and any AC layer is connected to any AC layer.

[0019] The present invention adopts the following technical scheme, a neural network affine connection system, and applies the steps of any one of the above technical schemes of the neural network affine connection method.

[0020] The neural network affine connection method and system disclosed in the present invention have the following beneficial effects:

[0021] 1. The AC layer is used as a plug-in for other neural networks, replacing or overwriting the original input layer, and accepting input of any dimension using the AC method. It is compatible with and improves traditional feedforward neural networks. With less additional training, the original well-functioning, trained neural network can be transformed into a neural network that can accept input of any dimension.

[0022] 2. The AC connection method can decouple the neural network and reduce its black-box degree. The weight data trained in one network can be taken out and inserted into other neural networks for use. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1A and Figure 1B They are respectively schematic diagrams of the principles of the neural network affine connection of the present application.

[0024] Figure 2A and Figure 2BThey are schematic diagrams of the principles of the sampling weight and the actual weight of this application.

[0025] Figure 3A and Figure 3B They are respectively schematic diagrams of the principles of the neural network affine connection method of the present application.

[0026] Figure 4 and Figure 5 They are respectively schematic diagrams of application scenarios of the neural network affine connection method of the present application. DETAILED DESCRIPTION

[0027] The present invention discloses a neural network affine connection method and system. Figures 1A to 5 , the specific implementation modes of the present invention are further described.

[0028] Those skilled in the art should note that the various embodiments of the present application may involve “Affine Connect”, abbreviated as “AC”, which is defined as neural network affine connection.

[0029] Those skilled in the art should note that the “B.sw” that may be involved in various embodiments of the present application is defined as a sampling matrix or a sampling weight matrix.

[0030] Those skilled in the art should note that the “B.aw” that may be involved in various embodiments of the present application is defined as an actual matrix or an actual weight matrix.

[0031] Those skilled in the art should note that Figure 1A and Figure 1B In various embodiments of the present application, the element in row i and column j of any matrix A is denoted as A[i][j].

[0032] Except for the input layer, any layer in the neural network is assumed to be A. A is composed of the neurons in this layer and the weights of the neural network before A used to connect the neurons in this layer.

[0033] Let A.dim represent the number of neurons in A (i.e., the dimension of A), and let the vector A.feature represent all neurons in A. It is easy to see that its dimension is equal to A.dim.

[0034] Let the matrix Aw represent the weights of A, where the tuple Awsize(n) represents the size of Aw in the nth dimension, and Awsize() is a tuple of the shape of Aw.

[0035] Let A.in_dim, A.out_dim represent the input and output dimensions of A, where A.in_dim = Awsize(1), A.out_dim = Awsize(0).

[0036] Example 1.

[0037] Preferably, in the neural network affine connection method, the actual weight matrix B.aw is generated by the sampling weight matrix B.sw through affine transformation during inference, and the neural network affine connection method includes the following steps:

[0038] Step S1: store the sampling matrix B.sw, and calculate the actual matrix B.aw in each forward inference of the neural network;

[0039] Step S2: stretch or compress each column of the sampling matrix B.sw so that the shape of the sampling matrix B.sw is converted from (B.sdim, B.dim) to (B.in_dim, B.dim), perform an affine transformation on the sampling matrix B.sw in the longitudinal direction, and map the sampling matrix B.sw to an actual matrix B.aw with a different longitudinal dimension;

[0040] Step S3: Perform forward propagation.

[0041] Wherein, step S3 is specifically implemented as the following steps:

[0042] Step S31: B accepts the feature value transmitted by A and sets B.in_dim according to the dimension of A;

[0043] Step S32: B.sw generates B.aw by executing steps S1 and S2;

[0044] Step S33: B uses B.aw to process the eigenvalues ​​of A;

[0045] Step S34: B outputs its own feature value B.feature.

[0046] Among them, in step S3, the forward propagation feature value calculation formula is: B.feature = A.feature × AF (B.sw), which can also be written as: out_feature = in_feature × AF (B.sw).

[0047] Among them, AF() is the affine transformation function, A.feature is the input feature value in_feature, and B.feature is the output feature value out_feature.

[0048] The neural network affine connection method further includes step P1, which is located after step S3:

[0049] Step P1: The AC layer acts as a plug-in for other neural networks, replacing or overwriting the original input layer, and accepting input of any dimension using the AC method.

[0050] Among them, the neural network affine connection method further includes step Q1, which is located after step S3:

[0051] Step Q1: After training, each AC layer can be separately extracted, and any AC layer can be connected to any AC layer.

[0052] It is worth mentioning that the rigorous affine transformation method is given by the following Python pseudocode:

[0053] for b in range(B.dim):

[0054] for a in range(A.dim):

[0055] i = a * B.sdim / / A.dim

[0056] B.aw[a][b] = B.sw[i][b]

[0057] It can be seen that B.sw maps each column to a dimension of A.dim.

[0058] Among them, in step S2, in order to use GPU parallel computing to improve the speed of the affine transformation code, the data is processed as follows:

[0059] 1. When the input feature value dimension is greater than the sampling dimension and the input feature value dimension cannot be divided evenly by the sampling dimension (i.e., A.dim > B.sdim, A.dim mod B.sdim ≠ 0), a reasonable algorithm defined by oneself can be used to fill or crop the input feature value A.feature so that its dimension becomes an integer multiple of the B sampling dimension (i.e., after processing, A.dim mod B.sdim = 0);

[0060] 2. When the input feature value dimension is less than the sampling dimension, i.e., A.dim < B.sdim, a special slicing method can be used to perform an affine transformation of the compression behavior on B.sw, and the existing interfaces in PyTorch can already implement this function.

[0061] It is worth mentioning that only the sampling weight matrix will be stored in the neural network, and only the parameters of the sampling weight matrix will participate in the training.

[0062] The following gives a specific example.

[0063] B.in_dim = 8, B.sdim = 4, B.out_dim = 2

[0064]

[0065] In the above example, the dimension of B is 2, the sampling dimension is 4, and the input dimension from A is 8. During the inference process, B uses affine transformation to process the sampling weight matrix, stretching it to a shape with a height of 8, and then generates the actual weight matrix.

[0066] After generating the actual weight matrix, the remaining reasoning steps are no different from those of traditional neural networks. The actual weight matrix is ​​equivalent to the weight matrix in traditional neural networks, and calculating the eigenvalue of B is equivalent to the following formula: B.features = A.feature×B.aw.

[0067] Preferably, the neural network affine connection system applies the steps of any one of the above technical solutions of the neural network affine connection method.

[0068] The working principle of the neural network affine connection method and system disclosed in this embodiment is explained below.

[0069] Assume that there is an arbitrary layer of neural network B using AC connection mode, and B receives the output of the previous layer of neural network A. It is easy to know that the dimension of the vector input to B is A.dim.

[0070] The sampling weights and actual weights are explained below.

[0071] Unlike traditional neural networks that use a fully connected approach, B using AC will have two weight matrices: the sample weight matrix (Sample Weight Matrix, denoted as B.sw) and the actual weight matrix (Actual Weight Matrix, denoted as B.aw). The only matrix that needs to be stored is the sample weight matrix B.sw, while the actual weight matrix B.aw is calculated in real time during each forward reasoning of the neural network. That is, the parameters stored in the AC layer are only the sample weight matrix, and only the parameters of the sample weight matrix are learned, and the actual weight matrix is ​​a temporary variable temporarily generated during forward reasoning.

[0072] See attached figure Figure 2A , AC layer B only stores the sampled weight matrix. The actual weight matrix is ​​generated only when there is an eigenvalue input, and the actual weight matrix (temporary variable) is used to process (multiply) the eigenvalue

[0073] See attached figure Figure 2B , for AC layer B, there are some restrictions on the shape of its weight matrix:

[0074] 1. As with traditional neural networks, the number of columns of the sampling weight matrix must be equal to the dimension of B, that is, B.sw.size(1) = B.dim.

[0075] 2. Unlike traditional neural networks, the number of rows in the sampling weight matrix is ​​independent of the dimension of the input eigenvalues ​​of B and can be any positive integer, that is, B.sw.size(0)≠B.in_dim.

[0076] 3. In order to be able to multiply with the input eigenvalues, the requirements for the actual weight matrix shape are the same as those for the traditional neural network weight matrix shape, with B.aw.size(0)=B.in_dim, B.aw.size(1)=B.dim.

[0077] In short, the reason why AC layer B can accept eigenvalues ​​of any dimension when the shape of the stored weight matrix is ​​fixed is that when the eigenvalue is passed in, it changes the shape of its own matrix according to a certain algorithm to adapt it to the input dimension, and then uses the changed matrix to process (multiply) the input eigenvalue. The above-mentioned change algorithm is the affine transformation.

[0078] B.sw.size(0) is called the sampling dimension of B (Sampledimension, denoted as B.sdim)

[0079] The forward propagation is described below.

[0080] During the forward propagation, layer B behaves in four steps:

[0081] 1. B accepts the eigenvalues ​​passed by A and sets B.in_dim according to the dimension of A;

[0082] 2.B.sw generates B.aw through affine transformation;

[0083] 3.B uses B.aw to process the eigenvalues ​​of A;

[0084] 4.B outputs its own feature value B.feature.

[0085] Therefore, to sum up, the formula for calculating the eigenvalue in the forward propagation is: B.feature = A.feature × AF (B.sw), which can also be written as: out_feature = in_feature × AF (B.sw).

[0086] Among them, AF() is the affine transformation function, A.feature is the input feature value in_feature, and B.feature is the output feature value out_feature.

[0087] The following explains the algorithm principle of neural network affine connection.

[0088] The AC connection method is based on a hypothesis: during the reasoning process, dimensions with similar overall positions in eigenvalues ​​express similar information.

[0089] For example: If the ath dimension of the eigenvalue f (denoted as f[a]) is at the 32% position of f (i.e. ), the bth dimension is at 32.1% of f, because 32 is very close to 32.1, so we think f[a] and f[b] express similar information. Therefore, for similar information, the AC connection method uses the same weight for processing, that is, affine transformation is performed on each row of sampling weights, stretching or shrinking it to the same dimension as the input eigenvalue, and each dimensional input eigenvalue is multiplied by the weight in the corresponding sampling matrix at its location.

[0090] Therefore, we will set a fixed size sampling matrix. During inference, each row of the sampling matrix will be affine transformed to make its dimensions match the input eigenvalues. Because it uses an affine transformation, features that are around a% of the input eigenvalues ​​will be treated by weights that are also around a% on the sampling matrix. Here is a specific example

[0091] See attached figure Figure 3A and Figure 3B , the AC layer B in the figure, B.sdim = 4, B.out_dim = 3, B.in_dim = 8. After affine transformation, B.sw is transformed from the original (3,4) shape to (3,8) shape B.aw, and B.aw is used to process the input 8-dimensional eigenvalues. Figure 5 Two weight matrices of B are given.

[0092] The above example uses the weights of row 0 as an example, and the same weights are represented by the same color. It can be seen that after the affine transformation, B uses weights with similar overall positions to process its corresponding eigenvalues. The eigenvalues ​​at about one quarter are processed by the weights at one quarter, and the eigenvalues ​​at about two quarters are processed by the weights at two quarters.

[0093] The following describes the application scenarios of neural network affine connections.

[0094] See attached figure Figure 4 The AC layer is used as a plug-in for other neural networks, replacing or overwriting the original input layer, and accepting input of any dimension using the AC method. It is compatible with and improves traditional feedforward neural networks. With less additional training, the original well-functioning and trained neural network can be transformed into a neural network that can accept input of any dimension.

[0095] See attached figure Figure 5, if AC layers are also used extensively inside a neural network, each AC layer can be extracted and used separately after training. That is, the trained AC layer has a specific function of extracting specific information. Each layer of a traditional neural network should also have similar functions, but due to the limited connection method and the incompatibility of different neural network shapes and structures, neural networks with different functions in each layer can only be used in their native neural networks. The introduction of the AC connection method allows neural networks of different levels to be built like building blocks - because any AC layer can be connected to any AC layer. The AC connection method can decouple neural networks, reduce their black box nature, and the weight data trained in one network can be taken out and inserted into other neural networks for use.

[0096] It is worth mentioning that the technical features such as the specific selection of GPU involved in the patent application of this invention should be regarded as the prior art. The specific structure, working principle and possible control method and spatial layout method of these technical features can be selected by conventional choices in the field, and should not be regarded as the inventive point of the patent of this invention. The patent of this invention will not be further elaborated.

[0097] For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural network affine connection method, characterized in that: The actual weight matrix B.aw is generated by the sampled weight matrix B.sw through affine transformation during inference. The neural network affine connection method includes the following steps: Step S1: store the sampling matrix B.sw, and calculate the actual matrix B.aw in each forward inference of the neural network; Step S2: stretch or compress each column of the sampling matrix B.sw so that the shape of the sampling matrix B.sw is converted from (B.sdim, B.dim) to (B.in_dim, B.dim), perform an affine transformation on the sampling matrix B.sw in the longitudinal direction, and map the sampling matrix B.sw to an actual matrix B.aw with a different longitudinal dimension; Step S3: Perform forward propagation.

2. The neural network affine connection method according to claim 1, characterized in that: Step S3 is specifically implemented as the following steps: Step S31: B accepts the feature value transmitted by A and sets B.in_dim according to the dimension of A; Step S32: B.sw generates B.aw by executing steps S1 and S2; Step S33: B uses B.aw to process the eigenvalues ​​of A; Step S34: B outputs its own feature value B.feature.

3. The neural network affine connection method according to claim 2, characterized in that: In step S3, the eigenvalue calculation formula for forward propagation is: B.feature=A.feature×AF(B.sw); wherein AF() is an affine transformation function, A.feature is the input eigenvalue in_feature, and B.feature is the output eigenvalue out_feature.

4. The neural network affine connection method according to claim 1, characterized in that: The neural network affine connection method further includes step P1, which is located after step S3: Step P1: The AC layer acts as a plug-in for other neural networks, replacing or overwriting the original input layer, and accepting input of any dimension using the AC method.

5. The neural network affine connection method according to claim 1, characterized in that: The neural network affine connection method further includes step Q1, which is located after step S3: Step Q1: Each AC layer is extracted separately after training, and any AC layer is connected to any AC layer.

6. A neural network affine connection system, characterized in that: The step of applying the neural network affine connection method as claimed in any one of claims 1 to 5.

Images