A deep neural network plug-in and image recognition method based on attention mechanism

Abstract

The present invention relates to a deep neural network plug-in and an image recognition method based on an attention mechanism, wherein the plug-in is composed of two LSTM layers of the same size and a CNN layer of a multi-layer structure; wherein one LSTM layer is used to memorize context information and generate a mask image with significant features, and the other LSTM layer is used to realize a "glance" function and generate classification confidence; the CNN layer of the multi-layer structure is used to downsample, extract image features, and transmit context information to the LSTM unit. By using a CNN layer of a multi-layer structure, an attention mechanism is applied to guide the CNN layer of the multi-layer structure to focus on the key features of the target object in the image, eliminate secondary features and background pixels, and the recognition ability is high. Moreover, the target object is gradually recognized through multiple propagations, the main features of the object are gradually remembered, and the secondary features of the object are forgotten. Multiple backward feedbacks are performed, and the CNN layer of the multi-layer structure has an error correction function.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and image recognition, and specifically relates to a deep neural network plug-in and an image recognition method based on an attention mechanism. Background Art

[0002] At present, the deep neural network used for image recognition is mainly convolutional neural network (CNN); however, the applicant found that the existing convolutional neural network (CNN) has the following defects in the image recognition process:

[0003] 1. We know that objects in images usually only occupy a part of the entire space, or even a very small part, and in many cases, there are a large number of background pixels in the image. Many of these pixels in the image are irrelevant to the recognition target, or even interfere with the recognition target; however, for CNN, all pixels in the image have equal weight;

[0004] 2. During the recognition process, CNN can only propagate forward once. Therefore, if the perturbation in the adversarial sample is effective during this process, it is likely to cause CNN recognition errors. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a deep neural network plug-in with high recognition ability and error correction function by using a multi-layer CNN and applying an attention mechanism to guide the multi-layer CNN to focus on the key features of the target object in the image and eliminate secondary features and background pixels, as well as an image recognition method using a deep neural network plug-in based on the attention mechanism.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A deep neural network plug-in based on the attention mechanism, consisting of two LSTM layers of the same size and a multi-layer CNN;

[0008] One of the LSTM layers is used to memorize context information and generate a mask image with significant features, and the other LSTM layer is used to implement a "glance" function and generate classification confidence;

[0009] The multi-layer CNN is used for downsampling, extracting image features and transmitting context information to the LSTM unit.

[0010] Furthermore, the multi-layer CNN includes four convolutional layers; wherein the first convolutional layer is used to generate a feature vector and pass it to the LSTM unit, the second convolutional layer is used to perform secondary feature extraction on the data after dot multiplication to obtain a feature matrix containing attention information, and the fourth convolutional layer is used to output a one-dimensional vector containing multiple elements spliced ​​from a multi-dimensional feature map.

[0011] An image recognition method using a deep neural network plug-in based on an attention mechanism, using a multi-layer CNN for downsampling, extracting image features and transmitting context information to a layer of LSTM, then the LSTM layer screens, forgets, and memorizes the received image features to generate a mask image with significant features, then the multi-layer CNN performs secondary feature extraction to obtain a feature matrix containing attention information, and forms an attention mechanism with context association capabilities through cyclic transmission. Specifically, the following steps are included:

[0012] a1. Initialize LSTM, calculate the average value of the input feature vector to initialize the hidden layer state h0 and storage state c0 of the two LSTM layers;

[0013] a2. Generate a mask image. The feature vector x generated by the convolution layer of the multi-layer CNN is passed to the LSTM unit of a layer of LSTM. The feature vector x is calculated by the LSTM layer to obtain a parsed and filtered feature matrix, and a dot multiplication operation is performed with the original image to generate a feature image with a mask, so that the key features retained by the feature matrix are displayed and the features at other positions are hidden, completing the task of high-resolution information reconstruction and continuing to pass it down to the next convolution layer;

[0014] a3. Generate a mask image with significant features. First, use the multi-layer CNN to re-extract the feature map of the mask image to obtain a feature matrix containing attention information. Then transfer the feature matrix to the output gate of a layer of LSTM and address the output feature vector. On the one hand, it serves as the input of feedback. On the other hand, through back propagation, the weights of the two layers of LSTM and the multi-layer CNN are adjusted. Finally, after multiple iterations, the information of the hidden state is constantly changing, and the information of the stored state is retained. The key features of the target object in the mask image are retained, and non-key features or irrelevant features are hidden, so as to generate a mask image with significant features.

[0015] Further, the initial hidden layer state h0 and the storage state c0 of the two LSTM layers are obtained according to formula 1 and the feature vector.

[0016]

[0017] Furthermore, step a2 is specifically as follows:

[0018] First, a layer of LSTM is based on the hidden layer state h at the previous moment. t-1 The feature vector matrix x passed this time is input to the input gate, forget gate, and output gate to perform operations of formulas 2 to 8.

[0019] i t =σ(W i ·[h t-1 ,x t ]+b i ) Formula 2,

[0020]

[0021] f t =σ(W f ·[h t-1 ,x t ]+b f ) Formula 4,

[0022] o t =σ(W0·[h t-1 ,x t ]+b0) Formula 5,

[0023]

[0024]

[0025] h t =o t *tanh(c t ) Formula 8,

[0026] Among them, i t refers to the state moment of the updated information, refers to the new information that needs to be saved, t is the state matrix of the forgotten information, o t refers to the state matrix of the output information, σ refers to the S-type function, W i refers to the weight matrix of the input gate, W c is the weight matrix of the new cell, W f Refers to the weight matrix of the forget gate, W o refers to the weight matrix of the output gate, h t-1 refers to the cell output at the previous moment, x t refers to the current input, [h t-1 , x t ] refers to the merging of two input vectors, b i is the forget gate bias, b c is the new cell bias, b f is the input gate bias, bo refers to the output gate bias, h t refers to the current cell output, c t refers to the current storage (or memory) state, c t-1 It refers to the storage (or memory) state at the previous moment;

[0027] Then, the LSTM layer screens, forgets and memorizes the features to obtain the feature matrix mask after feature deconstruction and filtering. The feature matrix mask is transformed by the linear transformation formula y=xW T +b and the matrix of the fully connected output, and concatenate the fully connected matrix to convert it into a matrix A of the same size as the original image, where x is the input matrix, W is the weight matrix, and b is the bias matrix;

[0028] Next, matrix A is multiplied with the original image, that is, the dot multiplication is performed according to formula 9 to generate a feature image with a mask, so that the key features retained by the feature matrix mask are displayed and the features at other positions are hidden, completing the task of high-resolution information reconstruction and continuing to pass it down to the next convolutional layer of the multi-layer CNN.

[0029]

[0030]

[0031] Among them, matrix A is the output o of LSTM t (The matrix consists of only 0 and 1, 1 means retaining the features of the area, and 0 means forgetting the features of the area), and matrix B is the original image.

[0032] Furthermore, in step a3, the loss function is calculated by back propagation of the cross entropy loss function formula 10,

[0033]

[0034] The present invention mainly has the following beneficial effects:

[0035] The deep neural network plug-in based on the attention mechanism described in the present invention uses a multi-layer CNN and applies an attention mechanism to guide the multi-layer CNN to focus on the key features of the target object in the image and remove the secondary features and background pixels, thereby improving the recognition ability, automatically focusing on the focus of the image, having more human-like functions, being more intelligent, and gradually identifying the target object through multiple propagations. During the propagation process, the main features of the object are gradually remembered and the secondary features of the object are forgotten. Due to the multiple propagation mechanism, even if an error occasionally occurs in a certain propagation process, the context information will be considered to ignore the error, and multiple backward feedbacks are similar to a human "glance", which has an error correction function. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a structural diagram of a multi-layer CNN in a deep neural network plug-in based on an attention mechanism according to an embodiment of the present invention;

[0037] Figure 2 is a flow chart of an image recognition method using a deep neural network plug-in based on an attention mechanism according to an embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of the effect of the mask image generated in the image recognition method of the present invention;

[0039] Figure 4 It is a schematic diagram of the effect of generating a mask image with significant features in the image recognition method of the present invention;

[0040] Figure 5 It is a schematic diagram of the principle of an image recognition method using a deep neural network plug-in based on an attention mechanism described in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] The deep neural network plug-in based on the attention mechanism described in the embodiment of the present invention is composed of two LSTM layers of the same size and a CNN with a multi-layer structure.

[0043] One of the LSTM layers is used to memorize context information and generate a mask image with significant features, and the other LSTM layer is used to implement a "glance" function and generate classification confidence, where "glance" refers to the process equivalent to human eyes distinguishing objects. For example, when you see a person, you may not see it clearly at first glance, and you often need to "glance" several times to finally recognize the person;

[0044] The multi-layer CNN is used for downsampling, extracting image features and transmitting context information to the LSTM unit; Figure 1 As shown, the multi-layer CNN includes four convolutional layers; the first convolutional layer is used to generate a feature vector and pass it to the LSTM unit, the second convolutional layer is used to perform secondary feature extraction on the data after dot multiplication to obtain a feature matrix containing attention information, and the fourth convolutional layer is used to output a one-dimensional vector containing multiple elements spliced ​​from a multi-dimensional feature map.

[0045] The present invention discloses an image recognition method using a deep neural network plug-in based on an attention mechanism, wherein a multi-layer CNN is used for downsampling, extracting image features and transmitting context information to a layer of LSTM, and then the LSTM layer screens, forgets and memorizes the received image features to generate a mask image with significant features, and then the multi-layer CNN performs secondary feature extraction to obtain a feature matrix containing attention information, and forms an attention mechanism with context association capability through cyclic transmission.

[0046] like Figure 2-5 As shown in , the image recognition method of the present invention specifically includes the following steps:

[0047] Step S100. Initialize LSTM, calculate the average value from the input feature vector to initialize the hidden layer state h0 and the storage state c0 of the two LSTM layers; specifically, the initial hidden layer state h0 and the storage state c0 of the two LSTM layers can be calculated according to Formula 1 and the feature vector,

[0048]

[0049]

[0050] In LSTM, the hidden state h0 is used for short-term memory, and its information is background information or minor features, which will disappear in subsequent calculations; the storage state c0 is used for long-term memory, and the key characteristics of its information object will remain unchanged. At the beginning of the model operation, the hidden state h0 and the storage state c0 are initialized with the average value of the generated feature vector.

[0051] Step S200. Generate a mask image. The feature vector x generated by the convolution layer of the multi-layer CNN is passed to the LSTM unit of a layer of LSTM. The feature vector x is calculated by the LSTM layer to obtain a feature matrix after parsing and filtering, and a dot multiplication operation is performed with the original image to generate a feature image with a mask, so that the key features retained by the feature matrix are displayed, and the features at other positions are hidden, completing the task of high-resolution information reconstruction, and continuing to pass it down to the next convolution layer; specifically:

[0052] First, a layer of LSTM is based on the hidden layer state h at the previous moment. t-1 The feature vector matrix x passed this time is input to the input gate, forget gate, and output gate to perform operations of formulas 2 to 8 to obtain the feature vector that needs to be retained (that is, the key feature of the object).

[0053] i t =σ(W i ·[h t-1 ,x t]+b i ) Formula 2,

[0054]

[0055] f t =σ(W f ·[h t-1 ,x t ]+b f ) Formula 4,

[0056] o t =σ(W0·[h t-1 ,x t ]+b0) Formula 5,

[0057]

[0058]

[0059] h t =o t *tanh(c t ) Formula 8,

[0060] Among them, i t It means, means, b i It means;

[0061] Then, the LSTM layer screens, forgets and memorizes the features to obtain the feature matrix mask after feature deconstruction and filtering. The feature matrix mask is transformed by the linear transformation formula y=xW T +b and the matrix output by the full connection, and concatenate the matrix after the full connection to convert it to a matrix A of the same size as the original image, where x is the input matrix, W is the weight matrix, and b is the bias matrix. For example, the feature matrix mask is a 1*512 mask feature matrix, and through the linear transformation formula and the matrix of batchsize*1024 output by the full connection, and concatenate the matrix after the full connection, it is converted to a 32*32 matrix A of the same size as the original image, and W is a 512*1024 weight matrix. Since h t-1 With the participation of attention mechanism, when selecting the current feature point memory, the context information, that is, the features of the previous memory, is fully considered;

[0062] Next, matrix A is multiplied with the original image, that is, the dot multiplication is performed according to formula 9 to generate a feature image with a mask (such as Figure 3, brighter pixels are key features, i.e., the key parts of the target object, and weak pixels are features that can be discarded, i.e., the non-key or irrelevant parts of the target object), so that the key features retained by the feature matrix mask are displayed, and the features at other positions are hidden, completing the task of high-resolution information reconstruction, and continue to pass it down to the next convolutional layer of the multi-layer CNN;

[0063]

[0064]

[0065] Among them, matrix A is the output o of LSTM t (The matrix consists of only 0 and 1, 1 means retaining the features of the area, and 0 means forgetting the features of the area), matrix B is the original image;

[0066] In this process, LSTM needs to go through the operations of forget gate, input gate and output gate; among them, the forget gate is used to decide which feature information should be discarded or retained, and the information from the previous hidden state h{t-1} and the current input x_t is transmitted to the sigmoid function at the same time, and the output value is between 0 and 1; the closer to 0, the more it should be forgotten, and the closer to 1, the more it should be retained, and then the storage state of the previous moment is multiplied to obtain the state matrix of the forgotten information, and determine which storage state information of the previous moment needs to be forgotten; the input gate is used to decide which information in the current input is important and needs to be added, and the hidden layer state h{t-1} of the previous iteration and the feature vector matrix x~ut passed by this iteration are input to the input gate to update the storage state c, specifically: first, the information of the previous hidden state h{t-1} and the current input x_t is transmitted to the sigmoid function, and the result value is set between 0 and 1 to determine which features in x_t are important and need to be Updated, then the feature vector of the previous hidden state and the feature vector of the current input are passed to the tanh activation function to calculate the information stored in the cell state, and then the control signal of the input gate is multiplied by the information stored in the storage state to obtain the updated state matrix, and the previously calculated forget state matrix is ​​added to obtain the storage state c at the current moment; the output gate is used to determine the value of the next hidden state, which contains the previous information, specifically: first, the previous hidden state h{t-1} and the current input state x_t are transferred to the forget gate, and then the sigmoid function sets the value between 0 and 1 to determine the feature information to be output, and generates a state matrix of the output information, then the current storage state c_t is transferred to the tanh function, and the output of the tanh function is multiplied by the state matrix of the output result to determine the information that the hidden state h_t should carry, and finally the new storage state c_t and the new hidden state h_t are transferred to the next time step. Through the operation of the forget gate, input gate and output gate, the hidden layer outputs the feature vector matrix (i.e., a 1×512 mask matrix), and then the mask matrix is ​​linearly transformed, and then the linearly transformed mask matrix is ​​transformed into 1024 dimensions and reshaped into 32×32, which is consistent with the original image. Since the eigenvector at h{t-1} participates in the calculation, the attention mechanism takes into account the contextual information, i.e., the eigenvector of the past memory, when selecting the current eigenvector.

[0067] Step S300. Generate a mask image with significant features. First, use a multi-layer CNN to re-extract the feature map of the mask image to obtain a feature matrix containing attention information. Then transfer the feature matrix to the output gate of a layer of LSTM, and address the output feature vector (i.e., address it according to step S200). On the one hand, it is used as the input of feedback. On the other hand, through back propagation, the weights of the two layers of LSTM and the multi-layer CNN are adjusted. Finally, after multiple iterations, the information of the hidden state is constantly changing, and the information of the storage state is retained. The key features of the target object in the mask image are retained, and non-key features or irrelevant features are hidden, so as to generate a mask image with significant features. Among them, the loss function is calculated through back propagation of the cross entropy loss function formula 10.

[0068]

[0069] That is: the second convolutional layer of the multi-layer CNN performs secondary feature extraction on the data after dot multiplication to obtain a feature matrix containing attention information, and forms an attention mechanism with contextual association ability through cyclic transmission. This principle is used to perform long-term contextual memory of key features under the action of the two LSTMs attached to the first layer. The two LSTMs of the first layer simulate the memory of the context in the text field, perform context-enhanced memory on features that appear more times in the picture, forget features with lower frequency of appearance, and continuously add adjacent features related to previous memories to the memory, so that the range of high-frequency features is expanded. In this way, the picture processed by the attention mechanism with contextual association ability can be output, and the main features of the picture become clearer and clearer, such as Figure 4 As shown, as the number of iterations increases, the light and dark pixels of the mask image gradually change, and the key features of the target object gradually stand out.

[0070] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image recognition method using a deep neural network plug-in based on an attention mechanism, characterized in that: The deep neural network plug-in consists of two LSTM layers of the same size and a CNN with a multi-layer structure; One of the LSTM layers is used to memorize context information and generate a mask image with significant features, and the other LSTM layer is used to implement a "glance" function and generate classification confidence; The multi-layer CNN is used for downsampling, extracting image features and transmitting context information to the LSTM unit; The image recognition is performed using a deep neural network plug-in based on an attention mechanism. The method includes: The multi-layer CNN is used to downsample, extract image features and transmit context information to a layer of LSTM, and then the LSTM layer screens, forgets and memorizes the received image features to generate a mask image with significant features. Then the multi-layer CNN performs secondary feature extraction to obtain a feature matrix containing attention information, and forms an attention mechanism with context association capability through cyclic transmission; Specifically, it includes: generating a mask image, the feature vector x generated by the convolution layer of the multi-layer CNN is passed to the LSTM unit of a layer of LSTM, the feature vector x is calculated by the LSTM layer to obtain a parsed and filtered feature matrix, and a dot multiplication operation is performed with the original image to generate a feature image with a mask, so that the key features retained by the feature matrix are displayed, and the features at other positions are hidden, thus completing the task of high-resolution information reconstruction, and continuing to pass it down to the next convolution layer; To generate a mask image with significant features, we first use a multi-layer CNN to re-extract the feature map of the mask image to obtain a feature matrix containing attention information. Then, we transfer the feature matrix to the output gate of a layer of LSTM and address the output feature vector. On the one hand, it serves as the input of feedback. On the other hand, through back propagation, we adjust the weights of the two layers of LSTM and the multi-layer CNN. Finally, after multiple iterations, the information of the hidden state keeps changing, and the information of the stored state is retained. The key features of the target object in the mask image are retained, and non-key features or irrelevant features are hidden, so as to generate a mask image with significant features.

2. The method according to claim 1, characterized in that: The multi-layer CNN is composed of four convolutional layers, which are used to generate feature vectors and pass them to LSTM units, perform secondary feature extraction on the data after dot multiplication, obtain a feature matrix containing attention information, and output a one-dimensional vector containing multiple elements spliced ​​from multi-dimensional feature maps.

3. The method according to claim 1, characterized in that The image recognition further includes: Before generating the mask image, the LSTM is initialized, and the average value is calculated from the feature vector input by the multi-layer CNN structure to initialize the hidden layer state h0 and the storage state c0 of the two LSTM layers.

4. The method according to claim 3, characterized in that According to formula 1 and the feature vector calculation, the initial hidden layer state h0 and storage state c0 of the two LSTM layers are obtained.

5. The method according to claim 3, characterized in that: The specific steps to generate the mask image are: First, a layer of LSTM is based on the hidden layer state h at the previous moment. t-1 The feature vector matrix x passed this time is input to the input gate, forget gate, and output gate to perform operations of formulas 2 to 8 to obtain the feature vector that needs to be retained. i t =σ(W i ·[h t-1 ,x t ]+b i ) Formula 2, f t =σ(W f ·[h t-1 ,x t ]+b f ) Formula 4, o t =σ(W0·[h t-1 ,x t ]+b0) Formula 5, h t =o t *tanh(c t ) Formula 8, Among them, i t refers to the state moment of the updated information, refers to the new information that needs to be saved, t is the state matrix of the forgotten information, o t refers to the state matrix of the output information, σ refers to the S-type function, W i refers to the weight matrix of the input gate, W c is the weight matrix of the new cell, W f Refers to the weight matrix of the forget gate, W o refers to the weight matrix of the output gate, h t-1 refers to the cell output at the previous moment, x t refers to the current input, [h t-1 , x t ] refers to the merging of two input vectors, b i is the forget gate bias, b c is the new cell bias, b f is the input gate bias, b o is the output gate bias, h t refers to the current cell output, c t refers to the current storage or memory state, c t-1 It refers to the storage or memory state at the previous moment; Then, the LSTM layer screens, forgets and memorizes the features to obtain the feature matrix mask after feature deconstruction and filtering. The feature matrix mask is transformed by the linear transformation formula y=xW T +b and the matrix of the fully connected output, and concatenate the fully connected matrix to convert it into a matrix A of the same size as the original image, where x is the input matrix, W is the weight matrix, and b is the bias matrix; Next, matrix A is multiplied with the original image, that is, the dot multiplication is performed according to formula 9 to generate a feature image with a mask, so that the key features retained by the feature matrix mask are displayed and the features at other positions are hidden, completing the task of high-resolution information reconstruction and continuing to pass it down to the next convolutional layer of the multi-layer CNN. Among them, matrix A is the output o of LSTM t , the matrix is ​​composed only of 0 and 1, 1 represents retaining the features of the area, 0 represents forgetting the features of the area, and matrix B is the original image.

6. The method according to claim 3, characterized in that In the step of generating a mask image with significant features, the loss function is calculated by back propagation of the cross entropy loss function formula 10:

Images