A fast learnable large-scale image description method

Abstract

A large-scale image description method with learnable linear attention with fast normalization, comprising the following steps: S100: adopting a deep convolutional neural network to perform preliminary feature extraction on a large-scale image to be described, to obtain first feature information of the large-scale image to be described; S200: inputting the first feature information into an encoder containing learnable linear attention to perform encoding, to obtain second feature information; S300: inputting the second feature information into a decoder, to obtain a textual representation of the large-scale image to be described. The method can accelerate the inference speed of the model, so that the large-scale image description model is relieved in the processing efficiency problem, and the feature representation capability of the model is enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a large-scale image description method capable of being quickly learned. BACKGROUND

[0002] Image description is a research hotspot in computer vision and natural language processing, and is one of specific implementations of combining multi-modal (visual and language) information. Its application covers many fields, mainly including the following aspects: 1) image retrieval and search: image description application can make images have natural language description, and can be used as input to realize text-based image retrieval. For example, a large number of images can be described by using the image description application to form an image library. When a user inputs a natural language query statement, the query statement and the image description in the library can be matched to return the matched image. 2) visual aid / blind aid: image description application can be applied in visual aid or blind aid, and can help visually impaired persons to better understand the surrounding environment. For example, real-time images captured by a camera are imported into the image description application, and the system can provide the main information of the image to the blind or visually impaired persons through description. 3) visual aid for decision making: image description application can also help people to understand data more intuitively. For example, a large amount of data is visualized as an image, and then the natural language description of the data is generated by the image description application, so that people can more intuitively understand the data and be used in policy making and other fields. 4) applied in education field: image description application can also be applied in the education field, for example, machines can automatically describe some graphical content in academic papers to provide some references for teaching. In addition, it can also help students to better understand images, and for job seekers, it is also a way to show comprehensive quality. In summary, image description application has very wide application scenarios and has important influence in many fields.

[0003] The purpose of image description is to capture the semantic content of the input image and then generate a corresponding sentence, which has become a basic task of visual understanding. However, the current Transformer-based image description model faces certain challenges in generating description sentences for large-scale images, for example, if the input picture resolution is too large, the calculation amount increases in quadratic.

[0004] The mainstream framework of image understanding model is based on the architecture of encoding-decoding, but this early model cannot be calculated in parallel, and the small input resolution (input size of 224x224) limits the performance of image description model. When the input resolution of the model increases, the processing speed of the model is slow. In order to solve the model operation speed, the image description model based on Transformer is used. Since Transformer can be calculated in parallel to speed up the processing speed of the model, and the input size is increased to 384x384, the model has superior performance in performance. The above indicates that the size of the resolution has an impact on the performance of the image description model. In other visual fields, it is also indicated that a larger resolution brings better performance, for example, experiments are carried out on 384x384, 448x448, 576x576 and 768x768 different scales in the classification task, and the experimental results prove that the model has better performance on a larger scale. Although the model based on Transformer has fast processing speed and good performance, it also has defects, for example, the problem of large amount of calculation of the model in other visual fields, which also exists when processing large-scale images in the image description field. This paper also aims to solve the problem of large amount of calculation of the image description model in processing large-scale images.

[0005] With the success of Transformer architecture in natural language processing, due to its self-attention that can capture long-term dependencies between sequences, it has been widely used in computer vision. GasHis-Transformer is used for detection of medical cancer cells, and Autoformer also shows advanced performance in visual recognition field. Transformer has achieved good results in tracking field. Many recent works have also studied its application in image description task and achieved good performance. However, many current works find that self-attention calculation has a quadratic complexity problem in image fields such as classification, detection and segmentation, of course, similar problems also exist in image description task, especially when describing large-scale images, this problem is more prominent. At the same time, the standard Transformer also has certain defects in feature representation, and with the increase of model depth, the long-term dependency between layers will be weakened. Similarly, our baseline model also has such defects, and the loss or loss of shallow layer fine-grained information will affect the performance of the model, and the lack of global richer semantic information of the picture. SUMMARY

[0006] In order to solve the problem of large amount of calculation of large-scale image description model and enhance the feature representation ability of the model, the application discloses a large-scale image description method with fast normalization and learnable linear attention, which comprises the following steps:

[0007] S100: a deep convolutional neural network is used to preliminarily extract features of a large-scale image to be described, and first feature information of the large-scale image to be described is obtained;

[0008] S200: the first feature information is input into an encoder containing learnable linear attention for encoding, and second feature information is obtained;

[0009] S300: the second feature information is input into a decoder, and a textual representation of the large-scale image to be described is obtained.

[0010] In order to solve the problem of limited processing of large-scale images in the field of image captioning, an efficient image description model is used. The model includes a learnable linear attention (Learnable Linear-Attention), and the hidden relationship between data can be mined to improve the feature representation capability of the model, while greatly reducing the calculation and parameter cost, reducing the calculation amount by 30% and the parameter amount by 20%. At the same time, a fast normalization method is used to speed up the processing speed of the model. In addition, the inter-block residual connection is used to make up for the loss of shallow fine-grained information with the increase of network depth, and the feature representation capability of the model is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a large-scale image description method flowchart with fast normalization of learnable linear attention provided in an embodiment of the present application;

[0012] Figure 2 is a large-scale image description method overall network structure diagram with fast normalization of learnable linear attention provided in an embodiment of the present application;

[0013] Figure 3 is a self-attention and linear learnable attention comparison diagram provided in an embodiment of the present application;

[0014] Figure 4 is a residual module comparison diagram provided in an embodiment of the present application;

[0015] Figure 5 is a schematic diagram of generating a diagnostic report using the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order for those skilled in the art to understand the technical solutions disclosed in the present application, the following will be combined with the embodiments and related drawings Figures 1 to 5 The technical solutions of various embodiments are described, and the described embodiments are part of the embodiments of the present application, but not all embodiments.

[0017] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all referring to a common set of embodiments, although they can. As used herein, the term“or” as used herein, without additional context, can mean either / or (i.e., the logical OR) and / or the inclusive OR (i.e., the logical OR, or any combination of the items named). Generally, the term“or” as used herein means any one member of a particular list and also any combination of members of the group. As used herein, the term“based on” can mean“based at least in part on” and / or“based at least in part on.” As used herein, the term“exemplary” is used in the sense of an example instance rather than an ideal instance. Those of skill in the art will understand that the embodiments described herein can be combined with other embodiments.

[0018] Referring to Figure 1 In one embodiment, the present application discloses a large-scale image description method with learnable linear attention normalization, comprising the following steps:

[0019] S100: using a deep convolutional neural network to perform preliminary feature extraction on a large-scale image to be described, to obtain first feature information of the large-scale image to be described;

[0020] S200: inputting the first feature information into an encoder containing learnable linear attention to perform encoding, to obtain second feature information;

[0021] S300: inputting the second feature information into a decoder to obtain a textual representation of the large-scale image to be described.

[0022] For this embodiment, to solve the problem of large amount of Transformer calculation, the features can be divided into groups, and the self-attention is calculated within the group to reduce the calculation amount of the original Self-Attention. The Key and Value in Self-Attention are set to be learnable to reduce the calculation amount. For the description task of large-scale images, a learnable linear attention image description method with fast fusion is proposed for large-scale image captioning, which is abbreviated as LLAFF-Generator. The model first uses a linear attention to alleviate the problem of Self-Attention calculation complexity. The attention has linear complexity, and the learnable linear attention is implemented by using two small, learnable memories. The key to the lightweight characteristics of linear learnable attention is that the number of elements in the memory is much smaller than the number of input features, resulting in a linear number of input elements. The external memory is used to learn the most distinctive features in the entire dataset, capture the most informative part, and exclude interference information from other samples. A similar idea can also be found in sparse coding or dictionary learning. At the same time, Fast-Normlization is used inside the linear attention, and the Softmax fusion method in self-attention also affects the calculation amount of the Transformer. A fast normalization method suitable for our model is used to further reduce the calculation amount of the model. The fast normalization method first maps the pixel values to values greater than zero, and then calculates the attention score. Further, a feature enhancement module is used to alleviate the problem of missing fine-grained features with increasing depth in the Transformer. To address the shortcomings of visual Transformer models in feature representation, we compensate for the loss or loss of shallow semantic information during the increase in depth, making the model's feature representation more suitable for our downstream tasks, thereby improving the model's performance. The change in our model structure does not bring too much calculation amount and parameter amount.

[0023] The method is based on the CNN-Transformer structure, and proposes a large-scale image description model with learnable attention with fast normalization. First, the method uses learnable linear attention with fast normalization to reduce the calculation amount of the model and speed up the inference speed of the model, so that the efficiency problem of large-scale image description model is alleviated. At the same time, the inter-block residual connection is used to compensate for the loss of shallow fine-grained features in visual Transformer with increasing network depth, and to enhance the feature representation capability of the model.

[0024] Based on the Transformer algorithm, the overall network structure of the method is as shown in the figure. Figure 2 We use the architecture of CNN+Transformer, our model uses the deep convolutional neural network ResNet-101 to extract bottom-up visual features, due to the great progress of Transformer in machine translation and computer vision, we use the standard visual Transformer to establish the dependence between sequences. Although the Transformer has better performance in computer vision, the computational complexity grows quadratically when processing long sequences, which hinders the development of Transformer in the field of computer vision, and solving the efficiency of Transformer has become an urgent problem, we propose a learnable linear attention image description model with fast fusion to alleviate the problem of processing large-scale images in the field of image description.

[0025] First, the picture is processed into the data format of the model input, and then input into the backbone network (Backbone) ResNet101 for feature extraction. The backbone network (Backbone) can also be replaced by other feature extraction networks, and then input into the Transformer encoder for encoding. The encoder is used to establish the dependence between pixels and enhance the feature representation ability of the model. The encoder consists of a learnable linear attention (Learnable Linear-Attention), a feedforward network (Feedforward) and layer normalization (LayerNorm). Then input into the decoder, the decoder generates a description consistent with the picture information according to the visual feature information. The decoder consists of self-attention (Self-Attention), cross-attention (Cross-Attention), a feedforward network (Feedforward) and layer normalization (LayerNorm).

[0026] Large-scale images generally refer to images with a size of hundreds to tens of thousands of pixels. The current image description model uses image scales of 224*224 or 384*384, and our model scales can be 384*384, 448*448, 576*576 and 768*768 respectively.

[0027] The training process of the neural network is as follows:

[0028] Step one: First, preprocess the data, crop the picture to the size we need (for example: 384*384, 448*448, 576*576, 768*768), and convert it to a format that the neural network can accept.

[0029] Step two: weight initialization, before the training starts, the weights and biases of the neural network need to be randomly initialized;

[0030] Step three: forward propagation, using the model that has been randomly initialized, the output of the neural network is calculated by inputting the features; the loss function is calculated, and the loss function is calculated according to the predicted results of the model and the label;

[0031] Step four: back propagation, according to the loss function calculation result, the model weight and bias are updated by back propagation algorithm to minimize the loss function; update the weight, according to the weight gradient calculated by back propagation, use the optimizer Adam to update the weight and bias of the model;

[0032] Step five: repeat the above steps until the preset iteration number is reached or the loss function reaches a certain convergence degree; use the data set that does not participate in the training to verify and test the model performance. The judgment standard is to reach the preset iteration number, and there is corresponding improvement compared with the benchmark algorithm.

[0033] Steps three to five are the training process of the whole model. Our training is divided into two stages. The initial learning rate of the first stage is 2e-5. On the basis of the first stage, the best weight is selected (you can refer to the loss graph. The number of rounds when the loss value is close to smooth, the vertical axis of the loss graph is the average loss of each round, and the horizontal axis is the training round). The second stage loads the best weight of the first stage, and the initial learning rate is half of the first stage. Finally, the best weight is selected to test the model performance (you can refer to the loss graph. The number of rounds when the loss value is close to smooth, the vertical axis of the loss graph is the average loss of each round, and the horizontal axis is the training round).

[0034] In another embodiment, the encoder further comprises layer normalization and a feedforward network.

[0035] In another embodiment, the decoder comprises self-attention, cross-attention, a feedforward network and layer normalization.

[0036] In another embodiment, the learnable linear attention is specifically: the output of the backbone network is taken as the query of self-attention, the key and value are obtained by setting the learnable parameters, which simplifies the multiplication of the transposed query matrix and the key matrix, and then a fast normalization function is used to generate the weight, and finally the learned value matrix is multiplied to output.

[0037] For this embodiment, the encoding layer is used to encode the image features extracted by CNN, establish the correlation between sequences, and our encoding layer mainly consists of a learnable linear attention, a normalization layer and a small feedforward network. First, the original picture is subjected to feature extraction by the feature extraction network RestNet-101, and the feature extraction network can also be replaced by other backbone networks, then it is subjected to the encoder, which consists of a learnable linear attention, a normalization layer and a feedforward network layer, and finally it is input into the decoder. We mainly use a learnable linear attention in this part, and its structure is as shown in Figure 3 (b) shown.

[0038] Figure 3 The original self-attention structure in (a) takes the output of the backbone network as the query, key and value of the self-attention. The product of the transpose of the query matrix and the key matrix is subjected to a Softmax function to generate a weight between 0 and 1, and then multiplied by the value matrix, and then input into the Transformer decoder. Figure 3 The fast learnable linear attention in (b) takes the output of the backbone network as the query, key and value of the self-attention (since the fast learnable linear attention is an improvement of the self-attention, by setting the key and value of the self-attention as learnable parameters, the improved self-attention is a fast learnable linear attention, which is a self-attention here), which are obtained by setting learnable parameters. Simplify the product of the transpose of the query matrix and the key matrix, then pass through a fast normalization function to generate a weight, and finally multiply it by the learned value matrix and input it into the Transformer decoder.

[0039] Self-Attention: Standard self-attention, given an input F N ×d from a visual feature extraction network, N represents the number of input features (generally represented as N = w x h), d represents the number of feature channels, and the feature is converted into a matrix, a matrix, a matrix, and the self-attention can be expressed as:

[0040]

[0041] A N×N represents the attention matrix score, and a represents the attention score between the i-th pixel and the j-th pixel.

[0042] The self-attention can be simplified as:

[0043] A = (a) i，j = Softmax(FF T )

[0044] Attention is obtained by computing the similarity between pixels, and the output is an exact representation of the input.

[0045] Although the attention is simplified, the computational complexity O(N 2 ) is still a major drawback of attention, and self-attention pays more attention to the relationship within the sample, ignoring the potential relationship between samples, so we introduce a learnable linear attention to alleviate this problem.

[0046] Learnable linear attention: Linear attention computes the attention between input pixels and external memory unit M ∈ R N×d by:

[0047] Attention = (a) i，j = Softmax(FM T ) = A

[0048] F out = AM

[0049] Unlike standard self-attention, a i，j in A = (a) T = Softmax(FF i，j ) is the attention score between the i-th pixel of M and the j-th row, where M is a learnable parameter independent of the input, as a memory of the entire training data set. A is the attention map inferred from the learned data set prior knowledge, and finally we update the input features of M through the similarity in A.

[0050] Two different memory units M k and M v are used as keys and values to improve the ability of the network. This changes the way linear attention is calculated.

[0051]

[0052] F out = AM v

[0053] The computational complexity of this attention is O(dSN); since d and S are hyperparameters, the algorithm is linear in the number of pixels. Therefore, learnable linear attention is more efficient than self-attention, allowing it to be directly applied to large-scale inputs. We also note that the computational load of learnable linear attention is roughly equivalent to a 1x1 convolution.

[0054] In another embodiment, the normalization is LayerNorm normalization.

[0055] For this embodiment, we tried two normalization methods in the learnable linear attention, using BatchNorm and LayerNorm respectively, and found that LayerNorm works better. We analyzed that in the image description field, global information of the image is very important when describing a picture, whether it is background information or target information. In a specific scenario, target information can be inferred from background information, while other tasks may focus more on the positioning and attributes of the target. Therefore, we use LayerNorm normalization method, and when initializing the weight, we want to ensure that the output features of this layer in the initial state are largely maintained, so that the training process is more stable and effective.

[0056] The fast normalization function simplifies the softmax function and improves the inference speed of the model. LayerNorm is a normalization technique that normalizes each feature point of each sample in a batch to make the average value of each feature point 0 and the variance 1. This makes the training process more stable and effective.

[0057] In another embodiment, the fast normalization method is represented by the formula:

[0058] For this embodiment, W i is a pixel point in the feature map, i takes a value of 1-n, and n is the number of pixel points in the feature map. W j is a pixel point in the feature map, j takes a value of 1-n, and n is the number of pixel points in the feature map. I i is a pixel point in the feature map, i takes a value of 1-n, and n is the number of pixel points in the feature map. When calculating the weight, a pixel value W i is divided by the sum of all pixel values W j (j takes a value of 1-n, and n is the number of pixel points in the feature map), to get a value between 0 and 1, and then multiplied by I i , and so on to get the weighted feature map of the entire feature map.

[0059] Softmax fusion method: In the original work, Transformer uses the Softmax fusion method to calculate the weight of sequence correlation when calculating the weight of sequence correlation. The value is between 0 and 1, representing the importance of different pixels in the sequence. Its formula can be represented as:

[0060]

[0061] As can be seen, Softmax introduces more computational overhead. To improve the computational speed of the model on large-scale images, we propose a fast fusion method.

[0062] Fast Normalization: The efficiency limitations of Transformer stem not only from self-attention but also from the Softmax algorithm used to calculate attention scores. We address this issue using a fast fusion method, which can be expressed as:

[0063]

[0064] Attention can then be further expressed as:

[0065]

[0066] By each w i Then a Softplus function is applied to ensure w i The values ​​are ≥0, with each weight ranging from 0 to 1. Since Softmax is not used, this method is more efficient. Ablation experiments show that this method has very similar learning behavior and performance to Softmax, but is more efficient.

[0067] In another embodiment, the learnable linear attention further includes inter-block residual connections that fuse the output and input of the previous layer as the input of the current layer and input the last layer result of the encoder into the decoder.

[0068] In this embodiment, as Figure 4 As shown in (i), the standard Transformer network structure is formed by stacking N layers of encoders. The output of the previous layer is the input of the current layer, and the result of the last layer of the encoder is input into the decoder. This structure only uses the last layer of the encoder to decode and ignores the shallow, fine-grained information. Figure 4 (ii) is our inter-block residual connection network structure. We fuse the output and input of the previous layer as the input of the current layer and input the last layer result of the encoder into the decoder, which makes up for the shallow fine-grained information lost as the network depth increases.

[0069] With the increase of network depth, the semantic information of low layers will be weakened, but as the outstanding contribution of ResNet network, the reasonable use of residual block in most current deep learning explicitly and effectively enhances the expression of visual features. Therefore, in the linear attention learning module we proposed, in order to make the hidden layer semantic information more rich, generate more detailed description sentences, such as Figure 4 (ii) The corresponding residual connection is introduced.

[0070] In another embodiment, the inter-block residual connection is expressed by the formula: Where X represents the input of each sub-layer of the Transformer, EncoderLayer(·) represents the complete operation of each sub-layer of the Transformer, and the symbol represents element addition, and l represents the first layer of the Transformer encoding layer.

[0071] For this embodiment, specifically, in our network, the input and output of the ith layer are fused to serve as the input of the i+1th layer, which can be expressed by the formula:

[0072]

[0073] X represents the input of each sub-layer of the Transformer, EncoderLayer(·) represents the complete operation of each sub-layer of the Transformer, and the symbol represents element addition. We do not directly fuse the features of the lowest layer with the features of the highest layer, but adopt a layer-by-layer fusion method, because fusing the features of the lowest layer into the relatively higher layer will bring noise to the high layer information.

[0074] In another embodiment, the deep convolutional neural network in step S100 is ResNet-101.

[0075] For this embodiment, when judging whether the model is terminated, the loss curve of the model can be observed. When the loss curve is smooth and stable, it means that the model has converged, and at this time the weight of the model is the best value.

[0076] In another embodiment, most of the current hospital medical image diagnosis reports require doctors to manually produce medical image report results, which is an additional workload for doctors, and the efficiency of manual processing is low. Therefore, we propose a large-scale image description method with fast normalization and learnable linear attention, which can be applied to medical image diagnosis reports. This method can automatically generate descriptions that conform to the content of the picture according to the image content. Doctors can give the final diagnosis result according to the report content, improving work efficiency. For example, Figure 5As shown, the medical image is extracted by the backbone network, enhanced by the encoder, and finally the decoder produces a descriptive sentence according to the image content. For example Figure 5 The results are as follows: no pleural effusion or pleural effusion, no need to consolidate key areas, and small T-shaped spiny osteophytes. Our model has small calculation amount and fast processing speed, so it can process large-scale medical images to produce more accurate descriptions, greatly improving the work efficiency of doctors and reducing the workload of doctors.

[0077] Finally, it should be noted that many forms of variation can be made by those skilled in the art under the inspiration of this specification and without departing from the scope protected by the claims of the present application, and these all belong to the protection of the present application.

Claims

1.A large-scale image description method with learnable linear attention and fast normalization, the method comprising the following steps: S100: performing preliminary feature extraction on a large-scale image to be described by using a deep convolutional neural network to obtain first feature information of the large-scale image to be described; S200: inputting the first feature information into an encoder containing learnable linear attention to encode and obtain second feature information; S300: inputting the second feature information into a decoder to obtain a textual representation of the large-scale image to be described; wherein, the learnable linear attention is specifically: the output of a backbone network is taken as the query of self-attention, the key and the value are obtained by setting as learnable parameters, which simplifies the multiplication of the transposed query matrix and the key matrix, then the weight is generated through a fast normalization function, and finally the multiplication of the weight and the learned value matrix is outputted; the normalization is LayerNorm normalization; The fast normalization is expressed by a formula as follows: , W i is a pixel point in the feature map, i is 1-n, and n is the number of pixel points in the feature map; W j is a pixel point in the feature map, j is 1-n, and n is the number of pixel points in the feature map; I i is a pixel point in the feature map, i is 1-n, and n is the number of pixel points in the feature map; when the weight is calculated, a pixel value W i is divided by the sum of all pixel values W j to obtain a value between 0 and 1, which is multiplied by I i , and the weighted feature map of the entire feature map is obtained in this way. the learnable linear attention further comprises an inter-block residual connection, which fuses the output of the previous layer and the input as the input of the current layer, and inputs the result of the last layer of the encoder into the decoder; The inter-block residual connection is expressed by the formula: where X represents the input of each sub-layer of the Transformer, represents the complete operation of each sub-layer of the Transformer, the symbol represents element addition, l represents the first l layer of the Transformer encoding layer. 2.The method of claim 1, wherein the encoder further comprises LayerNorm and a feedforward network. 3.The method of claim 1, wherein the decoder comprises self-attention, cross-attention, a feedforward network and LayerNorm. 4.The method of claim 1, wherein the deep convolutional neural network in step S100 is ResNet-101.

Images