An image labeling method based on local feature enhancement and parallel decoder

Abstract

The application discloses an image labeling method based on local feature enhancement and parallel decoder, and comprises the following steps: extracting image features from target picture data; inputting the image features into a pre-trained image labeling model; the image labeling model comprises an encoder module of local feature enhancement, a word group prediction module and a decoder module of parallel word prediction; and outputting image labeling of the target picture data. Based on the image labeling model, the encoder module of local feature enhancement, the word group prediction module and the decoder module of parallel word prediction constructed by the image labeling model can effectively capture the relationship between image features and input labeling, generate image labeling, and thus improve the speed and flexibility of image labeling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image annotation method based on local feature enhancement and parallel decoder. BACKGROUND

[0002] Image annotation has attracted more and more attention due to its wide application prospects in early childhood education, blind aid and intelligent navigation. Meanwhile, in image annotation, accuracy and speed are two key factors. However, due to different ways of generating annotations, it is not easy to generate annotations that take into account both indicators. Therefore, designing a fast and small error image annotation method is still a challenge. Existing image annotation methods can be divided into two directions, one direction is a self-recurrent decoder, and the other direction is a non-self-recurrent decoder. The self-recurrent decoder generally adopts a word-by-word generation method, in which the word to be predicted needs to rely on the previously generated sentence and image features as input, and only one word can be generated each time. The other branch is a non-self-recurrent decoder, which only needs to input image features to generate all annotations at once. The self-recurrent decoder has the disadvantage of very high computational complexity, and the non-self-recurrent decoder can quickly generate annotations, but the accuracy is low, and there are problems such as repeated annotations.

[0003] Therefore, designing a fast image annotation model and reducing errors is still an urgent task in the field of image annotation. In recent years, some effective algorithms have been proposed, but they still have the following problems. First, the local features between image features are often ignored and are not usually effectively combined with global features. Second, the generation process of annotations needs to delete the generated repeated word groups, which usually brings additional time overhead. Finally, the flexibility of the annotation generation process is also important, and some methods have achieved good results in recent years, but each step can only generate a fixed length of word group. Therefore, based on the existing image annotation method, how to quickly and flexibly generate accurate annotations has become a problem that needs to be solved by those skilled in the art. SUMMARY

[0004] The purpose of the present application is to provide an image annotation method based on local feature enhancement and parallel decoder, which solves the problem of being unable to quickly and flexibly generate annotations for images in the prior art.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0006] The present application provides an image annotation method based on local feature enhancement and parallel decoder, comprising the following steps:

[0007] S1, extracting image features from target picture data;

[0008] S2, input the image features into a pre-trained image annotation model; the image annotation model comprises an encoder module with local feature enhancement, a word group prediction module, and a decoder module for predicting words in parallel;

[0009] S3, output the image annotation of the target picture data.

[0010] Further, the image annotation model training process in step S2 comprises:

[0011] S21, extract image features from training picture data, perform syntactic analysis on the real annotation corresponding to the training picture data, divide the real annotation into different word groups, and construct a sample data set required for training;

[0012] S22, input the image features into the encoder module with local feature enhancement, use multi-head self-attention mechanism and residual network to extract global and local features between the image features respectively, and fuse the global and local features;

[0013] S23, divide the annotation to be predicted into k word groups, predict the number of words in the kth word group through the features output by the encoder, and according to the word groups divided in step S21, combine whether the word at the current position in the annotation needs to participate in the prediction of the next word group, and design a mask matrix for the decoder;

[0014] S24, in the decoder module for predicting words in parallel, through the features output by the encoder and the corresponding real annotation, combine the mask matrix in step S23, predict the vector of each word in each word group, and finally obtain the output vector of the decoder;

[0015] S25, according to the output vector of the decoder, for each position word, use the word with the maximum prediction probability to represent the word at the current position, and connect all the words to form the final result as the predicted image annotation of the training picture data;

[0016] S26, generate a total loss function by calculating the cross-entropy loss between the predicted image annotation of the training picture data and the corresponding real annotation, and the mean square error between the number of words in the k predicted word groups and the number of divided words; train the image annotation model according to the total loss function.

[0017] Further, the step S1 comprises:

[0018] The Faster-RCNN with the backbone network ResNet101 is used to obtain image features from the target picture data.

[0019] Further, in the step S22, a global feature between image features is extracted by using a multi-head self-attention mechanism.

[0020] GF(X) = Attention(XW q ,XW k ,XW v ) (1)

[0021] In formula (1), X represents an image feature; GF(X) represents a global feature; Attention represents a self-attention operation, W q ,W k ,W v respectively represent a query, a key and a value weight matrix.

[0022] Further, in the step S22, a local feature between image features is extracted by using a residual network:

[0023] LF(X) = δ(BN(Conv(X) + X)) (2)

[0024] In formula (2), LF(X) represents a local feature calculated by a residual network; Conv and BN respectively represent a convolution and a batch normalization operation, and δ represents a Relu activation function.

[0025] Further, in the step S22, the global and local features are fused, including:

[0026] 1) initially fusing the global and local features:

[0027] X' = GF(X) + LF(X) (3)

[0028] In the above formula, X' represents a global feature enhanced by a local feature;

[0029] 2) secondary fusion:

[0030] Y = LayerNorm(X' + FFN(X')) (4)

[0031] In the above formula, Y represents an output of the encoder; LayerNorm represents a layer normalization operation, and FFN represents a feedforward neural network.

[0032] Further, in the step S23, the number of words of the kth word group is predicted by using the feature output by the encoder, including:

[0033] The number of words of the kth word group is predicted by using the feature output by the encoder according to the following formula:

[0034] s k = δ(Y'W l ) (5)

[0035] In the above formula, s k represents the number of words in the kth word group; δ represents the Relu activation function; Y' represents a one-dimensional vector obtained after Y is unfolded; W l represents a projection matrix, which projects Y' to a specified dimension size.

[0036] Further, in the step S23, the mask matrix of the decoder is designed, comprising:

[0037] The mask matrix in the modified attention mechanism is replaced with the mask matrix of the decoder.

[0038]

[0039] In the above formula, M ij represents the mask matrix of the modified decoder, i represents the ith word of the matrix, j represents the jth word of the matrix, allow represents that the current word participates in prediction, and disallow represents that the similarity between the ith word and the jth word is -∞, i.e. the current word does not participate in prediction.

[0040] Further, the step S24 comprises:

[0041] S241, the real label is subjected to word vector embedding and position coding, and the sum thereof is taken as a label vector and input into the decoder;

[0042] S242, the label vector is input into the first multi-head self-attention module by using the mask matrix of the modified decoder, to obtain a new label vector;

[0043] S243, the new label vector is taken as a query vector, the features output by the encoder are taken as keys and values, and the query vector, the keys and the values are input into the second multi-head self-attention module; the output of the multi-head self-attention module is taken as the input of the feedforward neural network;

[0044] S244, the output of S243 is taken as the input of the next encoding layer, and the steps S242-S243 are repeatedly executed for a preset number of times, to obtain an output vector of the decoder.

[0045] Further, the overall loss function in the step S26 is:

[0046]

[0047]

[0048] (7) In the formula, L word represents the cross-entropy loss between the predicted word and the real word, and L lendenotes the mean square error between the predicted word group and the real word group; T denotes the length of the real label, and denotes the parameters of the model, denotes the predicted word, denotes the real label, p θ denotes the probability of predicting on the basis of , k denotes the number of word groups, and s i denotes the number of words in the predicted word group, denotes the number of words in the real word group;

[0049] (8) In the formula, L group denotes the cross-entropy loss between the predicted word group and the real word group, S i denotes the currently predicted word group, S <i denotes the first i-1 word groups participating in, w ij denotes the jth word of the ith word group, p θ denotes the probability of predicting the word w <i on the basis of S ij , and the model is trained by continuously reducing the loss between the predicted word and the real labeled word.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] The image labeling method based on local feature enhancement and parallel decoder provided by the embodiment of the present application comprises the following steps: extracting image features from target picture data; inputting the image features into a pre-trained image labeling model; the image labeling model comprises an encoder module of local feature enhancement, a word group prediction module and a decoder module of parallel predicted words; and outputting the image labeling of the target picture data. Based on the image labeling model, the encoder module of local feature enhancement, the word group prediction module and the decoder module of parallel predicted words constructed by the image labeling model can effectively capture the relationship between the image features and the input label, generate image labeling, and thus improve the speed and flexibility of image labeling. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The flowchart of the image labeling method based on local feature enhancement and parallel decoder provided by the embodiment of the present application;

[0053] Figure 2 The training process flowchart of the image labeling model provided by the embodiment of the present application;

[0054] Figure 3 The principle diagram of the image labeling method based on local feature enhancement and parallel decoder provided by the embodiment of the present application. DETAILED DESCRIPTION

[0055] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0056] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0057] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0058] Reference Figure 1 As shown, the present invention provides an image annotation method based on local feature enhancement and parallel decoder, comprising the following steps:

[0059] S1. Extract image features from the target image data;

[0060] S2. Input the image features into the pre-trained image annotation model; local feature enhancement encoder module, word group prediction module, and parallel word prediction decoder module;

[0061] S3. Output the image annotation of the target image data.

[0062] In the step S1, the image features can be obtained from the target picture data by using a Faster-RCNN with a backbone network of ResNet101; the image annotation model in the step S2 comprises a local feature enhanced encoder module, a word group prediction module and a parallel word prediction decoder module; the local feature enhanced encoder module is used for fusing global and local features between the image features; the word group prediction module is used for predicting the number of words in the kth word group; the parallel word prediction decoder module is used for iteratively processing the fused features to obtain an output vector, and for each position word, the word with the maximum prediction probability is used to represent the word at the current position, and all the words are connected to form a predicted image annotation; and the step S3 outputs the predicted image annotation. The method can effectively capture the relationship between the image features and the input annotation, generate the image annotation, and thus improve the speed and flexibility of the image annotation.

[0063] In one embodiment, referring to Figure 2 the image annotation model training process of the step S2 comprises the following steps:

[0064] S21, extracting image features from training picture data, performing syntax analysis on real annotations corresponding to the training picture data, dividing the real annotations into different word groups, and constructing a sample data set required for training;

[0065] S22, inputting the image features into a local feature enhanced encoder module, respectively extracting global and local features between the image features by using a multi-head self-attention mechanism and a residual network, and fusing the global and local features;

[0066] S23, dividing the annotations to be predicted into k word groups, predicting the number of words in the kth word group by using the features output by the encoder, designing a mask matrix of the decoder according to the word groups divided in the step S21 and whether the word at the current position in the annotation needs to participate in the prediction of the next word group;

[0067] S24, in the parallel word prediction decoder module, predicting the vector of each word in each word group by using the features output by the encoder and the corresponding real annotation, and combining the mask matrix in the step S23, to finally obtain an output vector of the decoder;

[0068] S25, according to the output vector of the decoder, for each position word, using the word with the maximum prediction probability to represent the word at the current position, and connecting all the words to form a final result as the predicted image annotation of the training picture data;

[0069] S26, generating a total loss function by calculating cross-entropy loss of predicted image labels of the training picture data and corresponding real labels, and mean square error between predicted word numbers of the k word groups and divided word numbers; training the image labeling model according to the total loss function.

[0070] In step S21, corresponding image features can also be obtained from a large amount of training picture data by using Faster-RCNN with a backbone network of ResNet101; a labeling is parsed using a deep learning method, and the labeling is divided into different word groups according to a subject-predicate-object structure and the like; a sample data set required for training is constructed;

[0071] In steps S22-S24, in the local feature enhancement module, global and local features are extracted respectively using multi-head self-attention mechanism and residual network, and the two features are fused using a feedforward neural network; the step S22 is repeatedly executed for a preset number of times; features extracted by the last execution of the step S22 are taken as input to predict the word number of the kth word group, the mask matrix is modified according to the word number of the word group divided in S21, and the mask matrix in the decoder is replaced; through the features extracted by the last execution of the step S22 in S23 and the real labeling corresponding to the picture, the vector of each word in each word group is predicted, and finally the output vector of the decoder is obtained.

[0072] In the above step S22, the multi-head self-attention mechanism is used to extract global features between image features:

[0073] GF(X)=Attention(XW q ,XW k ,XW v ) (1)

[0074] In formula (1), X represents image features; GF(X) represents global features; Attention represents self-attention operation, W q ,W k ,W v respectively represent weight matrices of queries, keys and values.

[0075] The residual network is used to extract local features between image features:

[0076] LF(X)=δ(BN(Conv(X)+X)) (2)

[0077] In formula (2), LF(X) represents local features calculated by the residual network; Conv and BN respectively represent convolution and batch normalization operations, and δ represents a Relu activation function.

[0078] Fusing global and local features includes:

[0079] 1) Primary fusion of the global and local features:

[0080] X' = GF(X) + LF(X) (3)

[0081] In the above formula, X' represents the global feature enhanced by the local feature;

[0082] 2) Secondary fusion:

[0083] Y = LayerNorm(X' + FFN(X')) (4)

[0084] In the above formula, Y represents the output of the encoder; LayerNorm represents the layer normalization operation, and FFN represents the feedforward neural network.

[0085] In the above step S23, the number of words in the kth word group is predicted by the features output by the encoder, including:

[0086] The number of words in each word group is predicted by the features output by the encoder using the following formula:

[0087] s k = δ(Y'W l ) (5)

[0088] In the above formula, s k represents the number of words in the kth word group; δ represents the Relu activation function; Y' represents a one-dimensional vector obtained by unfolding Y; W l represents a projection matrix that projects Y' to a specified dimension size.

[0089] In step S23, the mask matrix of the decoder is designed, including:

[0090] The mask matrix in the attention mechanism is modified to replace the mask matrix of the decoder:

[0091]

[0092] In the above formula, M ij represents the modified mask matrix of the decoder, i represents the ith word of the matrix, j represents the jth word of the matrix, allow represents that the current word participates in prediction, and disallow represents that the similarity between the ith word and the jth word is -∞, i.e. the current word does not participate in prediction.

[0093] The above step S24 includes:

[0094] S241, word vector embedding and position coding are performed on the real label, and the sum is taken as a label vector and input into the decoder;

[0095] S242, input the label vector into the first multi-head self-attention module by using the mask matrix of the modified decoder, and obtain a new label vector;

[0096] S243, input the new label vector as a query vector, the features output by the encoder as keys and values into the second multi-head self-attention module; the output of the multi-head self-attention module is input into a feedforward neural network as an input of the feedforward neural network;

[0097] S244, input the output of S243 into the next encoding layer as an input, and repeat steps S242-S243 for a preset number of times, and then obtain an output vector of the decoder.

[0098] The total loss function in step S26 is:

[0099]

[0100]

[0101] (7) In the formula, L word represents the cross-entropy loss between the predicted word and the real word, L len represents the mean square error between the predicted word group and the real word group; T represents the length of the real label, and θ represents the parameters of the model, represents the predicted word, represents the real label, p θ represents the probability of predicting based on , k represents the number of word groups, and s i represents the number of words in the predicted word group, represents the number of words in the real word group.

[0102] (8) In the formula, L group represents the cross-entropy loss between the predicted word group and the real word group, S i represents the currently predicted word group, S <i represents the first i-1 word groups participating in, w ij represents the jth word in the ith word group, p θ represents the probability of predicting the word w <i based on S ij , and the model is trained by continuously reducing the loss between the predicted word and the real label word.

[0103] The image labeling method of the present application will be described in detail below: Figure 3

[0104] ​First, the image-labeled dataset is processed, and the image features extracted by Faster-RCNN have a size of 50*512. The real label corresponding to the image is analyzed by using a deep learning method, and the label is divided into k different word groups. Referring to Figure 3 As shown in the figure, the global and local features of the 50*512 size image features are extracted by the multi-head self-attention operation and the residual network respectively, and then they are further aggregated by using the feedforward neural network and the layer normalization operation.

[0105] Specifically: after batch loading the dataset features, the global feature size of 50*512 is obtained based on the multi-head self-attention module; the residual network sequentially passes the image features through Conv1d(512, 512, 1) (the first 512 refers to the number of channels of the input convolution layer, the second 512 refers to the number of channels of the output convolution layer, and 1 refers to the size of the convolution kernel), and then the obtained image features are passed through the batch normalization BatchNorm and added to the input features, and then passed through the Relu activation function, to obtain the local features between the features, with a size of 50*512. After adding the global and local features, input them into the feedforward neural network for further fusion, and after residual connection and layer normalization, the output feature of the encoding layer is finally obtained as 50*512. For example, repeat this step three times, and the last encoding layer obtains a feature of 50*512, which is taken as the output of the encoder.

[0106] Referring to Figure 3 As shown in the figure, after obtaining the encoded features, input them into the word group prediction module to predict the number of words in the kth word group (for ease of description, the number of word groups k is set to 2 in the figure, and the same below). The linear prediction module is composed of one fully connected layer and one Relu activation function. After the encoder output feature is expanded to 50*512=25600, the k numbers representing the number of words in the kth word group are obtained by the linear prediction module. As shown in the figure, s1 represents the number of words in the first word group, and there are s1 words in total. According to the number of words in the word group divided in S21, combined with whether the current word group participates in predicting the next word group, a new mask matrix is constructed. As Figure 3 As shown in the figure, the real label corresponding to the current picture is "a man is walking in the park" a man on aboard", wherein " indicates the start of the prediction label. The number of words of the first word group divided in S21 is 2, and the second word group is 3, so the first column of the first and second rows in the matrix is set to 0 (white, indicating the current position in the label "aman onaboard" does not participate in the prediction of the next word group). The first row indicates that by "predict the first word "a", the second line indicates by" "predict the second word "man". The first three columns of rows three through five are then set to zero (indicating "on a board"), and the rest of the columns are set to 1 (indicating that "on a board" does not participate in prediction), and so on until all the matrices are constructed. Replace the mask matrix in the first self-attention mechanism in the original decoder with the mask matrix to achieve flexible prediction of image annotation results.

[0107] After constructing the mask matrix, the image corresponding annotation is word embedding and position encoding, and the word and position information are converted into 512 size vectors, which are added together as the input of the decoder module, with a size of seq len*512 (seq len is the length of the word group). After the input of the decoder module passes through the modified first multi-head self-attention module, a feature of seq len*512 is obtained. Take it as the query of the next multi-head self-attention module, and the output feature of the encoder as the key and value, pass it through the second multi-head self-attention module to obtain a feature of seq len*512, and then input it into the feedforward neural network, perform residual connection and layer normalization to obtain the input of the encoding layer, with a size of seq len*512. Repeat this step for the number of predictions, and the last decoding layer obtains a feature of seq len*512 size, which is taken as the output of the decoder. After the vector passes through a fully connected layer, a vector of seq len*vocab size size is obtained (vocab size is the size of the dictionary). Select the word at the position of the maximum probability as the predicted word, and finally concatenate it to form the final annotation "a man on aboard <e> ”( <e>denotes the end symbol, representing the end of prediction).

[0108] Wherein, the calculation process based on local feature enhancement and parallel decoder refers to the above formula (1)-(6).

[0109] Secondly, by calculating the cross-entropy loss of the predicted label and the real label, and the mean square error between the number of words of each word group of the prediction and the number of words of the real, and then obtaining the overall loss function. Referring to the above formulas (7), (8); according to the overall loss function (7), (8), the image labeling model is trained.

[0110] Finally, using the trained image labeling model, the image features extracted from the target picture data are input, and the image labeling of the target picture data is output. By using the combination of autoregressive and non-autoregressive in this example, not only can the labeling be generated quickly and accurately, but also the generation process has a certain flexibility, which can generate different number of word groups in each step according to different images, and splice them together to form the final result, realizing the fast, flexible and accurate generation of labeling for the target picture.

[0111] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.< / e> < / e>

Claims

1. An image labeling method based on local feature enhancement and parallel decoder, characterized in that, The method comprises the following steps: S1, extracting image features from target picture data; S2, inputting the image features into a pre-trained image annotation model; The image annotation model comprises a local feature enhanced encoder module, a word group prediction module, and a parallel word prediction decoder module; S3, outputting image annotation of the target picture data; The image annotation model training process in step S2 comprises: S21, extracting image features from training picture data, performing syntactic analysis on real annotations corresponding to the training picture data, dividing the real annotations into different word groups, and constructing a sample data set required for training; S22, inputting the image features into the local feature enhanced encoder module, extracting global and local features between the image features by using a multi-head self-attention mechanism and a residual network, and fusing the global and local features; S23, dividing the annotations to be predicted into k word groups, predicting the number of words in the kth word group by using the features output by the encoder, designing a mask matrix of the decoder according to the word groups divided in step S21 and whether the word at the current position in the annotation needs to participate in the prediction of the next word group; S24, in the parallel word prediction decoder module, predicting the vector of each word in each word group by using the features output by the encoder and the corresponding real annotation, and combining the mask matrix in step S23, to finally obtain the output vector of the decoder; S25, according to the output vector of the decoder, using the word with the maximum prediction probability to represent the word at each position, and connecting all the words to form the final result as the predicted image annotation of the training picture data; S26, generating a total loss function by calculating the cross-entropy loss between the predicted image annotation of the training picture data and the corresponding real annotation, and the mean square error between the number of words in the k predicted word groups and the number of divided words; and training the image annotation model according to the total loss function. 2.The image annotation method based on local feature enhancement and parallel decoder according to claim 1, characterized in that, The step S1 comprises: A Faster-RCNN using a backbone network of ResNet101 is used to obtain image features from target picture data. 3.The image annotation method based on local feature enhancement and parallel decoder according to claim 1, characterized in that, In step S22, the global features between the image features are extracted by using a multi-head self-attention mechanism; GF(X) = Attention(XW q ,XW k ,XW v ) (1) (1) where X denotes image features; GF(X) denotes global features; Attention denotes a self-attention operation, W q ,W k ,W v denote weight matrices for queries, keys, and values, respectively.

4. The image annotation method based on local feature enhancement and parallel decoder according to claim 3, characterized in that, In step S22, the local features between the image features are extracted by using a residual network: LF(X) = δ(BN(Conv(X) + X)) (2) In formula (2), LF(X) represents the local features calculated by the residual network; Conv and BN represent convolution and batch normalization operations respectively, and δ represents a Relu activation function.

5. The image annotation method based on local feature enhancement and parallel decoder according to claim 4, characterized in that, In step S22, the global and local features are fused, which comprises: 1) initially fusing the global and local features: X' = GF(X) + LF(X) (3) In the above formula, X' represents the global features after local feature enhancement; 2) secondary fusion: Y = LayerNorm(X' + FFN(X')) (4) In the formula, Y represents the output of the encoder; LayerNorm represents a layer normalization operation, and FFN represents a feedforward neural network.

6. The image annotation method based on local feature enhancement and parallel decoder according to claim 5, characterized in that, In the step S23, the number of words of the kth word group is predicted by using the features output by the encoder, and the step includes: The number of words of the kth word group is predicted by using the features output by the encoder according to the following formula: S k = δ(Y'W l ) (5) In the above formula, s k represents the number of words in the kth word group; δ represents the Relu activation function; Y' represents a one-dimensional vector obtained after Y is unfolded; W l represents a projection matrix, which projects Y' to a specified dimension size.

7. The image annotation method based on local feature enhancement and parallel decoder according to claim 6, characterized in that, In the step S23, the mask matrix of the decoder is designed, and the step includes: The mask matrix in the modified self-attention mechanism is used to replace the mask matrix of the decoder. In the above formula, M ij represents the mask matrix of the modified decoder, i represents the i-th word of the matrix, j represents the j-th word of the matrix, allow represents that the current word participates in prediction, and disallow represents that the similarity between the i-th word and the j-th word is -∞, that is, the current word does not participate in prediction.

8. The image annotation method based on local feature enhancement and parallel decoder according to claim 7, characterized in that, The step S24 includes: S241, word vector embedding and position coding are performed on the real label, and the sum thereof is taken as a label vector and input into the decoder; S242, the label vector is input into the first multi-head self-attention module by using the modified mask matrix of the decoder, and a new label vector is obtained; S243, the new label vector is taken as a query vector, the features output by the encoder are taken as keys and values, and the query vector, the keys and the values are input into the second multi-head self-attention module; the output of the multi-head self-attention module is taken as the input of the feedforward neural network; S244, the output of S243 is taken as the input of the next encoding layer, and the steps S242-S243 are repeatedly executed for a preset number of times, and then the output vector of the decoder is obtained. 9.The image annotation method based on local feature enhancement and parallel decoder according to claim 8, wherein, The overall loss function in the step S26 is: (7) where L word represents the cross-entropy loss between the predicted words and the true words, L len represents the mean squared error between the predicted word groups and the true word groups; T represents a true labeled length, θ represents a parameter of a model, represents a predicted word, represents a true label, p θ represents a probability of predicting on the basis of , k represents a number of word groups, s i represents a number of words of a predicted word group, represents a number of words of a true word group; (8) where L group represents the cross-entropy loss between the predicted word group and the real word group, S i represents the current predicted word group, S <i represents the first i-1 word groups involved, w ij represents the jth word of the ith word group, p θ represents the probability of predicting the word w <i based on S ij The model is trained by continuously reducing the loss between the predicted word and the real labeled word.

Images