An image description method based on semantically guided feature selection

Abstract

An image description method based on semantically guided feature selection, involving artificial intelligence technology, addresses the feature fragmentation drawback of methods using grid features. The method comprises the following steps: 1) Extracting grid features from the image using a convolutional neural network; 2) Enhancing the grid features using a traditional self-attention encoder; 3) Selecting features from the grid features extracted in step 2) using a feature selection layer perceptible to spatial and channel relationships; 4) Performing average pooling on the grid features selected in step 3) to obtain global features; 5) Performing multi-label classification on the global features obtained in step 4) to determine whether objects in the image appear in the corresponding descriptive statements; 6) Inputting the visual features output from step 4) into a decoder to generate image descriptive statements; 7) Defining a loss function by combining classification loss and image description generation loss. This method fully utilizes textual concepts, incorporating fine-grained semantic knowledge into the selection process.

Description

Technical Field

[0001] This invention relates to automatic image description in the field of artificial intelligence, and in particular to an image description method based on semantically guided feature selection that describes the objective content of an image using natural language. Background Technology

[0002] Image captioning is a machine intelligence task proposed by the artificial intelligence community in recent years. It aims to automatically generate natural language descriptions for a given image and has aroused widespread interest in the artificial intelligence and multimedia communities.

[0003] With the development of artificial intelligence technology, especially computer vision technology, tasks such as object classification, object detection, semantic segmentation, and instance segmentation can no longer meet people's production needs. Furthermore, with the development of the internet and short videos, there is an urgent need for automatic and objective description of visual content. Unlike detection and segmentation tasks, automatic image description requires using natural, accurate, and human-understandable language to provide a holistic and objective description of objects, attributes, relationships between objects, and corresponding scenes in an image. This task is one of the important directions for the development of computer vision understanding and even artificial intelligence, and is considered a significant indicator of AI advancement.

[0004] Early automatic image captioning methods primarily employed retrieval-based and template-based approaches. In recent years, influenced by advancements in natural language processing, encoder-decoder frameworks have become more prevalent. These typically utilize a pre-trained variant of a convolutional neural network (CNN) as the image encoder and a recurrent neural network (RNN) as the decoder. Furthermore, the introduction of attention mechanisms and reinforcement learning-based objective functions has significantly advanced this task.

[0005] Xu et al. [1] This paper introduces an attention mechanism for the first time in an image captioning task, focusing on generating the most relevant region for each word in each word generation step. (Lu et al.) [2] Develop adaptive attention methods to determine whether to rely on visual features or linguistic context in the decoding and generation step of each word. (Rennie et al.) [3] Introducing reinforcement learning techniques into image captioning tasks has significantly improved the performance of image captioning methods. Recently, Anderson et al. [4] They also proposed two attention techniques: top-down and bottom-up. First, an object detector is used to extract the most salient regions in the image. Then, based on these image region features, the most relevant salient regions for each word are found for word decoding and generation. (Huang et al.) [5]Pan et al. were the first to introduce the transformer model from natural language processing into image description tasks. [6] and Cornia [7] Others have used the Transformer model to replace recurrent neural networks and achieved the best performance. Furthermore, Liu et al. [8][9] and Li et al.

[10] The Transformer is used to integrate visual information and semantic concepts provided by external labelers to accomplish image description tasks. Luo et al.

[11] The comprehensive information contained in grid features is used to complement region features, thereby improving the accuracy of image description. (Zhang et al.)

[12] Furthermore, only grid features are used, and positional encoding and language models are employed to strengthen the correlation between grid features in order to improve the performance of image description.

[0006] However, in these models for image description tasks, especially those using grid features, the drawbacks of information redundancy and fragmentation caused by the comprehensiveness of grid features have not yet been taken into account.

[0007] References:

[0008] [1].Xu,K.;Ba,J.;Kiros,R.;Cho,K.;Courville,A.;Salakhudinov,R.;Zemel,R.;and Bengio,Y.2015.Show,attend and tell:Neural image caption generation with visual attention.In ICML.

[0009] [2]. Lu, J.; Xiong, C.; Parikh, D.; and Socher, R. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR.

[0010] [3]. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. InCVPR.

[0011] [4].Anderson,P.;He,X.;Buehler,C.;Teney,D.;Johnson,M.;Gould,S.;andZhang,L.2018.Bottom-up and top-down attention for image captioning and visualquestion answering.In CVPR.

[0012] [5].Huang,L.;Wang,W.;Chen,J.;andWei,X.-Y.2019.Attention on Attentionfor Image Captioning.In ICCV.

[0013] [6].Yingwei Pan,Ting Yao,Yehao Li,and Tao Mei.2020.X-linear attentionnetworks for image captioning.In CVPR.

[0014] [7].Cornia,M.;Stefanini,M.;Baraldi,L.;and Cucchiara,R.2020.Meshed-Memory Transformer for Image Captioning.In CVPR.

[0015] [8].Fenglin Liu,Yuanxin Liu,Xuancheng Ren,Xiaodong He,and XuSun.2019.Aligning visual regions and textual concepts forsemantic-groundedimage representations.In NeurIPS.

[0016] [9].Fenglin Liu,Xian Wu,Shen Ge,Xiaoyu Zhang,Wei Fan,and YuexianZou.2020.Bridging the gap between vision and language domains for improvedimage captioning.In ACM MM.

[0017]

[10] . Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. 2019. Entangled transformer for image captioning. In ICCV.

[0018]

[11] . Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin and Rongrong Ji. 2021. Dual-Level Collaborative Transformer for Image Captioning. In AAAI

[0019]

[12] . Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang and Rongrong Ji. 2021. Rstnet: Captioning with adaptive attention on visual and non-visual words, “. In CVPR Summary of the Invention

[0020] The purpose of this invention is to provide an image description method based on semantically guided feature selection. Addressing the issues of noise and fragmented features introduced by grid feature-based image description methods, this invention proposes a feature selection module that is both spatially and channelally perceptible. Simultaneously, a multi-label classification task is used to determine if a feature contains corresponding label text, thereby providing semantic guidance and supervision, thus improving the accuracy of the generated description statements.

[0021] This invention includes the following steps:

[0022] 1) A convolutional neural network is used to extract the grid features V = {v1, v2, ..., v} of the image to be described. N}, where v i ∈R d , i = 1, 2, ..., N, d is the dimension of each feature vector, and N is the number of grid features;

[0023] 2) An encoder layer consisting of a self-attention layer and a feedforward neural network layer is used to enhance the grid features of the image to be described, and the output of the last layer of the encoder is obtained. Where L is the number of layers in the encoder;

[0024] 3) The enhanced mesh features obtained in step 2) are fed into the feature selection layer that is aware of spatial and channel relationships for feature selection;

[0025] 4) Input the mesh features obtained in step 3) into the decoder to generate a description of the image;

[0026] 5) Simultaneously with step 4), the grid features obtained in step 3) are averaged to obtain global features, and multi-label classification is performed on them to determine whether they contain words from the corresponding sentences to provide semantic guidance for the features.

[0027] 6) Define the loss function based on the image description statement generated in step 4) and the loss calculated by multi-label classification in step 5).

[0028] In step 1), the training method of the convolutional neural network is as follows: using the deep convolutional residual network backbone in the Faster-RCNN framework, the network is first trained on the classic object detection dataset PASCALVOC2007 using an end-to-end method, and then further trained and fine-tuned on the multimodal dataset Visual Genome.

[0029] In step 2), the specific process of enhancing the grid features of the image to be described by the encoder layer composed of a self-attention layer and a feedforward neural network layer is as follows: The grid features V are input into the self-attention layer of the encoder as follows:

[0030] Given an index Q, a key K, and a value V,

[0031]

[0032] MultiHead(Q,K,V)=Concat(H1,...,H h W o ,

[0033] H i =Attention(QW i Q ,KW i K VW i V ),

[0034] Where Q is a denoted ... q A matrix of n index vectors, where K and V are matrices with n index vectors respectively. k A matrix of key and value vectors, all with the same dimension, where the softmax function is a normalized exponential function. is the projection matrix of the network, d is the hidden dimension of the model, and h is the number of heads in the multi-head attention.

[0035] In step 2), the encoder layer composed of the self-attention layer and the feedforward neural network layer is:

[0036] For a given input feature

[0037]

[0038] FFN(x) = max(0, xW1+b1)W2+b2,

[0039] X l+1 =LayerNorm(O l+1 +FFN(O l+1 )),

[0040] X l+1 =EnLayer(X l )

[0041] Where l is the l-th encoder layer, These are the projection matrix and bias of the feedforward neural network. d is the hidden dimension of the model, and LayerNorm is the layer normalization operation. These parts together form the EnLayer layer.

[0042] In step 3), the enhanced mesh features from step 2) are fed into a spatially perceptible feature selection layer, i.e.:

[0043] f θs (x i ) = ReLU(BN(W θs x i )),

[0044] f φs (x j ) = ReLU(BN(W φs x j )),

[0045] r i,j =f θs (x i ) T f φs (x j ),

[0046]

[0047] r i =[R s (i,:),R s (:,i)]

[0048] f ψs (r i =ReLU(W) ψs r i ),

[0049]

[0050] α i =Sigmoid(W2ReLU(W1u) i )),

[0051]

[0052] Where X is the mesh feature enhanced in step 2), ReLU is the activation function, BN is batch normalization, Sigmoid is the activation function, and W... θs W φs , W ψs The linear transformation parameter matrix, It is a matrix of mesh feature correlations, r i,j x represents i and x j The similarity score. i It is the concatenation of the scores of the i-th row related to xi and the scores of the i-th column of xi related to other grid features, α i It is x i Importance score for x i Weighted selection is applied; SRS stands for Spatial Relation-Aware Selection. It is the final grid feature selected based on spatial relationships.

[0053] In step 3), the enhanced mesh features from step 2) are fed into a channel-relationship-aware feature selection layer, i.e.:

[0054] f θc (x i ) = ReLU(BN(W θc x i )),

[0055] f φc (x j ) = ReLU(BN(W φc x j )),

[0056] r i,j =f θc (x i ) T f φc (xj ),

[0057]

[0058] r i =[R c (i,:),R c (:,i)]

[0059] f ψc (r i =ReLU(W) ψc r i ),

[0060]

[0061] α i =Sigmoid(W2ReLU(W1u) i )),

[0062]

[0063] Where X is the mesh feature enhanced in step 2), ReLU is the activation function, BN is batch normalization, Sigmoid is the activation function, and W... θc W φc , W ψc The linear transformation parameter matrix, It is a matrix of mesh feature correlations, where C is the number of channels and r is the number of channels. i,j x represents i and x j Channel similarity score; r i It is the concatenation of the scores of the i-th row related to xi and the scores of the i-th column of xi related to other grid features, α i It is x i Importance score for x i Weighted selection, CRS stands for Channel Relation-Aware Selection, which means that channel relationship-aware selection is achieved. It is the final mesh feature selected through channel relationships.

[0064] In step 3), the enhanced mesh features obtained in step 2) are fed into a feature selection layer that is aware of spatial and channel relationships. The overall process is as follows:

[0065] V 0 =W v V,

[0066] V L =Encoder(V 0 ),

[0067] V f =CRS(SRS(V L )),

[0068] Among them, V 0 W is the image grid feature extracted by the convolutional network. v These are the embedding layer parameters. The Encoder is the entire encoder consisting of L EnLayer layers, and V... f It is the final grid feature obtained after selection by spatial relationship perception and channel relationship perception.

[0069] In step 4), the grid features obtained in step 3) are input into the decoder to generate a descriptive statement for the image. The loss function for generating the sentence is defined, and the specific process is as follows:

[0070] First, assume t is the t-th word in the sentence sequence, w t Let represent the word vector corresponding to the input word at time t. Thus, the word vector input matrix for the decoder is:

[0071] W t-1 =(w0,w1,...,w t-1 ),

[0072] Similar to the encoder, the decoder also consists of L layers of the same structure. We define the input of the (l+1)th layer as... The output is The process definition for this layer is as follows:

[0073]

[0074] V g =(V L g F ),

[0075]

[0076] Among them, W y Given the parameter matrix to be trained, the hidden states are mapped to the vocabulary. For the predicted sentence Y... T ={y0, ...,y T In this context, the probability of generating the entire sentence is obtained by multiplying the probabilities of each word, i.e.:

[0077]

[0078] Where T is the sentence length; the model is trained through two stages: supervised learning and reinforcement learning; in the supervised learning stage, cross-entropy is used for a given target sentence. In this regard, its loss function is defined as:

[0079]

[0080] In the reinforcement learning phase, reinforcement learning is used for training, and the gradient of its loss function is defined as:

[0081]

[0082] in, This is the i-th sentence sampled by beamsearch, b = (∑ i r(Y i )) / k is used as the baseline value.

[0083] In step 5), the grid features obtained in step 3) are averaged to obtain global features, and then multi-labeled to classify them to determine whether they contain words from the corresponding sentences to provide semantic guidance for the features. The specific process is as follows:

[0084]

[0085] Where pooling is the average pooling layer, sigmoid is the activation function, and W... c It is a projection matrix, z = [z1, z2, ..., z M [] is the tag vector; when the j-th word appears in the corresponding tag text, z... j =1, otherwise z j =0. p c =[p c1 ,p c2 ,...,p cM ] represents the predicted probability of each word appearing in the image, and M is the size of the entire word list.

[0086] In step 6), the loss function is defined based on the image description generated in step 4) and the loss calculated by multi-label classification in step 5). The specific process is as follows:

[0087] L=L(θ)+λL s ,

[0088] Where L(θ) is the loss for image description, L s It is the loss for multi-label classification, and θ is a general term for the encoder and decoder parameters.

[0089] The advantages of this invention are as follows:

[0090] The proposed method provides an image description approach based on semantically guided feature selection, comprising a relation-aware feature selection module and a fine-grained semantic guidance strategy module. The relation-aware feature selection (RAS) module captures more detailed interactions between grid features to select salient visual information and suppress background information. The fine-grained semantic guidance (FSG) strategy explicitly guides the selection process through end-to-end multi-task learning. It fully leverages textual concepts, incorporating fine-grained semantic knowledge into the selection process. Experimental evaluations on the industry-recognized baseline dataset MS COCO demonstrate that the proposed semantically guided feature selection-based image description method achieves state-of-the-art performance. Attached Figure Description

[0091] Figure 1 This is a flowchart of the image description method based on semantically guided feature selection according to the present invention.

[0092] Figure 2 These are comparison images of sentences generated by different image description models. The Transformer is the model used for comparison; it's a classic image description baseline method. Detailed Implementation

[0093] The present invention will now be described in detail with reference to the accompanying drawings.

[0094] This invention includes the following steps:

[0095] 1) A convolutional neural network is used to extract the grid features V = {v1, v2, ..., v} of the image to be described. N}, where v i ∈R d , i = 1, 2, ..., N, d is the dimension of each feature vector, and N is the number of grid features;

[0096] 2) An encoder layer consisting of a self-attention layer and a feedforward neural network layer is used to enhance the grid features of the image to be described, and the output of the last layer of the encoder is obtained. Where L is the number of layers in the encoder;

[0097] 3) The enhanced mesh features obtained in step 2) are sent to the feature selection layer that is aware of spatial and channel relationships for feature selection.

[0098] 4) Input the mesh features obtained in step 3) into the decoder to generate a description of the image;

[0099] 5) and 4) are performed simultaneously. The grid features obtained in step 3) are averaged and pooled to obtain global features. Multi-label classification is then performed on these features to determine whether they contain words from the corresponding sentences, thus providing semantic guidance for the features.

[0100] 6) Define the loss function based on the image description statement generated in step 4) and the loss calculated by multi-label classification in step 5).

[0101] The flowchart of the overall process of the present invention is as follows: Figure 1 As shown.

[0102] Each module is as follows:

[0103] 1. Deep Convolutional Feature Extraction and Descriptive Data Preprocessing

[0104] Stop words were removed from all training data text, and all English words were lowercased. Then, the text was segmented by spaces, resulting in 10201 words. Words appearing less than five times in the dataset description were removed. <unk>"Replace it, and add a start character." <bos>"and end symbol" <end>"They appear at the beginning and end of the sentence, respectively."

[0105] First, a pre-trained object detector is used to extract 36 fixed candidate regions or the image is divided into 7×7 feature regions. Then, a residual deep convolutional network is used to extract the features V = {v1, v2, ..., v} corresponding to each candidate region. N }, where v i ∈R d Let i = 1, 2, ..., N, d be the dimension of each feature vector, k = 50, and d = 2048. First, the formal description of image feature extraction from the query image and the image library to be retrieved is: for each image in the training set, extract d-dimensional features to obtain a d×n original visual feature matrix X = [x1, x2, ..., xn]. n ]∈R d×n Where n represents the number of training samples in the training set, x i Let the i-th column of matrix X represent the eigenvector of the i-th dimension d in the sample set.

[0106] Word statistics are performed on all labeled data in the training data, and only the top 1000 nouns, verbs and adjectives are selected. Supervision signals for multi-label classification are obtained by extracting the corresponding words from each image description training sample.

[0107] 2. Feature-enhancing encoder network:

[0108] A feature-enhanced encoder network is used to enhance the encoding of the input image features and semantic concept features. For a given index Q, key K, and value V,

[0109]

[0110] MultiHead(Q,K,V)=Concat(H1,...,H h W o ,

[0111] H i =Attention(QW i Q ,KW i K VW i V ),

[0112] Where Q is a denoted ... q A matrix of n index vectors, where K and V are matrices with n index vectors respectively. k A matrix of key and value vectors, all with the same dimension, where the softmax function is a normalized exponential function. is the projection matrix of the network, d is the hidden dimension of the model, and h is the number of heads in the multi-head attention.

[0113] For a given input feature

[0114]

[0115] FFN(x) = max(0, xW1+b1)W2+b2,

[0116] X l+1 =LayerNorm(O l+1 +FFN(O l+1 )),

[0117] X l+1 =EnLayer(X l )

[0118] Where l is the l-th encoder layer, These are the projection matrix and bias of the feedforward neural network. d represents the hidden dimension of the model, and LayerNorm is the layer normalization operation. These parts together form the EnLayer layer. The L EnLayer layers are stacked to form the entire encoder network.

[0119] 3. Relationship-aware feature selection network

[0120] 3.1, Spatial Relationship-Aware Feature Selection Layer, namely:

[0121] f θs (x i ) = ReLU(BN(W θs x i )),

[0122] f φs (x j ) = ReLU(BN(W φs x j )),

[0123] r i,j =f θs (x i ) T f φs (x j ),

[0124]

[0125] r i =[R s (i,:),R s (:,i)]

[0126] f ψs (r i =ReLU(W) ψs r i ),

[0127]

[0128] α i =Sigmoid(W2ReLU(W1u) i )),

[0129]

[0130] Where X is the grid feature input, ReLU is the activation function, BN is batch normalization, Sigmoid is the activation function, and W is the activation function. θs W φs , W ψs The linear transformation parameter matrix, It is a matrix of mesh feature correlations, r i,j x represents i and x j The similarity score. i It is the concatenation of the scores of the i-th row related to xi and the scores of the i-th column of xi related to other grid features, α i It is x i Importance score for x i Weighted selection is applied; SRS stands for Spatial Relation-Aware Selection. It is the final grid feature selected based on spatial relationships.

[0131] 3.2, a feature selection layer that is aware of channel relationships, namely:

[0132] f θc (x i ) = ReLU(BN(W θc x i )),

[0133] f φc (x j ) = ReLU(BN(W φc x j )),

[0134] r i,j =f θc (x i ) T f φc (x j ),

[0135]

[0136] r i =[R c (i,:),R c (:,i)]

[0137] f ψc (r i =ReLU(W) ψc r i ),

[0138]

[0139] α i =Sigmoid(W2ReLU(W1u) i )),

[0140]

[0141] Where X is the input grid feature, ReLU is the activation function, BN is batch normalization, Sigmoid is the activation function, and W... θc W φc , W ψc The linear transformation parameter matrix, It is a matrix of mesh feature correlations, where C is the number of channels and r is the number of channels. i,j x represents i and x j Channel similarity score. i It is the concatenation of the scores of the i-th row related to xi and the scores of the i-th column of xi related to other grid features, α i It is x i Importance score for x i Weighted selection is applied. CRS stands for Channel Relation-Aware Selection, which means that channel relationship-aware selection is performed. It is the final mesh feature selected through channel relationships.

[0142] 3.3 Relationship-Aware Feature Selection Network

[0143] The combination of the aforementioned spatial relationship and channel relationship-aware feature selection layers constitutes a relationship-aware feature selection network, namely:

[0144] V 0 =W v V,

[0145] V L =Encoder(V 0 ),

[0146] V f =CRS(SRS(V L )),

[0147] Among them, V 0 W is the image grid feature extracted by the convolutional network. v These are the embedding layer parameters. The Encoder is the entire encoder consisting of L EnLayer layers, and V... f It is the final grid feature obtained through spatial relationship perception and channel relationship perception.

[0148] 4. Description of statement generation and corresponding loss function:

[0149] First, assume t is the t-th word in the sentence sequence, w t Let represent the word vector corresponding to the input word at time t. Thus, the word vector input matrix for the decoder is:

[0150] W t-1 =(w0,w1,...,w t-1 ),

[0151] Similar to the encoder, the decoder also consists of L layers of the same structure, with the input of the (l+1)th layer defined as... The output is The process definition for this layer is as follows:

[0152]

[0153]

[0154] V g =(V L g F ),

[0155]

[0156] Among them, W y Given the parameter matrix to be trained, the hidden states are mapped to the vocabulary. For the predicted sentence Y... T ={y0, ...,y T In this context, the probability of generating the entire sentence is obtained by multiplying the probabilities of each word, i.e.:

[0157]

[0158] Where T is the sentence length; the model is trained through two stages: supervised learning and reinforcement learning; in the supervised learning stage, cross-entropy is used for a given target sentence. In this regard, its loss function is defined as:

[0159]

[0160] In the reinforcement learning phase, reinforcement learning is used for training, and the gradient of its loss function is defined as:

[0161]

[0162] in, This is the i-th sentence sampled by beamsearch, b = (∑ i r(Y i )) / k is used as the baseline value.

[0163] 5. Fine-grained semantic-guided multi-label classification

[0164] The obtained grid features are average-pooled to obtain global features, and then multi-labeled for classification. The semantic guidance of the features is determined by whether they contain words from the corresponding sentence.

[0165]

[0166] Where pooling is the average pooling layer, sigmoid is the activation function, and W... c It is a projection matrix, z = [z1, z2, ..., z M [] is the tag vector; when the j-th word appears in the corresponding tag text, z... j =1, otherwise z j =0. p c =[p c1 ,p c2 ,...,p cM ] represents the predicted probability of each word appearing in the image, and M is the size of the entire word list.

[0167] 6. Overall Loss Function

[0168] The loss function is defined based on the loss calculated from the descriptive loss of the generated image and the loss obtained from multi-label classification, namely:

[0169] L=L(θ)+λL s ,

[0170] Where L(θ) is the loss for image description, L s It is the loss for multi-label classification, θ is the collective term for encoder and decoder parameters, and the overall process consists of... Figure 1 As shown.

[0171] Specific experimental results: To verify the feasibility and advancement of the method proposed in this invention, the model was evaluated on the MSCOCO dataset, a common dataset for evaluating image description methods. The quantitative comparison with state-of-the-art automatic image description methods is shown in Table 1.

[0172] Table 1

[0173]

[0174] As can be seen from Table 1, the method proposed in this invention exhibits significant advantages across various evaluation metrics and different image features. Furthermore, as... Figure 2 As shown, by visualizing the text descriptions generated from the input image (the example descriptions are in English; the automatic generation process for Chinese descriptions is similar), it can be seen that because the method of this invention employs feature selection and semantic guidance, it achieves a significant enhancement in image description compared to methods that do not use this approach (Transformer). It can accurately describe the various types of information contained in the image and combine them into reasonable natural language. In contrast, baseline methods tend to ignore inconspicuous objects or mix them with background information, generating incorrect descriptive statements. The method of this invention solves the problem caused by the redundancy and fragmentation of grid feature information, and can generate higher-quality descriptive statements.

[0175] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.< / end> < / bos> < / unk>

Claims

1. An image description method based on semantically guided feature selection, characterized in that... Includes the following steps: 1) A convolutional neural network is used to extract the grid features V = {v1, v2, ..., v} of the image to be described. N }, where v i ∈R d , i = 1, 2, ..., N, d is the dimension of each feature vector, and N is the number of grid features; 2) An encoder layer consisting of a self-attention layer and a feedforward neural network layer is used to enhance the grid features of the image to be described, and the output of the last layer of the encoder is obtained. Where L is the number of layers in the encoder; 3) The enhanced mesh features obtained in step 2) are fed into the feature selection layer that is aware of spatial and channel relationships for feature selection; 4) Input the mesh features obtained in step 3) into the decoder to generate a description of the image; 5) Perform average pooling on the grid features obtained in step 3) to obtain global features, and perform multi-label classification on them to determine whether they contain words from the corresponding sentences to provide semantic guidance for the features. The specific process is as follows: Where pooling is the average pooling layer, V f The final result is a grid feature selected through spatial relationship perception and channel relationship perception. Sigmoid is the activation function, and W... c It is a projection matrix, z = [z1, z2, ..., z M [] is the tag vector; when the j-th word appears in the corresponding tag text, z... j =1, otherwise z j =0; p c =[p c1 ,p c2 ,...,p cM [] represents the predicted probability of each word appearing in the image, and M is the size of the entire vocabulary; 6) Based on the image description generated in step 4) and the loss function calculated in step 5) for multi-label classification, define the loss function. The specific process is as follows: L=L(θ)+λL s , Where L(θ) is the loss for image description, L s It is the loss for multi-label classification, and θ is a general term for the encoder and decoder parameters.

2. The image description method based on semantically guided feature selection as described in claim 1, characterized in that... In step 1), the training method of the convolutional neural network is as follows: using the deep convolutional residual network backbone in the Faster-RCNN framework, the network is first trained on the classic object detection dataset PASCALVOC2007 using an end-to-end method, and then further trained and fine-tuned on the multimodal dataset Visual Genome.

3. The image description method based on semantically guided feature selection as described in claim 1, characterized in that... In step 2), the specific process of enhancing the grid features of the image to be described by the encoder layer composed of a self-attention layer and a feedforward neural network layer is as follows: The grid features V are input into the self-attention layer of the encoder as follows: Given an index Q, a key K, and a value B, MultiHead(Q,K,B)=Concat(H1,...,H h )W o , H i =Attention(QW i Q ,KW i K ,BW i V ), Where Q is a denoted ... q A matrix of n index vectors, where K and B are matrices with n index vectors respectively. k A matrix of key and value vectors, all with the same dimension, where the softmax function is a normalized exponential function. is the projection matrix of the network, U is the hidden dimension of the model, and h is the number of heads in the multi-head attention.

4. The image description method based on semantically guided feature selection as described in claim 1, characterized in that... In step 2), the encoder layer composed of the self-attention layer and the feedforward neural network layer is: For a given input feature FFN(x) = max(0, xW1+b1)W2+b2, X l+1 =LayerNorm(O l+1 +FFN(O l+1 )), X l+1 =EnLayer(X l ) Where l is the l-th encoder layer, All of these are the projection matrix and bias of the feedforward neural network; d is the hidden dimension of the model; LayerNorm is the layer normalization operation. These parts together constitute the EnLayer layer.

5. The image description method based on semantically guided feature selection as described in claim 1, characterized in that... In step 3), the enhanced mesh features from step 2) are fed into a spatially perceptible feature selection layer, i.e.: f θs (x i )=ReLU(BN(W θs x i )), f φs (x j )=ReLU(BN(W φs x j )), r i,j =f θs (x i ) T f φs (x j ), r i =[R s (i,:),R s (:,i)] f ψs (r i )=ReLU(W ψs r i ), α i =Sigmoid(W2ReLU(W1u i )), Where X is the mesh feature enhanced in step 2), ReLU is the activation function, BN is batch normalization, Sigmoid is the activation function, and W... θs W φs , W ψs The linear transformation parameter matrix, It is a matrix of mesh feature correlations, r i,j x represents i and x j Similarity score; r i It is the concatenation of the scores of the i-th row related to xi and the scores of the i-th column of xi related to other grid features, α i It is x i Importance score for x i Weighted selection is applied; SRS stands for Spatial Relation-Aware Selection. It is the final grid feature selected based on spatial relationships.

6. The image description method based on semantically guided feature selection as described in claim 1, characterized in that... In step 3), the enhanced mesh features from step 2) are fed into a channel-relationship-aware feature selection layer, i.e.: f θc (x i )=ReLU(BN(W θc x i )), f φc (x j )=ReLU(BN(W φc x j )), r i,j =f θc (x i ) T f φc (x j ), r i =[R c (i,:),R c (:,i)] f ψc (r i )=ReLU(W ψc r i ), α i =Sigmoid(W2ReLU(W1u i )), Where X is the mesh feature enhanced in step 2), ReLU is the activation function, BN is batch normalization, Sigmoid is the activation function, and W... θc W φc , W ψc The linear transformation parameter matrix, It is a matrix of mesh feature correlations, where C is the number of channels and r is the number of channels. i,j x represents i and x j Channel similarity score; r i It is the concatenation of the scores of the i-th row related to xi and the scores of the i-th column of xi related to other grid features, α i It is x i Importance score for x i Weighted selection is applied. CRS stands for Channel Relation-Aware Selection, which means that channel relationship-aware selection is performed. It is the final mesh feature selected through channel relationships.

7. The image description method based on semantically guided feature selection as described in claim 1, characterized in that... In step 3), the enhanced mesh features obtained in step 2) are fed into a feature selection layer that is aware of spatial and channel relationships. The overall process is as follows: V 0 =W v V, V L =Encoder(V 0 ), V f =CRS(SRS(V L )), Among them, V 0 W is the image grid feature extracted by the convolutional network. v These are the embedding layer parameters. The Encoder is the entire encoder consisting of L EnLayer layers, and V... f It is the final grid feature obtained through spatial relationship perception and channel relationship perception.

8. The image description method based on sample semantic guidance for feature selection as described in claim 1, characterized in that... In step 4), the grid features obtained in step 3) are input into the decoder to generate a descriptive statement for the image. The loss function for generating the sentence is defined, and the specific process is as follows: First, assume t is the t-th word in the sentence sequence, w t Let represent the word vector corresponding to the input word at time t. Thus, the word vector input matrix for the decoder is: IN t-1 =(w0,w1,...,w t-1 ), Similar to the encoder, the decoder also consists of L layers of the same structure, with the input of the (l+1)th layer defined as... The output is The process definition for this layer is as follows: V g =(V L ;g F ), Among them, W y Given the parameter matrix to be trained, the hidden states are mapped to the vocabulary; for the predicted sentence Y... T ={y0,…,y T In this context, the probability of generating the entire sentence is obtained by multiplying the probabilities of each word, i.e.: Where T is the sentence length; the model is trained through two stages: supervised learning and reinforcement learning; in the supervised learning stage, cross-entropy is used for a given target sentence. In this regard, its loss function is defined as: In the reinforcement learning phase, reinforcement learning is used for training, and the gradient of its loss function is defined as: in, This is the i-th sentence sampled by beam search, b = (∑ i r(Y i )) / k is used as the baseline value.

Images