A traffic scene generative image description method

Abstract

The application relates to a transformer-based traffic scene generative image description method, which comprises the following steps: manually labeling an existing traffic scene dataset, constructing an image description dataset under a traffic scene, and dividing the image description dataset into a training set and a test set; constructing an image description model, which comprises a target detector, a multi-modal feature fusion module, a transformer encoder, a text encoding module and a decoder connected in sequence; training the constructed image description model on the training set, describing the images in the test set, evaluating the model from two aspects of quantity and quality, obtaining an evaluation result and a descriptive sentence generated for each image. Compared with the prior art, the application has the advantages of comprehensive and accurate description of traffic scene images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving vehicle scene understanding, and in particular to a traffic scene generative image description method applied to an automatic driving vehicle road environment perception system. BACKGROUND

[0002] Image description is a task of generating a corresponding description sentence according to an input image, which can be widely applied to various scenes, including automatic driving cars, remote sensing images and robots, and the task involves two research fields of computer vision and natural language processing. The generated sentence should be able to accurately describe the attributes, actions and relationships between different objects of the core target object in the image.

[0003] Early image description methods are mainly based on retrieval and sentence templates. Such methods obtain visual information such as objects, attributes, actions and scenes from images, and generate descriptive sentences through language templates, but the sentences generated by such methods are usually single in structure and lack diversity, and due to the limitations of language templates, the generalization ability of the model needs to be improved. With the wide application of deep learning methods, the current mainstream image description method is based on the structure of encoder-decoder, such as using convolutional neural network (CNN) to encode visual features, long short term memory network (LSTM) to generate predicted text, or using a bidirectional LSTM method to obtain more accurate text generation.

[0004] Based on the CNN-LSTM architecture, many people have introduced attention mechanisms into it. In addition, many people have applied the transformer based on the self-attention mechanism to the image description problem, such as Full transformer network for image captioning directly sequences the input image, without the need for additional CNN for visual feature extraction; the method mentioned in Image captioning: Transforming objects into words encodes the relative position relationship of the target object in the image into the transformer structure.

[0005] The encoder-decoder structure based on transformer has relatively high accuracy for image description, but there are also problems of difficulty in multi-modal feature fusion and low accuracy of generated sentences, and at the same time, there are few image description methods for traffic scenes. SUMMARY

[0006] The application aims to provide a transformer-based traffic scene generative image description method to accurately and comprehensively describe images of traffic scenes.

[0007] The application aims to provide a transformer-based traffic scene generative image description method to accurately and comprehensively describe images of traffic scenes.

[0008] A transformer-based traffic scene generative image description method comprises the following steps.

[0009] Step 1) Manually label an existing traffic scene dataset to construct an image description dataset in a traffic scene and divide it into a training set and a test set.

[0010] Step 2) Construct an image description model, which comprises a target detector, a multi-modal feature fusion module, a transformer encoder, a text encoding module and a decoder connected in sequence, wherein the target detector extracts visual, attribute, category and geometric position features in each picture in the dataset; the multi-modal feature fusion module fuses visual features and semantic features in each picture to obtain enhanced visual features, wherein the semantic features include attribute features and category features; the transformer encoder takes the enhanced visual features and the geometric position features corresponding to the picture as input and outputs encoded visual features; the text encoding module encodes the textual description of the image to obtain text features; the transformer decoder fuses the text features and the encoded visual features and outputs through an output layer to complete the prediction of words and obtain a generated descriptive sentence.

[0011] Step 3) Train the constructed image description model on the training set constructed in step 1) and describe the images of the test set, evaluate the model from quantitative and qualitative angles to obtain evaluation results and descriptive sentences generated for each image.

[0012] The manual labeling of the existing traffic scene dataset specifically comprises giving a paragraph description composed of five sentences as a label for each picture, which describes whether there are vehicles on both sides of the road in front, whether there are pedestrians and cyclists around, the position of the vehicle, the position of the pedestrian, the color of the traffic light and the meaning of the traffic sign.

[0013] The multi-modal feature fusion module concatenates the input attribute features, category features and visual features, extracts a global vector representation after the concatenated features pass through a global average pooling layer, then calculates an attention weight vector through a nonlinear activation function, guides and enhances the input visual features using the obtained attention weight, and obtains enhanced visual features.

[0014] An expression of the multi-modal feature fusion module is:

[0015] X atten =ψ(GAP(C(X v ,X A ,X L )))

[0016] X EV =X atten ×X v +X v

[0017] Wherein, X V , X A , X L respectively represent the target visual feature, attribute feature and category feature detected by the target detector on an image, C represents splicing the three features along the feature length dimension, GAP represents a global average pooling operation, ψ represents a Sigmoid activation function, X atten represents an attention vector, and X EV represents the visual feature enhanced by the multi-modal feature fusion module.

[0018] The encoder comprises a plurality of encoding layers and a spatial convolution pyramid module connected in sequence, an input of each encoding layer comprises a visual feature and a geometric position feature detected by the target detector, and an output is an encoded visual feature, wherein the visual feature input to the first encoding layer is the visual feature enhanced by the multi-modal feature fusion module, and the visual feature input to the remaining encoding layers is the visual feature output by the previous encoding layer; the spatial convolution pyramid module adopts two one-dimensional convolution layers with different kernel sizes in parallel to enhance the visual feature output by the last encoding layer from the perspective of receptive field extraction of two features.

[0019] Each encoding layer comprises a geometric multi-head self-attention layer, a feedforward network layer, two residual connections and two regularization layers, wherein an expression of the first encoding layer is:

[0020] X1’=GeoMSA(LN(X EV ),X G )+X EV

[0021] X1=FFN(LN(X1’))+X1’

[0022] Wherein, X EV represents the visual feature enhanced by the multi-modal feature fusion module, and X GGeoMSA represents a geometric multi-head self-attention layer, LN represents a layer normalization operation, FFN represents a feed-forward neural network layer, X1' represents a feature vector obtained after the multi-head self-attention layer, and X1 represents a visual feature output by the first encoding layer;

[0023] The expression of the other encoding layers is as follows:

[0024] X i ' = GeoMSA (LN (X i-1 ), X G ) + X i-1

[0025] X i = FFN (LN (X i ')) + X i '

[0026] wherein 2≤i≤M, i is a positive integer, M is the number of encoding layers, X i represents a visual feature output by the i-th encoding layer.

[0027] The expression of the spatial convolution pyramid module is as follows:

[0028] X c = C (Conv1 (X M ), Conv3 (X M ))

[0029] X atten = Softmax (GAP (X c ))

[0030] X EV = X atten * X M + X M

[0031] wherein X M represents a visual feature output by the last encoding layer, Conv1 is a one-dimensional convolution layer with a kernel size of 1, Conv3 is a one-dimensional convolution layer with a kernel size of 3, C represents concatenation of two feature vectors in the feature length dimension, X c represents a concatenated feature vector, GAP represents a global average pooling operation, Softmax represents a Softmax function, X atten represents an attention weight vector, and X EV represents a visual feature enhanced by the spatial convolution pyramid module, i.e., a feature finally output by the encoder.

[0032] The text encoding module performs the following steps:

[0033] All sentences of all picture annotations are merged, punctuation is removed, and all words are converted to lowercase, repeated words and low-frequency words are removed, and a dictionary corresponding to the current data set is obtained;

[0034] Each word in the dictionary is assigned a number;

[0035] The current annotated sentence is mapped to a feature sequence;

[0036] According to the words in the dictionary and the corresponding numbers, a linear layer is used to add position encoding to each word in the current feature sequence to obtain text features.

[0037] The transformer decoder includes multiple decoding layers, wherein the number of decoding layers is the same as the number of encoding layers in the encoder, each decoding layer includes a masked self-attention structure, a cross self-attention structure, a feedforward network layer, three residual connections and three regularization layers, and the expression of the first decoding layer is:

[0038] Y1 0 =MaskedMSA(X′ word ,X′ word ,X′ word )+X′ word

[0039] Y1 1 =CrossMSA(Y1 0 ,X EV ,X EV )+Y1 0

[0040] Y1=FFN(Y1 1 )+Y1 1

[0041] Wherein, X′ word represents the text feature after word mapping and position encoding, X EV represents the visual feature input to the decoder after the encoder enhancement, MaskedMSA represents the masked self-attention structure, CrossMSA represents the cross self-attention structure, FFN represents the feedforward network layer, and Y1 represents the output of the first decoding layer.

[0042] The expression of other decoding layers is:

[0043] Y i 0 =MaskedMSA(Y i-1 ,Y i-1 ,Y i-1 )+Y i-1

[0044] Y i 1 = CrossMSA(Y i 0 ,X EV ,X EV )+Y i 0

[0045] Y i = FFN(Y i 1 )+Y i 1

[0046] wherein 2≤i≤M, i is a positive integer, M is the number of decoding layers, Y i represents the feature output by the i-th decoding layer.

[0047] The output layer transforms the feature dimension of the decoder output feature to the dimension of the entire dictionary through a linear layer, and classifies by using Softmax to obtain the probability of each word corresponding.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] (1) The present application annotates the existing data set Cityscapes, and comprehensively describes the image through a paragraph composed of five sentences, solving the problem of lack of traffic scene image description data set, and can more comprehensively and accurately describe the vehicles, pedestrians, traffic lights and other targets affecting driving safety in the traffic scene; and the image description model trained based on the data set can have higher description accuracy, which is beneficial to improve the accuracy of decision-making.

[0050] (2) The image description model based on transformer proposed by the present application enhances the visual features through a multi-modal feature fusion module, and designs a spatial convolution pyramid structure in the encoder structure to further extract visual features, which can better extract visual features, and combines the text features added with position encoding, thereby improving the description generation accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the flow chart of the method of the present application;

[0052] Figure 2 is the annotation process of the traffic scene data set Cityscapes in the embodiment of the present application;

[0053] Figure 3 is the image description model structure diagram in the embodiment of the present application;

[0054] Figure 4 is a structural diagram of a multi-modal feature fusion module VSA in an embodiment of the present application;

[0055] Figure 5 is a structural diagram of a transformer encoder in an embodiment of the present application;

[0056] Figure 6 is a structural diagram of a spatial convolution pyramid in an embodiment of the present application;

[0057] Figure 7 is a structural diagram of a transformer decoder in an embodiment of the present application;

[0058] Figure 8 is a description result diagram of an image description model for a traffic scene image in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and gives a detailed implementation and specific operation process, but the protection scope of the present application is not limited to the following embodiments.

[0060] The present embodiment provides a transformer-based traffic scene generative image description method, which describes the target object in the traffic scene from the perspective of text generation. The constructed image description module includes a visual semantic attention (VSA) module, which integrates multi-modal information from a pre-trained target detector to obtain global attention to guide and enhance the input visual feature representation. In order to further enrich and enhance the visual features extracted by the encoder, a new spatial convolution pyramid (SCP) structure is designed after the transformer encoding layer, which effectively expands the receptive field of visual information and can capture more context associations. Finally, the effectiveness of the method is verified based on the constructed traffic scene image dataset Cityscapes.

[0061] Specifically, as shown in Figure 1 , the following steps are included:

[0062] Step 1) Manually label the existing traffic scene dataset, construct an image description dataset under the traffic scene, and divide it into a training set and a test set.

[0063] The traffic scene dataset used in this embodiment is Cityscapes, which contains 2975 training images, 500 validation images and 1525 test images. It is developed for the traffic scene semantic segmentation task, and lacks text description labels for image scenes. Therefore, first, each image in the dataset is annotated with a text description. Generally, traffic scene images are complex and have a large amount of information, and it is difficult to completely describe them with one or two sentences. Therefore, as shown in Figure 2 , this embodiment uses a paragraph containing five sentences to label each image. In each image, whether there are vehicles on both sides of the road ahead, whether there are pedestrians and cyclists around, the position of the vehicle, the position of the pedestrian, the color of the traffic light, the meaning of the traffic sign, etc. are described.

[0064] Step 2) Build an image description model.

[0065] The image description model includes a target detector, a multi-modal feature fusion module, a transformer encoder, a text encoding module and a decoder connected in turn, as shown in Figure 3 .

[0066] (21) Target detector

[0067] The image description task requires a target detector to detect the vision, attributes, categories and positions of the target objects in the image. The pre-trained FasterRCNN target detector is selected in this embodiment, and the pre-training dataset is the Visual Genome visual genome dataset. This dataset annotates a large amount of rich information, including: region description, object, attribute, relationship, region graph, scene graph and question and answer pair. For each picture, the detected vision, attribute, category and geometric position features are saved separately in a.npy file. After that, the model reads each.npy file of the training set and test set during the training process.

[0068] (22) Multi-modal feature fusion module VSA

[0069] The attribute and category features extracted by the target detector belong to the text modality features (i.e. semantic features), and the visual features extracted belong to the image modality features. These two different modalities of features need to be fused before being input into the encoder. As shown in Figure 4 , the multi-modal feature fusion module VSA is built to concatenate the input attribute features, category features and visual features, extract global vector representations through a global average pooling layer, and then pass them through a Sigmoid activation function to obtain an attention weight vector to guide and enhance the input visual features. The enhanced visual features are used as the input of the encoder network.

[0070] In the VSA module, for the input visual features attribute features and category features where n0, n1, n2 represent the number of visual, attribute and category features obtained by the target detector respectively, and d represents the dimension of each feature after alignment. The three kinds of features are spliced in the first dimension (feature number dimension) to obtain a more rich context representation Then, a global average pooling operation is used to weight average the spliced features, and the global feature vector is calculated in the first dimension, and then a nonlinear Sigmoid activation function is used to obtain the attention representation Finally, the input visual feature X V is weighted by the attention representation X atten to obtain the enhanced visual input and X V have the same feature dimension.

[0071] X atten = ψ(GAP(C(X V , X A , X L ))) (1)

[0072] X EV = X atten × X V + X V (2)

[0073] where X V , X A , X L represent the visual feature, attribute feature and category feature of a target object in an image respectively, C represents the splicing of visual, attribute and category features in the feature length dimension, GAP represents the global average pooling operation, ψ represents the Sigmoid activation function, X atten represents the attention vector, and X EV represents the visual feature enhanced by the VSA module. After that, the visual feature X EV enhanced by the VSA will be input into the transformer encoder.

[0074] (23)transformer encoder

[0075] The structure of the encoder is as follows Figure 5As shown, in the embodiment, a transformer encoding layer and a spatial convolution pyramid module are cascaded in four layers, the input of each encoding layer includes visual features and geometric position features detected by a target detector, and the output is encoded visual features, wherein the visual features input to the first encoding layer are visual features enhanced by the multi-modal feature fusion module, and the visual features input to the remaining encoding layers (the second, third and fourth encoding layers) are visual features output by the previous encoding layer; the spatial convolution pyramid module enhances the visual features output by the last encoding layer from the perspective of receptive field by using two one-dimensional convolution layers with different kernel sizes in parallel.

[0076] (231)Encoding layer

[0077] ①Overall structure of the encoding layer: each encoding layer includes a geometric multi-head self-attention layer, a feedforward network layer, two residual connections and two regularization layers, wherein the expression of the first encoding layer is:

[0078] X1’=GeoMSA(LN(X EV ),X G )+X EV (3)

[0079] X1=FFN(LN(X1’))+X1’ (4)

[0080] wherein X EV represents visual features enhanced by the multi-modal feature fusion module, X G represents target geometric position features detected by the target detector, GeoMSA represents the geometric multi-head self-attention layer, LN represents the layer normalization operation, FFN represents the feedforward neural network layer, X1’ represents the feature vector obtained after the multi-head self-attention layer, and X1 represents the visual features output by the first encoding layer;

[0081] The expression of the other encoding layers is:

[0082] X i ’=GeoMSA(LN(X i-1 ),X G )+X i-1 (5)

[0083] X i =FFN(LN(X i ’))+X i ’ (6)

[0084] wherein 2≤i≤M, i is a positive integer, M is the number of encoding layers, in the embodiment, M=4, and X i represents the visual features output by the i-th encoding layer.

[0085] ②GeoMSA geometric multi-head self-attention layer

[0086] The conventional multi-head self-attention structure MSA only has one visual feature as input, while the GeoMSA considers both visual and geometric feature inputs. In the GeoMSA structure, the visual and geometric features need to be calculated to obtain the visual attention weight and the geometric attention weight, respectively. Assuming that the position coordinates corresponding to the mth and nth target detection boxes are (x m , y m , ω m , h m ) and (x n , y n , ω n , h n ), respectively, the relative position relationship geo mn between the two detection boxes can be represented as:

[0087]

[0088] The relative position relationship between the mth and nth detection boxes corresponds to the geometric attention weight, which can be represented as:

[0089]

[0090] In the multi-head self-attention mechanism, for each input feature, three linear layer transformations are required to obtain three feature matrices Q, K, and V. These three feature values have the same input dimension, based on which the visual attention weight between the mth and nth target detection boxes can be calculated, and then the visual attention weight between all detection boxes can be obtained as follows:

[0091]

[0092] where d represents the dimension value of the feature matrix Q, K, and V, represents the visual attention weight between the mth and nth detection boxes, Q mn represents the Q matrix corresponding to the mth and nth detection boxes, K mn represents the K matrix corresponding to the mth and nth detection boxes. After obtaining the visual attention weight and the geometric attention weight, the self-attention weight between the mth and nth detection boxes can be represented as:

[0093]

[0094] wherein, represents the geometric attention weight between the mth and nth detection boxes, represents the visual attention weight between the mth and nth detection boxes, denotes the geometric attention weight between the m-th and the 1-st bounding box, denotes the visual attention weight between the m-th and the 1-st bounding box, N denotes the number of bounding boxes in a picture, 1 denotes the 1-st bounding box in a picture, and e denotes the exponential power.

[0095] So far, the relative attention weight ω mn between the m-th and the n-th bounding box has been obtained, and all these relative attention weights constitute the self-attention weight matrix ω of the picture. In the GeoMSA structure, the obtained self-attention weight matrix is used to weight the V matrix obtained by mapping, and the final output feature is obtained, that is:

[0096] y = ω × V (11)

[0097] where ω denotes the self-attention weight vector calculated by the GeoMSA structure, V denotes the feature vector obtained by input mapping of the GeoMSA structure, and y denotes the feature vector output by the GeoMSA structure.

[0098] ③ FFN feedforward neural network structure: the feedforward neural network layer includes two linear layers, an activation function layer and a Dropout layer in composition, in order to extract the context association of the input sequence.

[0099] A3 = Dropout (ReLU (Linear1 (A2))) (12)

[0100] A4 = Linear2 (A3) (13)

[0101] where Linear1 and Linear2 are linear layers, ReLU denotes a linear ReLU activation function layer, Dropout denotes a Dropout layer, which makes a part of neurons randomly inactive to avoid overfitting problem when the network is too deep, A2 denotes the feature extracted by the multi-head self-attention layer, A3 denotes the feature vector after the first linear layer, and A4 denotes the output feature of the feedforward neural network. It is assumed that the input vector n denotes the length of the feature, d denotes the dimension of each feature mapping, and after the first linear layer Linear1, the feature dimension is expanded to 4 times the input, that is, and after the second linear layer Linear2, the feature dimension is restored to the size of the input dimension, that is, This dimension expansion operation is helpful for better feature learning.

[0102] (232) Spatial convolution pyramid module SCP

[0103] At the end of the transformer encoder, a spatial convolution pyramid module is built, as shown in Figure 6 Considering that the transformer is a sequence input, the feature extraction is based on the global receptive field, and the learning of the context association information is long distance, in order to better supplement the visual or semantic information between adjacent pixel points, the convolution structure is adopted, which can better learn the short distance feature dependence.

[0104] Due to the sequence form of the input and the text structure of the output, the convolution layer adopts a one-dimensional convolution structure. The features extracted by the fourth layer of the encoding layer respectively pass through a convolution layer with a size of 1 and a size of 3, and the two different convolution layer sizes help to expand the receptive field of feature extraction. In order not to introduce additional useless information, each convolution layer does not use zero padding operation, therefore, the visual feature size extracted by the convolution layer with a kernel size of 1 is The visual feature size extracted by the convolution layer with a kernel size of 3 is The features extracted by the two convolution layers have the same dimension and different lengths, therefore, the two vectors are spliced in the feature length direction (the first dimension), and the spliced feature has a longer feature length, and after the global average pooling operation and the Softmax function, the corresponding feature weight vector is obtained. Accordingly, the output feature X4 is weighted, and after combining the residual structure, the visual feature enhanced by multiple receptive fields is obtained

[0105] X c = C(Conv1(X4), Conv3(X4)) (14)

[0106] X atten = Softmax(GAP(X c )) (15)

[0107] X EV = X atten × X M + X M (16)

[0108] Wherein, X4 represents the visual feature output by the last layer of the encoding layer, Conv1 is a one-dimensional convolution layer with a kernel size of 1, Conv3 is a one-dimensional convolution layer with a kernel size of 3, C represents splicing two feature vectors in the feature length dimension, X c represents the spliced feature vector, GAP represents the global average pooling operation, Softmax represents the Softmax function, and X attendenotes the attention weight vector, X EV denotes the visual feature enhanced by the spatial convolution pyramid module, i.e., the final output feature of the encoder.

[0109] (24) text encoding module

[0110] The sentence annotated for each picture is a word text structure, which cannot be directly input to the transformer for feature learning. Therefore, the text feature is first mapped to a text sequence, and the position encoding of each word is added, and then input to the decoder.

[0111] The text encoding module performs the following steps:

[0112] All sentences annotated for all pictures are merged, punctuation is removed, and all words are converted to lowercase letters. The repeated words and the words with very low frequency of occurrence are removed to obtain a dictionary corresponding to the current data set.

[0113] Each word in the dictionary is assigned a number, and the input in text form is converted to a digital quantity.

[0114] The current annotated sentence is mapped to a feature sequence.

[0115] According to the words in the dictionary and their corresponding numbers, a linear layer is used to add position encoding to each word in the current feature sequence to obtain the text feature.

[0116] Suppose the length of an annotated sentence is N, and the dimension of the word vector is dim, then each sentence is mapped to a feature sequence of After that, position encoding is added to each word in the sentence so that the machine can remember the position of each word. The text feature encoding process of the decoder is:

[0117] X word = word_embedding(senten) (17)

[0118] X′ word = pos_encoding(X word ) (18)

[0119] Wherein, senten denotes the sentence annotated for the input picture, word_embedding denotes the word encoding process, X word denotes the sentence sequence obtained by encoding each word in the sentence, pos_encoding denotes the process of adding position encoding to the sentence sequence, X′ word denotes the text sequence after adding position encoding, which is input to the decoder module.

[0120] (25)transformer decoder

[0121] The decoder outputs the fused text and encoded visual features through an output layer to complete the prediction of the word and obtain the generated descriptive sentence.

[0122] In this embodiment, the number of decoding layers in the transformer decoder is the same as the number of encoding layers in the encoder, and therefore, the transformer decoder includes four cascaded decoding layers, as shown in the structure of Figure 7 The input of the decoder includes two parts: the visual features extracted by the encoder and the encoded text features.

[0123] Each decoding layer includes a masked self-attention structure, a cross self-attention structure, a feed-forward network layer, three residual connections, and three regularization layers. The expression of the first decoding layer is as follows:

[0124] Y1 0 = MaskedMSA(X′ word , X′ word , X′ word ) + X′ word (19)

[0125] Y1 1 = CrossMSA(Y1 0 , X EV , X Ev ) + Y1 0 (20)

[0126] Y1 = FFN(Y1 1 ) + Y1 1 (21)

[0127] wherein X′ word represents the text features after word mapping and position encoding, X EV represents the visual features input to the decoder after encoder enhancement, MaskedMSA represents the masked self-attention structure, CrossMSA represents the cross self-attention structure, FFN represents the feed-forward network layer, and Y1 represents the output of the first decoding layer.

[0128] The expression of the other decoding layers (the second, third, and fourth layers) is as follows:

[0129] Y i 0 = MaskedMSA(Y i-1 , Y i-1 , Y i-1 ) + Y i-1 (22)

[0130] Yi 1 = CrossMSA(Y i 0 , X Ev , X Ev ) + Y i 0 (23)

[0131] Y i = FFN(Y i 1 ) + Y i 1 (24)

[0132] where 2≤i≤M, i is a positive integer, M is the number of decoding layers, in the embodiment, M=4, Y i represents the feature output by the ith decoding layer.

[0133] The input of each mask self-attention structure only comes from the encoded text sequence, and each cross self-attention structure has two inputs, which are the text sequence and the visual feature, for fusing the text feature and the visual feature. The visual features input to the four decoding layers are the same, while the text features input to the decoding layers are different. For the first decoding layer, the input text feature comes from the encoded text sequence, while the text level input of the last three decoding layers comes from the output of the previous decoding layer.

[0134] After passing through the four decoding layers, the output feature is transformed into the dimension of the entire dictionary through a linear layer, and then passes through a Softmax classifier to obtain the probability of each word corresponding to the output feature. The specific feature processing process is as follows:

[0135] prob = Softmax(Linear(X decoder )) (25)

[0136] where X decoder represents the feature output by the last decoding layer, Linear represents a linear mapping layer, the dimension of the weight matrix of which is ω dim×vocab , dim represents the dimension of the feature of the decoder layer, and vocab represents the length of the dictionary. After passing through the Softmax function, the probability value prob of each predicted word is obtained.

[0137] Step 3) Train the built image description model on the training set constructed in step 1), describe the images in the test set, evaluate the model from two aspects of quantity and quality, and obtain the evaluation results and the descriptive sentences generated for each image.

[0138] After the image description model is built, the model is trained on the Cityscapes traffic scene dataset labeled in step 1), and the weight file obtained by training is stored in a.pth file. After the model training is completed, the test set image needs to be described by using the trained model, and the performance of the model is evaluated from the quantitative and qualitative two angles.

[0139] Quantitative evaluation: based on the labeled traffic scene dataset Cityscapes, under the same experimental setup, a plurality of currently advanced methods are used as the comparative method in the application, the evaluation index is B@1, B@2, B@3, B@4, METEOR, ROUGE, CIDEr, SPICE index commonly used in natural language processing, and the higher the value of the index, the higher the accuracy of the model for image description.

[0140] Qualitative evaluation: three images in the test set are selected, and the description results are as shown in Figure 8 It can be seen that the description results of most objects in the image by the method are better than the benchmark method.

[0141] The purpose of the application is to accurately describe the image of the traffic scene, and a transformer-based encoder-decoder model is provided, which specifically includes the following contents: first, in view of the problem of lack of image description dataset in the traffic scene, based on the existing traffic scene dataset Cityscapes, manual labeling is performed, a description sentence is labeled for each picture, and an image description dataset applied to the traffic scene is constructed. For each image, a pre-trained FasterRCNN target detector is used to extract the visual, category, attribute and position information in the image, and then a VSA module is used to fuse the multi-modal information, so as to better guide and enhance the visual features. The enhanced visual features and the geometric features of the target object are input into the transformer encoder structure for feature learning, which specifically includes 4 cascaded transformer encoder layers. At the end of the encoder structure, a spatial convolution pyramid SCP structure is built, and two one-dimensional convolution layers with different sizes are parallelly arranged to extract visual features from different size receptive fields. Then, the text sequence after word mapping and adding position information coding is input into the decoder model together with the visual feature sequence output by the encoder, and the visual and semantic features are fused. The output layer of the decoder maps the feature sequence to the length of the dictionary, calculates the probability of each word predicted by the Softmax function, and selects the predicted word according to the probability value to form the finally generated sentence. On the constructed traffic scene dataset, the method proposed in the application and a plurality of advanced methods are compared, and the method in the application is evaluated from the quantitative index and the qualitative sentence, which shows that the method in the application has greater advantages in description accuracy and comprehensiveness.

[0142] The preferred embodiments of the application have been described above in detail. It should be understood that modifications and variations to the preferred embodiments could be made by those skilled in the art in light of the teachings above. It is therefore contemplated that the application can encompass other variations and modifications that fall within the scope of the claims.

Claims

1. A transformer-based generative image description method for traffic scenes, characterized in that, Includes the following steps: Step 1) Manually annotate the existing traffic scene dataset to construct an image description dataset for traffic scenes, and divide it into a training set and a test set; Step 2) Construct an image description model, which includes a target detector, a multimodal feature fusion module, a transformer encoder, a text encoding module, and a decoder connected in sequence. The target detector extracts visual, attribute, category, and geometric location features from each image in the dataset. The multimodal feature fusion module fuses visual and semantic features from each image to obtain enhanced visual features, including attribute and category features. The transformer encoder takes the enhanced visual features and the corresponding geometric location features of the image as input and outputs the encoded visual features. The text encoding module encodes the text description of the image to obtain text features. The transformer decoder fuses the text features and the encoded visual features and outputs them through the output layer to predict words and generate descriptive sentences. Step 3) Train the image description model built on the training set constructed in Step 1), and describe the images in the test set. Evaluate the model from both quantitative and qualitative perspectives to obtain the evaluation results and the descriptive sentences generated for each image.

2. The method for generative image description of traffic scenes based on transformer according to claim 1, characterized in that, The manual annotation of the existing traffic scene dataset specifically involves: providing a paragraph description consisting of five sentences for each image as an annotation, describing the following: whether there are vehicles on both sides of the road ahead, whether there are pedestrians and cyclists around, the position of the vehicles, the position of the pedestrians, the color of the traffic lights, and the meaning of the traffic signs.

3. The method for generative image description of traffic scenes based on transformer according to claim 1, characterized in that, The multimodal feature fusion module concatenates the input attribute features, category features, and visual features. The concatenated features are then processed by a global average pooling layer to extract global vector representations, and then by a nonlinear activation function to calculate attention weight vectors. The obtained attention weights are used to guide and enhance the input visual features, resulting in enhanced visual features.

4. The method for generative image description of traffic scenes based on transformer according to claim 3, characterized in that, The expression for the multimodal feature fusion module is: X atten =ψ(GAP(C(X V ,X A ,X L ))) X EV =X atten ×X v +X v Among them, X V X A X L Let X represent the visual features, attribute features, and category features of the target object detected by the object detector in an image, respectively. C represents the concatenation of these three features along the feature length dimension. GAP represents the global average pooling operation, ψ represents the Sigmoid activation function, and X represents the object's visual features, attribute features, and category features detected by the object detector in an image. atten Let X represent the attention vector. EV This represents the visual features enhanced by the multimodal feature fusion module.

5. The method for generative image description of traffic scenes based on transformer according to claim 1, characterized in that, The encoder includes multiple sequentially connected encoding layers and a spatial convolutional pyramid module. The input of each encoding layer includes visual features and geometric position features detected by the object detector, and the output is encoded visual features. The visual features input to the first encoding layer are enhanced visual features by the multimodal feature fusion module, and the visual features input to the remaining encoding layers are the visual features output by the previous encoding layer. The spatial convolutional pyramid module uses two one-dimensional convolutional layers with different kernel sizes connected in parallel to enhance the visual features output by the last encoding layer from the perspective of extracting receptive fields from two types of features.

6. The method for generative image description of traffic scenes based on transformer according to claim 5, characterized in that, Each coding layer comprises a geometric multi-head self-attention layer, a feedforward network layer, two residual connections, and two regularization layers. The expression for the first coding layer is: X1’=GeoMSA(LN(X EV ),X G )+X EV X1 = FFN(LN(X1')) + X1' Among them, X EV X represents the enhanced visual features after the multimodal feature fusion module. G This represents the geometric position features of the target detected by the target detector. GeoMSA represents the geometric multi-head self-attention layer, LN represents the layer regularization operation, FFN represents the feedforward neural network layer, X1' represents the feature vector obtained after passing through the multi-head self-attention layer, and X1 represents the visual features output by the first encoding layer. The expressions for the other coding layers are: X i ’=GeoMSA(LN(X i-1 ),X G )+X i-1 X i =FFN(LN(X i ’))+X i ’ Where 2≤i≤M, i is a positive integer, M is the number of coding layers, and X i This represents the visual feature output by the i-th coding layer.

7. The method for generative image description of traffic scenes based on transformer according to claim 5, characterized in that, The expression for the spatial convolutional pyramid module is: X c =C(Conv1(X M ),Conv3(X M )) X atten =Softmax(GAP(X c )) X EV =X atten ×X M +X M Among them, X M This represents the visual features output by the last encoding layer. Conv1 is a one-dimensional convolutional layer with a kernel size of 1, Conv3 is a one-dimensional convolutional layer with a kernel size of 3, C indicates concatenation of the two feature vectors along the feature length dimension, and X... c The concatenated feature vectors are represented by GAP, which indicates the global average pooling operation, and Softmax, which indicates the Softmax function. atten Let X represent the attention weight vector. EV This represents the visual features enhanced by the spatial convolutional pyramid module, i.e., the features finally output by the encoder.

8. The method for generative image description of traffic scenes based on transformer according to claim 1, characterized in that, The text encoding module performs the following steps: Merge all sentences labeled in all images, remove punctuation, convert all words to lowercase, remove duplicate words and words with low frequency, and obtain the dictionary corresponding to the current dataset; Assign a number to each word in the dictionary; Map the currently labeled sentence to a feature sequence; Based on the words in the dictionary and their corresponding numbers, a linear layer is used to add positional encoding to each word in the current feature sequence to obtain text features.

9. The method for generative image description of traffic scenes based on transformer according to claim 1, characterized in that, The transformer decoder includes multiple decoding layers, wherein the number of decoding layers is the same as the number of encoding layers in the encoder. Each decoding layer contains a masked self-attention structure, a cross self-attention structure, a feedforward network layer, three residual connections, and three regularization layers. The expression for the first decoding layer is: Y1 0 =MaskedMSA(X′ word ,X′ word ,X′ word )+X′ word Y1 1 =CrossMSA(Y1 0 ,X EV ,X EV )+Y1 0 Y1=FFN(Y1 1 )+Y1 1 Where, X′ word X represents the text features after word mapping and position encoding. EV Y1 represents the visual features input to the decoder after being enhanced by the encoder. MaskedMSA represents the masked self-attention structure, CrossMSA represents the cross self-attention structure, FFN represents the feedforward network layer, and Y1 represents the output of the first decoding layer. The expressions for the other decoding layers are: AND i 0 =MaskedMSA(Y i-1 ,AND i-1 ,AND i-1 )+Y i-1 Y i 1 =CrossMSA(Y i 0 ,X EV ,X EV )+Y i 0 Y i =FFN(Y i 1 ) + Y i 1 Where 2≤i≤M, i is a positive integer, M is the number of decoding layers, and Y i This represents the feature output by the i-th decoding layer.

10. The method for generative image description of traffic scenes based on transformer according to claim 1, characterized in that, The output layer transforms the feature dimensions of the decoder output to the dimensions of the entire dictionary through a linear layer, and uses Softmax for classification to obtain the probability of each word.

Images