A city traffic target recognition method based on cross-modal local cross attention

Abstract

The application provides a city traffic target recognition method based on cross-modal local cross attention, improves the SSAN core network, replaces the non-local attention network with a cross-modal local cross attention network, dynamically interacts local features obtained by image blocking with local features of a text semantic subpart, realizes cross-modal fine-grained feature alignment, and improves the fine-grained recognition ability of the model to targets under a shielding condition. The finally obtained network model has a good recognition accuracy to traffic targets under a shielding condition, and compared with global cross attention, the local cross attention of the application has a lower complexity and improves the calculation efficiency. The retrieval precision is improved by more than 15% under a shielding condition, and important technical support is provided for constructing an intelligent traffic management system.

Description

TECHNICAL FIELD

[0001] The present application relates to cross-modal target detection of image text, in particular to a city traffic target recognition method based on cross-modal local cross-attention. BACKGROUND

[0002] With the rapid rise of smart city construction and the in-depth development of intelligent traffic management system, today's traffic target accurate retrieval technology has become an important demand to realize the intelligent management of urban traffic. The current mainstream cross-modal retrieval method such as SSAN (Structured Self-Attention Network) still has obvious shortcomings in dealing with complex traffic scenes, especially in the presence of occlusion, its retrieval accuracy decreases significantly. The main reasons for this problem are as follows: first, there is significant heterogeneity between image data and text data in the feature space, which makes it difficult to align the cross-modal features of images and texts; second, the traditional global attention mechanism is difficult to establish the fine-grained correlation between the local regions of the image and the keywords of the text. SUMMARY

[0003] In order to solve the above problems, the present application provides a city traffic target recognition method based on cross-modal local cross-attention, which replaces and adds a cross-modal local cross-attention network in the main SSAN network, and improves the fine-grained recognition ability of the model under the occlusion condition. Through the improved double loss function cooperative optimization and the new evaluation function based on keyword co-occurrence, the accuracy of the model in many-to-many matching is improved.

[0004] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0005] A city traffic target recognition method based on cross-modal local cross-attention, comprising the following steps:

[0006] Step 1: Make a traffic object data set with text description, make a natural language description of the traffic scene data set, and divide it into training set, validation set and test set, then perform preprocessing operation on the data set data;

[0007] Step 2: Use the pre-trained ResNet-50 and bidirectional LSTM to process image information and text information respectively to obtain two modal feature maps; two modal feature maps are obtained by global maximum pooling respectively to obtain two global features, and the feature maps are reduced in dimension by convolution and segmented to obtain two local features of the two modalities;

[0008] Step three: replace the non-local attention network in the SSAN network model with a cross-modal local cross-attention network; in this network model, first, generate Key vectors, Value vectors and Query vectors of local features and global features of image and text modalities respectively through double-branch projection, then calculate double-feature projection self-attention of local features and global features within the image or text modality; then, calculate cross-modal scaled dot product attention using the Query vector of the current modality and the Key vector and Value vector of the other modality; finally, the model fuses the two attentions through a dynamic gating mechanism, introduces learnable gating parameters to regulate the weights of the two attentions, and uses a residual connection to retain some original features, finally obtains cross-modal local cross-attention, and realizes fine-grained alignment matching of traffic scene targets;

[0009] Step four: train the model through the training set and use a double-loss function cooperative optimization mechanism during the training process to optimize the loss function; after the optimization of the loss function is completed, use the evaluation function based on keyword co-occurrence to return all indexes that meet the matching conditions by detecting the co-occurrence of query text and candidate text, and realize city traffic target identification.

[0010] In step one, when the data set data is preprocessed, the model processes the image in two modes: when it is in training mode, the model performs random horizontal flipping, order shuffling, uniform size adjustment, tensor conversion and normalization processing on the image, which can increase the diversity of the data; when it is in test mode, the model does not perform random horizontal flipping on the image to ensure the stability of the test results; for text, the model normalizes the text sequence by padding and truncation to convert the text into fixed-length text data.

[0011] In step two, in the step of processing image information using the pre-trained ResNet-50, the global feature is obtained by performing global max pooling on the feature map obtained by ResNet-50; the feature map is horizontally divided into K sub-regions, and a convolutional dimension reduction is independently used for each local feature map to generate K local features; finally, the model processes the obtained features through a LeakyReLU activation function to complete the preliminary feature extraction of the image, where K is preferably set to 6.

[0012] In step two, in the step of processing text information using bidirectional LSTM, the text data after data preprocessing is packed using a packing sequence; after the packed sequence is processed by bidirectional LSTM, the text feature map is obtained by unpacking;

[0013] The text global feature is obtained by performing global max pooling on the text feature map; for the text local feature, the dynamic word weight operation is used when the text feature map is subjected to convolutional dimension reduction and segmentation:

[0014] In all text descriptions, there will be different text descriptions of the same id picture. When extracting local features, the model will obtain the text features associated with the same picture id for embedding comparison, and dynamically assign weights according to the frequency of repeated occurrence of features in multiple descriptions. This mechanism can highlight the local features with important parts and enhance the discriminability of text local features, that is, the preliminary extraction of local features is completed.

[0015] The cross-modal local cross-attention network in step three realizes the fine-grained alignment of images and texts through double-branch projection, dynamic gate fusion and residual connection.

[0016] The model first projects the local features and global features of the two modalities of images and texts through double-branch projection, respectively, to generate Key vectors, Value vectors and Query vectors of the local features and global features of the two modalities of images and texts, as follows:

[0017] K img =W·F img ,K txt =W·F txt

[0018] V img =W·F img ,V txt =W·F txt (1)

[0019] Q img =W·F img ,Q txt =W·F txt

[0020] In formula (1), F img , F txt represent the image and text local features, respectively, K img , K txt represent the Key vectors of the image and text local features, respectively, V img , V txt represent the Value vectors of the image and text local features, respectively, Q img , Q txt represent the Query vectors of the image and text local features, respectively, and W represents a dimension reduction conv convolution layer.

[0021] Then, through the three vectors, double-feature projection self-attention is calculated within the image or text modality, and the formula is as follows:

[0022] (2)

[0023] In formula (2), representing the dual feature projection self-attention, K and Q represent the Key vector and Query vector of the local feature within the modal, respectively;

[0024] Then, the cross-modal scaled dot-product attention, i.e., image cross text scaled dot-product attention or text cross image scaled dot-product attention, is calculated using the Q vector of the current modal and the Key vector and Value vector of the other modal, and the formula is as follows:

[0025] (3)

[0026] (4)

[0027] In formula (3) and (4), represents the cross-modal scaled dot-product attention weight matrix, represents the cross-modal scaled dot-product attention, is a scaling factor, Q represents the Q vector of the current modal, K' represents the Key vector of the other modal, and V' represents the Value vector of the other modal;

[0028] Finally, the model fuses the two attentions through a dynamic gating mechanism, introduces a learnable gating parameter to regulate the weights of the two attentions, and uses a residual connection to retain a certain amount of original features, and finally obtains the cross-modal local cross-attention, i.e., cross-modal feature:

[0029] (5)

[0030] (6)

[0031] In formula (5) and (6), cross_gate represents the gating parameter, whose initial value is 0, and embedding represents the initial local feature;

[0032] After obtaining the global feature, the local feature and the cross-modal feature, in the calculation process of the loss function, the model calculates the global similarity matrix , the local similarity matrix and the cross-modal similarity matrix respectively, and the formula is as follows:

[0033]

[0034] (7)

[0035]

[0036] In formula (7), represents the image global feature, represents the text global feature, representing image local features, representing text local features, representing image cross-modal features, representing text cross-modal features.

[0037] Then, the similarity matrix is calculated using the weighted fusion method , and the formula is as follows:

[0038] (8)

[0039] In formula (8), the three weight parameters are set as: α=0.5, β=0.3, and γ=0.2, the matrix is composed of elements s(i, j), and s(i, j) represents the matching score of the i-th image and the j-th text.

[0040] The double-loss function cooperative optimization in step four improves the accuracy of the model by combining ID Loss and Ranking Loss. ID Loss reduces the intra-class variance to enhance the ability to identify targets of the same class, while Ranking Loss increases the difference between positive and negative samples, thereby increasing the inter-class variance, so that the model can better distinguish targets of different classes. The final loss is the weighted fusion of the two, and the formula is as follows:

[0041] (9)

[0042] In formula (9), represents the weight of the ID Loss function, represents the weight of the Ranking Loss function, represents the ID Loss function, represents the Ranking Loss function. and are set to 0.4 and 0.6, respectively.

[0043] After the optimization of the loss function is completed, the model uses an evaluation function based on keyword co-occurrence to return all indexes that meet the matching conditions by detecting the co-occurrence of query text and candidate text, breaking through the strict ID comparison restriction in traditional evaluation functions.

[0044] The beneficial effects of the present application are:

[0045] The application provides a city traffic target recognition method based on cross-modal local cross-attention, realizes cross-modal fine-grained feature alignment by dynamically interacting local features obtained by image blocking with local features of text semantic subparts, improves the fine-grained recognition ability of the model to targets under the condition of occlusion by replacing and adding a cross-modal local cross-attention network in the main SSAN network. The finally obtained network model has good recognition accuracy for traffic targets under the condition of occlusion, and compared with global cross-attention, the local cross-attention of the application has lower complexity and improves the calculation efficiency. The retrieval precision is improved by more than 15% under the condition of occlusion, which provides important technical support for building an intelligent traffic management system. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The data preprocessing process of the city traffic target recognition method based on cross-modal local cross-attention.

[0047] Figure 2 The feature extraction flowchart of the city traffic target recognition method based on cross-modal local cross-attention.

[0048] Figure 3 The core network flowchart of the city traffic target recognition method based on cross-modal local cross-attention.

[0049] Figure 4 The keyword co-occurrence evaluation flowchart of the city traffic target recognition method based on cross-modal local cross-attention.

[0050] Figure 1 And Figure 2 The English sentence "A man in blue is walking on a zebra-crossing in the street with railings and barricades behind him." is an example of natural language description of traffic scene data set. DETAILED DESCRIPTION

[0051] The application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement it according to the description.

[0052] The application first provides a city traffic target recognition method based on cross-modal local cross-attention, and the implementation flowchart is as shown in the figure, and the specific method is as follows:

[0053] A city traffic target recognition method based on cross-modal local cross-attention, comprising the following steps:

[0054] Step one: make a traffic object data set with text description, make natural language description for the traffic scene data set, and divide the training set, validation set and test set, then perform preprocessing operation on the data set data.

[0055] When the data set data is preprocessed in step one, the model processes the image in two modes: when it is in training mode, the model performs random horizontal flip, sequence shuffle, uniform size adjustment, tensor conversion and normalization processing on the image, which can increase the diversity of data; when it is in test mode, the model does not perform random horizontal flip on the image to ensure the stability of the test result; for the text, the model normalizes the text sequence in length, and converts the text into fixed-length text data by padding and truncation.

[0056] Step two: use pre-trained ResNet-50 and bidirectional LSTM to process image information and text information respectively to obtain two modal feature maps; two modal feature maps are obtained by global maximum pooling respectively; the feature maps are reduced in dimension and segmented to obtain two modal local features;

[0057] In the step of using pre-trained ResNet-50 to process image information in step two, the global feature is obtained by performing global maximum pooling on the feature map obtained by ResNet-50; the feature map is horizontally segmented into K sub-regions, and convolution dimension reduction is independently used for each local feature map to generate K local features, and finally the model processes the obtained features through LeakyReLU activation function to complete the preliminary feature extraction of the image, wherein K is set to 6.

[0058] In the step of using bidirectional LSTM to process text information in step two, the text data after data preprocessing is packed using the packing sequence processing; after the packed sequence is processed by bidirectional LSTM, the text feature map can be obtained by unpacking;

[0059] The text global feature is obtained by performing global maximum pooling on the text feature map; for the text local feature, the dynamic word weight operation is used when the text feature map is reduced in dimension and segmented:

[0060] In all text descriptions, there will be different text descriptions of the same id picture, and when extracting local features, the model will obtain the text features associated with the same picture id for embedding comparison, and dynamically allocate weights according to the frequency of repeated appearance of the features in multiple descriptions. This mechanism can highlight the local features with important parts and enhance the discriminability of the text local features, that is, the preliminary extraction of the local features is completed.

[0061] Step three: improve the SSAN core network, replace its non-local attention network with a cross-modal local cross-attention network, in which the Key vector, Value vector and Query vector of the local features and global features of the image and text modalities are first generated through double-branch projection, then the double-feature projection self-attention of the local features and global features is calculated within the image or text modality, then the cross-modal scaled dot product attention is calculated using the Query vector of the current modality and the Key vector and Value vector of the other modality, finally, the model fuses the two attentions through a dynamic gating mechanism, introduces learnable gating parameters to regulate the weights of the two attentions, and uses a residual connection to retain some original features, and finally obtains cross-modal local cross-attention, realizing fine-grained alignment matching of traffic scene targets;

[0062] The cross-modal local cross-attention network in step three realizes fine-grained alignment of images and text through double-branch projection, dynamic gating fusion and residual connection;

[0063] The model first projects the local features and global features of the image and text modalities through double-branch projection, i.e. projects the local features and global features of the image and text modalities respectively to generate Key vectors, Value vectors and Query vectors of the local features and global features of the image and text modalities, as follows:

[0064] K img =W·F img ,K txt =W·F txt

[0065] V img =W·F img ,V txt =W·F txt (1)

[0066] Q img =W·F img ,Q txt =W·F txt

[0067] In formula (1), F img , F txt represent the image and text local features, K img , K txt represent the Key vectors of the image and text local features, V img , V txt represent the Value vectors of the image and text local features, Q img , Q txtQuery vectors representing image and text local features respectively, W refers to a dimension reduction conv layer;

[0068] Then, through the three vectors, the double feature projection self-attention is calculated within the image or text modal, and the formula is:

[0069]

[0070] In the formula, represents the double feature projection self-attention, K and Q represent the Key vector and Query vector of the local feature in the modal respectively;

[0071] After that, the Q vector of the current modal and the Key vector and Value vector of the other modal are used to calculate the cross-modal scaled dot-product attention, i.e. image cross-text scaled dot-product attention or text cross-image scaled dot-product attention, and the formula is:

[0072]

[0073]

[0074] In the formula, represents the cross-modal scaled dot-product attention weight matrix, represents the cross-modal scaled dot-product attention, is a scaling factor; K' represents the Key vector of the other modal, and V' represents the Value vector of the other modal;

[0075] Finally, the model fuses the two attentions through a dynamic gating mechanism, introduces a learnable gating parameter to regulate the weight of the two attentions, and uses a residual connection to retain a certain amount of original features, and finally obtains the cross-modal local cross-attention, i.e. cross-modal feature:

[0076]

[0077]

[0078] cross_gate represents the gating parameter, whose initial value is 0, and embedding represents the initial local feature;

[0079] After obtaining the global feature, the local feature and the cross-modal feature, in the calculation process of the loss function, the model calculates the global similarity matrix , the local similarity matrix and the cross-modal similarity matrix respectively. The formula is:

[0080]

[0081]

[0082]

[0083] wherein represents image global features, represents text global features, represents image local features, represents text local features, represents image cross-modal features, represents text cross-modal features.

[0084] Then, the similarity matrix S is calculated using the weighted fusion method , and the formula is as follows:

[0085]

[0086] wherein three weight parameters are set as: a = 0.5, b = 0.3, and g = 0.2, the matrix is composed of elements s(i, j), and s(i, j) represents the matching score of the i-th image and the j-th text.

[0087] Step four: the model is trained through the training set, and a double-loss function collaborative optimization mechanism is used in the training process for engineering optimization. After the optimization of the loss function is completed, an evaluation function based on keyword co-occurrence is used to return all indexs that meet the matching conditions by detecting the co-occurrence of query texts and candidate texts, so as to realize the city traffic target recognition.

[0088] The double-loss function collaborative optimization in step four combines ID Loss and Ranking Loss to improve the accuracy of the model. ID Loss reduces the intra-class variance to enhance the ability to identify targets of the same class, while Ranking Loss increases the difference between positive and negative samples, thereby increasing the inter-class variance, so that the model can better distinguish targets of different classes. The final loss is a weighted fusion of the two, and the formula is as follows:

[0089]

[0090] wherein, represents the weight of the ID Loss function, represents the weight of the Ranking Loss function, represents the IDLoss function, represents the Ranking Loss function; and The weights of and in the model are set to 0.4 and 0.6.

[0091] After the optimization of the loss function is completed, the model uses an evaluation function based on keyword co-occurrence to return all indexes that meet the matching conditions by detecting the co-occurrence of the query text and the candidate text, breaking through the strict comparison of IDs in the traditional evaluation function.

[0092] While embodiments of the application have been disclosed in connection with the above specification and drawings this description is not intended to limit the scope of the application and it will be readily appreciated that many modifications can be made to the described embodiments without departing from the spirit and scope of the application as set forth in the following claims.

Claims

1. A method for urban traffic target recognition based on cross-modal local cross-attention, characterized in that, The method includes the following steps: Step 1: Create a traffic object dataset with text descriptions, perform natural language descriptions on the traffic scene dataset, and divide it into training, validation, and test sets. Then, perform preprocessing operations on the dataset data. Step 2: Use pre-trained ResNet-50 and bidirectional LSTM to process image information and text information respectively to obtain feature maps of two modalities; use global max pooling to obtain two types of global features from the feature maps of the two modalities, and use convolution dimensionality reduction and segmentation to obtain local features of the two modalities. Step 3: Replace the non-local attention network in the SSAN network model with a cross-modal local cross-attention network. In this network model, firstly, key vectors, value vectors, and query vectors of local and global features for both image and text modalities are generated through bi-branch projection. Then, bi-feature projection self-attention of local and global features is calculated within the image or text modality. After that, cross-modal scaled dot product attention is calculated using the query vector of the current modality and the key and value vectors of the other modality. Finally, the model fuses these two attentions through a dynamic gating mechanism, introduces learnable gating parameters to regulate the weights of these two attentions, and uses residual connections to preserve certain original features, ultimately obtaining cross-modal local cross-attention to achieve fine-grained alignment and matching of targets in traffic scenarios. Step 4: Train the model using the training set and perform engineering optimization using a dual loss function collaborative optimization mechanism during training. After optimizing the loss function, use an evaluation function based on keyword co-occurrence to detect the co-occurrence of query text and candidate text, and return all indices that meet the matching conditions to achieve urban traffic target recognition.

2. The urban traffic target recognition method based on cross-modal local cross-attention as described in claim 1, characterized in that, In step one, when preprocessing the dataset, the model processes images in two modes: In training mode, the model performs random horizontal flipping, shuffling, uniform size adjustment, tensor transformation, and normalization on the images to increase data diversity; In testing mode, the model does not perform random horizontal flipping on the images to ensure the stability of the test results; For text, the model standardizes the length of the text sequence and converts the text into fixed-length text data through padding and truncation.

3. The urban traffic target recognition method based on cross-modal local cross-attention as described in claim 1, characterized in that, In step two, which uses a pre-trained ResNet-50 to process image information, global features are obtained by performing global max pooling on the feature map obtained by ResNet-50. The feature map is then horizontally divided into K sub-regions, and convolutional dimensionality reduction is performed independently on each local feature map to generate K local features. Finally, the model processes the obtained features using the LeakyReLU activation function to complete the initial feature extraction of the image.

4. The urban traffic target recognition method based on cross-modal local cross-attention as described in claim 1, characterized in that, In step two, which uses bidirectional LSTM to process text information, the preprocessed text data is first packaged using a packing sequence. After the packing sequence is processed by bidirectional LSTM, the text feature map can be obtained by unpacking. Global max pooling is performed on the text feature map to obtain the global text features; for the local text features, dynamic word weights are used during convolutional dimensionality reduction and segmentation of the text feature map. In all text descriptions, there will be different text descriptions of the same image ID. When extracting local features, the model will obtain the text features associated with the same image ID and embed them for comparison. The weights will be dynamically assigned according to the frequency of the features in multiple descriptions to enhance the discriminativeness of local text features and complete the initial extraction of local features.

5. The urban traffic target recognition method based on cross-modal local cross-attention as described in claim 1, characterized in that, In step three, the cross-modal local cross-attention network achieves fine-grained alignment between images and text through bi-branch projection, dynamic gating fusion, and residual connections. The model first uses a two-branch projection approach, projecting the local and global features of both the image and text modalities separately to generate Key, Value, and Query vectors for the local and global features of the image and text modalities, respectively, as shown in the following formula: K img =W·F img ,K txt =W·F txt V img =W·F img ,V txt =W·F txt (1) Q img =W·F img ,Q txt =W·F txt In equation (1), F img F txt K represents local features of the image and text, respectively. img K txt Key vectors representing local features of the image and text, respectively, V img V txt Value vectors and Q vectors represent local features of the image and text, respectively. img Q txt The Query vectors represent local features of the image and text, respectively, and W refers to a dimension-reduced conv convolutional layer; Then, using these three vectors, the dual-feature projection self-attention is calculated within the image or text modality, as shown in the formula: （2） In equation (2), This represents dual-feature projection self-attention, where K and Q represent the Key vector and Query vector of the local features within the modality, respectively. Next, the Q-vector of the current modality is used with the Key and Value vectors of the other modality to calculate the cross-modal scaling dot product attention, i.e., image-to-text scaling dot product attention or text-to-image scaling dot product attention. The formula is as follows: （3） （4） In equations (3) and (4), This represents the cross-modal scaled dot product attention weight matrix. Represents cross-modal scaled dot product attention. The scaling factor is Q, where Q represents the Q vector of the current mode, K' represents the Key vector of another mode, and V' represents the Value vector of another mode. Finally, the model fuses these two attention mechanisms through a dynamic gating mechanism, introducing learnable gating parameters to regulate the weights of these two attentions, and using residual connections to preserve some of the original features, ultimately obtaining cross-modal local cross-attention, i.e., cross-modal features: （5） （6） In equations (5) and (6), cross_gate represents the gate parameter, which has an initial value of 0, and embedding represents the initial local features; After obtaining the global features, local features, and cross-modal features, the model calculates the global similarity matrix during the loss function calculation process. Local similarity matrix and cross-modal similarity matrix ; The formula is: （7） In equation (7), Represents global features of an image, Representing global features of text Representing local features of an image Representing local features of text, Representing cross-modal features of images, Represents cross-modal features of text; Then, a weighted fusion method is used to calculate the similarity matrix. The formula is as follows: （8） In equation (8), the three weight parameters are set as follows: α=0.5, β=0.3, γ=0.

2. The matrix is ​​composed of elements s(i,j), where s(i,j) represents the matching score between the i-th image and the j-th text.

6. The urban traffic target recognition method based on cross-modal local cross-attention according to claim 1, characterized in that, The collaborative optimization of the two loss functions in step four combines ID Loss and Ranking Loss, and the final loss is a weighted fusion of the two, as shown in the following formula: （9） In equation (9), The weights represent the ID Loss function. The weights represent the weights of the Ranking Loss function. Represents the IDLoss function. Represents the Ranking Loss function; and The weights were set to 0.4 and 0.6 respectively.

Images