Traffic large model intelligent traffic event identification method based on cross-modal retrieval

Abstract

The invention discloses a large traffic model intelligent traffic event identification method based on cross-modal retrieval. The method comprises the following steps: inputting an image of a traffic event to be identified and text description thereof into a traffic event identification network model; a visual feature extraction module is used for extracting visual features in the image; the text feature extraction module is used for extracting text features in the text description; the visual selective filtering module is used for selectively filtering the irrelevant visual extraction features to obtain visual filtering features; the text selective filtering module is used for selectively filtering unnecessary text extraction features to obtain text filtering features; the cross-modal selective alignment module is used for performing cross-modal alignment on the visual and text extraction features to obtain visual alignment features and text alignment features; and the calculation module is used for fusing the visual filtering features and the visual alignment features as well as the text filtering features and the text alignment features to determine an identification result. According to the invention, the accuracy and efficiency of retrieval identification are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of large traffic models, and in particular relates to an intelligent traffic event recognition method for large traffic models based on cross-modal retrieval. Background Art

[0002] With the increasing complexity of urban traffic systems, the frequent occurrence of traffic incidents has brought great challenges to traffic management and monitoring. However, accurate traffic incident identification can effectively solve these problems and provide valuable reference information for traffic management departments. First, accurate traffic incident identification can help traffic management departments better coordinate traffic resources and emergency measures. Through cross-modal retrieval, identification and positioning of multimodal data such as traffic scene images and text descriptions, management departments can grasp road conditions in real time, reasonably arrange police forces and rescue equipment, and flexibly adjust traffic control strategies. This can not only ensure road safety and smooth traffic, but also maximize the utilization of traffic resources. Secondly, through intelligent identification of traffic incidents, traffic accidents, congestion and violations can be discovered in time, and measures can be taken quickly to reduce traffic delays and accidents. This refined management method helps to optimize traffic flow and enhance the public travel experience. Through efficient traffic incident identification, the reliance on human monitoring is reduced, and expenses are saved in the rational planning of transportation infrastructure and the optimization of resource allocation.

[0003] Most of the existing cross-modal retrieval technologies are based on deep learning and are mainly divided into coarse-grained matching and fine-grained matching. Among them, the fine-grained matching method can more accurately capture the subtle correspondence between images and texts. For example: a fine-grained matching method based on the attention mechanism, whose basic idea is: introduce the attention mechanism, dynamically pay attention to the key features in the image and text, and assign weights according to their importance to improve the accuracy of matching. In this way, the model can more effectively align the detailed information between the image and the text; a fine-grained matching method based on a graph neural network, whose basic idea is: construct the feature representation of the image and text into a graph structure, and use the graph neural network to capture the complex relationship and structural information between multimodal data; by performing information propagation and feature aggregation on the graph, the model can more deeply understand and associate data of different modalities, thereby improving the performance of cross-modal retrieval. These methods perform well in dealing with the complexity and diversity of traffic events, and help improve the accuracy and efficiency of traffic event identification and positioning.

[0004] However, the above existing methods have shortcomings: in the fine-grained matching method based on the attention mechanism, not every image contains a valid area, and every word in a sentence does not necessarily contribute to semantic alignment. Useless image areas and meaningless words (such as prepositions) may lead to redundant semantic associations and alignments, reducing the consistency of cross-modal representation alignment; in the fine-grained matching method based on graph neural networks, although the graph learning-based network can capture the complex relationship between modalities, it consumes large computing resources and has low training efficiency when processing large-scale data. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The embodiment of the present invention provides a method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval, the method comprising:

[0007] Collect images of traffic events to be identified and their corresponding text descriptions;

[0008] The image of the traffic event to be identified and its corresponding text description are input into the trained traffic event identification network model to obtain the identification result of the traffic event to be identified; wherein,

[0009] The trained traffic event recognition network model includes a visual feature extraction module, a text feature extraction module, a visual selective filtering module, a text selective filtering module, a cross-modal selective alignment module and a calculation module; the visual feature extraction module is used to extract visual features in the image of the traffic event to be identified and map the extracted visual features to a preset dimension to obtain visual extraction features; the text feature extraction module is used to extract text features in the text description corresponding to the traffic event to be identified and map the extracted text features to the preset dimension to obtain text extraction features; the visual selective filtering module is used to selectively filter irrelevant visual extraction features to obtain visual filtering features; the text selective filtering module, Used to selectively filter unnecessary text extraction features to obtain text filtering features; the cross-modal selective alignment module is used to perform cross-modal alignment processing on the visual extraction features and the text extraction features to obtain visual alignment features and text alignment features; the calculation module is used to fuse the visual filtering features and the visual alignment features to obtain visual fusion features, and fuse the text filtering features and the text alignment features to obtain text fusion features, calculate the similarity between the visual fusion features and the text fusion features, determine whether the similarity meets the similarity threshold, and use the image of the traffic event corresponding to the visual fusion feature that meets the similarity threshold as the recognition result of the traffic event to be recognized.

[0010] In one embodiment of the present invention, the visual feature extraction module is implemented using a Faster R-CNN network.

[0011] In one embodiment of the present invention, the text feature extraction module is implemented using a BERT encoder.

[0012] In one embodiment of the present invention, the visual selective filtering module includes a first RMS normalization layer, a first fully connected layer to a fifth fully connected layer, a first convolutional layer, a second convolutional layer, a first activation layer to a third activation layer, a first SSM module and a second SSM module; wherein,

[0013] The visual extraction feature is input into the first RMS normalization layer, the output end of the first RMS normalization layer is connected to the input ends of the first fully connected layer and the second fully connected layer respectively, the output end of the first fully connected layer is connected to the first convolutional layer, the first activation layer, and the first SSM module in sequence, the output end of the second fully connected layer is connected to the input end of the second activation layer, the output end of the second activation layer is point multiplied with the output end of the first SSM module to obtain a first point multiplication result, the output data of the first RMS normalization layer is flipped and respectively input into the third fully connected layer and the fourth fully connected layer, the output end of the third fully connected layer is connected to the second convolutional layer, the third activation layer, and the second SSM module in sequence, the output end of the fourth fully connected layer is connected to the input end of the fourth activation layer, the output end of the fourth activation layer is point multiplied with the output end of the second SSM module to obtain a second point multiplication result, the second point multiplication result is flipped and summed with the first point multiplication result and the summed result is input into the fifth fully connected layer, the output end of the fifth fully connected layer is summed with the visual extraction feature as the output of the visual selective filtering module.

[0014] In one embodiment of the present invention, the visual selective filtering module further includes a first average pooling layer; the first average pooling layer is used to perform average pooling processing on the visual filtering features and input the visual filtering features after the average pooling processing into the calculation module.

[0015] In one embodiment of the present invention, the text selective filtering module includes a second RMS normalization layer, a sixth fully connected layer to an eighth fully connected layer, a third convolutional layer, a fifth activation layer, a sixth activation layer and a third SSM module; wherein,

[0016] The text extraction feature is input into the second RMS normalization layer, the output end of the second RMS normalization layer is respectively connected to the input ends of the sixth fully connected layer and the seventh fully connected layer, the output end of the sixth fully connected layer is connected to the third convolutional layer, the fifth activation layer, and the third SSM module in sequence, the output end of the seventh fully connected layer is connected to the input end of the sixth activation layer, the output end of the sixth activation layer is dot-multiplied with the output end of the third SSM module and then input into the eighth fully connected layer, and the output end of the eighth fully connected layer and the text extraction feature are summed as the output of the text selective filtering module.

[0017] In one embodiment of the present invention, a second average pooling layer is further included after the text selective filtering module; the second average pooling layer is used to perform average pooling processing on the text filtering features, and input the text filtering features after the average pooling processing into the calculation module.

[0018] In one embodiment of the present invention, the cross-modal selective alignment module includes a concatenation layer, a third RMS normalization layer, a ninth fully connected layer to an eleventh fully connected layer, a fourth convolutional layer, a seventh activation layer, an eighth activation layer, a fourth SSM module and a splitting layer; wherein,

[0019] The visual extraction features and the text extraction features are input into the splicing layer, the output end of the splicing layer is connected to the input end of the third RMS normalization layer, the output end of the third RMS normalization layer is respectively connected to the input ends of the ninth fully connected layer and the tenth fully connected layer, the output end of the ninth fully connected layer is connected to the fourth convolutional layer, the seventh activation layer, and the fourth SSM module in sequence, the output end of the tenth fully connected layer is connected to the input end of the eighth activation layer, the output end of the eighth activation layer is dot-multiplied with the output end of the fourth SSM module and then input into the eleventh fully connected layer, the output end of the eleventh fully connected layer is summed with the output end of the splicing layer and then input into the splitting layer, and the output data of the splitting layer is used as the output of the cross-modal selective alignment module.

[0020] In one embodiment of the present invention, after the cross-modal selective alignment module, a third average pooling layer and a fourth average pooling layer are also included; the third average pooling layer is used to perform average pooling processing on the visual alignment features, and input the visual alignment features after the average pooling processing into the calculation module; the fourth average pooling layer is used to perform average pooling processing on the text alignment features, and input the text alignment features after the average pooling processing into the calculation module.

[0021] In one embodiment of the present invention, during the training process of the traffic event recognition network model, the loss function used is a two-way triplet contrast loss, which is expressed as follows:

[0022]

[0023] Where L represents the two-way triplet contrast loss, m represents the margin hyperparameter, [] + represents the max(0,·) operation, which means selecting the maximum value between 0 and another data. represents an image of any traffic event in the training sample set, Indicates the training sample set The corresponding text description, and Constitute a positive sample pair, Indicates that the training sample set Any image of a traffic incident outside Indicates that the training sample set The text description corresponding to any traffic event other than and and They constitute negative sample pairs respectively, and sim() represents the similarity function used to measure the similarity between image features and text features.

[0024] Beneficial effects of the present invention:

[0025] The invention proposes a method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval. The visual selective filtering module, the text selective filtering module and the cross-modal selective alignment module are introduced into the designed traffic event recognition network model. By implicitly and selectively matching the image and text of the traffic event, the model can effectively filter out irrelevant information, strengthen the perception and alignment of key features, thereby improving the accuracy and speed of cross-modal retrieval, and greatly improving the ability to accurately identify and locate traffic events. At the same time, the representation learned by the model is optimized and updated, solving the problem of invalid semantic alignment and high computational complexity caused by global allocation in the traditional cross-modal attention mechanism. In general, the invention is based on multimodal data of traffic scene images and text descriptions, and uses the designed traffic event recognition network model to achieve high-precision and high-efficiency traffic event recognition and positioning, while greatly reducing the computational complexity, so that the method proposed by the invention can be better used in the field of traffic monitoring and management, such as helping traffic management departments to monitor traffic conditions in real time, quickly respond to emergencies, optimize traffic flow, improve road safety, reduce the incidence of traffic accidents, promote the construction of intelligent transportation systems, improve the level of urban traffic management, and promote the development of smart cities.

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1It is a flow chart of a method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval provided by an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of the structure of a traffic event recognition network model provided by an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of the structure of a visual selective filtering module provided by an embodiment of the present invention;

[0030] Figure 4 is a structural diagram of a text selective filtering module provided by an embodiment of the present invention;

[0031] Figure 5 It is a structural diagram of a cross-modal selective alignment module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0033] See also Figure 1 The embodiment of the present invention provides a method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval, which specifically includes the following steps:

[0034] S10: Collect images of traffic events to be identified and their corresponding text descriptions.

[0035] S20, inputting the image of the traffic event to be identified and its corresponding text description into the trained traffic event recognition network model to obtain the recognition result of the traffic event to be identified; wherein the trained traffic event recognition network model includes a visual feature extraction module, a text feature extraction module, a visual selective filtering module, a text selective filtering module, a cross-modal selective alignment module and a calculation module; the visual feature extraction module is used to extract the visual features in the image of the traffic event to be identified and map the extracted visual features to a preset dimension to obtain visual extraction features; the text feature extraction module is used to extract the text features in the text description corresponding to the traffic event to be identified and map the extracted text features to a preset dimension to obtain text extraction features; the visual selective filtering module is used to extract the text features in the text description corresponding to the traffic event to be identified and map the extracted text features to a preset dimension to obtain text extraction features; A text selective filtering module is used to selectively filter irrelevant visual extraction features to obtain visual filtering features; a text selective filtering module is used to selectively filter unnecessary text extraction features to obtain text filtering features; a cross-modal selective alignment module is used to perform cross-modal alignment processing on visual extraction features and text extraction features to obtain visual alignment features and text alignment features; a calculation module is used to fuse the visual filtering features and the visual alignment features to obtain visual fusion features, and to fuse the text filtering features and the text alignment features to obtain text fusion features, calculate the similarity between the visual fusion features and the text fusion features, determine whether the similarity meets the similarity threshold, and use the image of the traffic event corresponding to the visual fusion feature that meets the similarity threshold as the recognition result of the traffic event to be recognized.

[0036] The traffic event recognition network model trained in S20 of the embodiment of the present invention is as follows: Figure 2 As shown. In the embodiment of the present invention, after the visual selective filtering module, it also includes a first average pooling layer; the first average pooling layer is used to perform average pooling processing on the visual filtering features, and input the visual filtering features after the average pooling processing into the calculation module; after the text selective filtering module in the embodiment of the present invention, it also includes a second average pooling layer; the second average pooling layer is used to perform average pooling processing on the text filtering features, and input the text filtering features after the average pooling processing into the calculation module; after the cross-modal selective alignment module in the embodiment of the present invention, it also includes a third average pooling layer and a fourth average pooling layer; the third average pooling layer is used to perform average pooling processing on the visual alignment features, and input the visual alignment features after the average pooling processing into the calculation module; the fourth average pooling layer is used to perform average pooling processing on the text alignment features, and input the text alignment features after the average pooling processing into the calculation module.

[0037] Furthermore, in the embodiment of the present invention, the visual feature extraction module is implemented using the Faster R-CNN network. The pre-trained Faster R-CNN network combined with the bottom-up attention mechanism is used to extract the visual features of m significant areas from the image of the traffic event to be identified as the visual extraction features: Among them, r i Represents the mean pooled convolutional features of the ith salient region extracted by the Faster R-CNN network, dimension d 1 According to the actual situation, for example, 1 It can be 2048. Next, the embodiment of the present invention further introduces a fully connected layer to adjust these visual extraction features so that they can better interact with the subsequently extracted text features across modalities:

[0038] v i =W r r i +b r ;

[0039] Among them, v i Represents r i Visual extraction features adjusted by the fully connected layer, W r 、b r They represent the weight and bias value of the fully connected layer introduced at this time. The visual extraction features adjusted by the fully connected layer are expressed as: At this time, it will be 1 The visual extraction feature of dimension is mapped to the visual extraction feature of dimension d, d>d 1 , for example d=1024.

[0040] Furthermore, in the embodiment of the present invention, the text feature extraction module is implemented using a BERT (Bidirectional Encoder Representations from Transformers) encoder. By using the BERT encoder as a text encoder. For a text description S containing n words, first add a [ClS] tag at the beginning of each sentence and a [SEP] tag at the end. This is the premise of using BERT as a text encoder. Then, by combining the word embedding, position embedding, and segment embedding of each word, and using the pre-trained BERT encoder, the embedding features of each word in the text description corresponding to the traffic event to be identified are obtained: n represents the number of words in the text description corresponding to the traffic event to be identified, ω j represents the embedding feature of the i-th word in the text description of the traffic event to be identified extracted by the BERT encoder, dimension d 2 According to the actual situation, for example,2 It can be 768. In order to better interact with visually extracted features across modalities, the embodiment of the present invention introduces a fully connected layer to map these embedded features to the same d dimension as the visually extracted features:

[0041] t j =W t ω j +b t ;

[0042] Among them, t j Represents ω j Text extraction features after mapping by the fully connected layer, W t , b t They represent the weight and bias value of the fully connected layer introduced at this time. The text extraction feature after mapping by the fully connected layer is expressed as:

[0043] Furthermore, since the visually extracted features represent a sequence lacking inherent sequence information, that is, the visually extracted features are non-sequential information, the embodiment of the present invention designs a visual selective filtering module for the visually extracted features, and uses the visual selective filtering module to selectively filter irrelevant visually extracted features to obtain visual filtering features. The specifically designed visual selective filtering module is as follows: Figure 3 As shown, it includes the first RMS (Root Mean Square) normalization layer, the first to fifth fully connected layers, the first convolutional layer, the second convolutional layer, the first to third activation layers, the first SSM (State Space Model, state space model) module and the second SSM module; wherein, the visual extraction feature is input into the first RMS normalization layer, the output end of the first RMS normalization layer is connected to the input end of the first fully connected layer and the second fully connected layer respectively, the output end of the first fully connected layer is connected to the first convolutional layer, the first activation layer, and the first SSM module in sequence, the output end of the second fully connected layer is connected to the input end of the second activation layer, the output end of the second activation layer is point multiplied with the output end of the first SSM module to obtain the first point multiplication result, the output data of the first RMS normalization layer is flipped and input into the third fully connected layer and the fourth fully connected layer respectively, the output end of the third fully connected layer is connected to the second convolutional layer, the third activation layer, and the second SSM module in sequence, the output end of the fourth fully connected layer is connected to the input end of the fourth activation layer, the output end of the fourth activation layer is point multiplied with the output end of the second SSM module to obtain the second point multiplication result, the second point multiplication result is flipped and summed with the first point multiplication result and the summed result is input into the fifth fully connected layer, and the output end of the fifth fully connected layer is summed with the visual extraction feature as the output of the visual selective filtering module. Specifically:

[0044] The state space model SSM is a linear time-invariant system model that models the dynamic behavior of the system through the relationship between the input signal and the hidden state, and can cope with time-varying and nonlinear systems. Therefore, it has strong versatility, especially in the fields of control theory, signal processing and linear systems. The embodiment of the present invention selects the SSM model because the core advantage of the SSM model lies in its modeling of temporal relationships and its application in dynamic systems. The embodiment of the present invention uses both the forward and reverse state space models SSM to construct a visual selective filtering module.

[0045] For visual feature extraction It is input into the state space model SSM from both the forward and reverse directions, thereby implicitly and selectively filtering irrelevant visual extraction information. Specifically:

[0046] For the forward direction, the visually extracted feature V is input into the state space model SSM, and the calculation formula is as follows:

[0047]

[0048] Among them, y forward It represents the output corresponding to the forward input of the visual extraction feature V to the state space model SSM, Norm 1 represents the first RMS normalization layer, W 1 Represents the first fully connected layer, Con 1 Represents the first convolutional layer, Silu 1 represents the first activation layer, SSM 1 Indicates the first SSM module, W 2 Represents the second fully connected layer, Silu 2 Represents the second activation layer. The first activation layer and the second activation layer both use SiLU activation function, the first convolution layer uses 1-D convolution operation, and the first SSM module uses the state space model.

[0049] For the backward direction, the visually extracted feature V is input into the state space model SSM, and the calculation formula is as follows:

[0050]

[0051] Among them, y backward It represents the output corresponding to the forward input of the visual feature V to the state space model SSM. 1 Indicates the first sequence flip, that is, the original input sequence is flipped into a reverse sequence, W 3 Represents the third fully connected layer, Con 2 Represents the second convolutional layer, Silu 3 Represents the third activation layer, SSM 2Indicates the second SSM module, W 4 Represents the fourth fully connected layer, Silu 4 Indicates the fourth activation layer, Flip 2 Indicates the second sequence flip, that is, flipping the current output to the form of the original input sequence.

[0052] After the above operations, we get the forward and reverse outputs: y forward With y backward Finally, the filtered visual feature, namely the visual filtering feature, is obtained by fusing it with the input visual extraction feature V through the following formula. The calculation formula is as follows:

[0053]

[0054] Among them, V intra represents the visual filtering feature, W 5 Represents the fifth fully connected layer.

[0055] Furthermore, for text extraction features, considering the inherent temporal nature of the text representation sequence, the embodiment of the present invention uses a unidirectional (forward) state space model to construct a text selective filtering module, and the constructed text selective filtering module is as follows: Figure 4 As shown, it includes a second RMS normalization layer, a sixth fully connected layer to an eighth fully connected layer, a third convolutional layer, a fifth activation layer, a sixth activation layer and a third SSM module; wherein the text extraction feature is input into the second RMS normalization layer, the output end of the second RMS normalization layer is respectively connected to the input ends of the sixth fully connected layer and the seventh fully connected layer, the output end of the sixth fully connected layer is sequentially connected to the third convolutional layer, the fifth activation layer and the third SSM module, the output end of the seventh fully connected layer is connected to the input end of the sixth activation layer, the output end of the sixth activation layer is dot-multiplied with the output end of the third SSM module and then input into the eighth fully connected layer, and the output end of the eighth fully connected layer is summed with the text extraction feature as the output of the text selective filtering module. Specifically:

[0056] The embodiment of the present invention extracts features from text Input into the unidirectional state space model SSM, thereby implicitly and selectively filtering unnecessary word embedding features (such as prepositions), and the calculation formula is as follows:

[0057]

[0058] Among them, y t It represents the output of the unidirectional state space model SSM corresponding to the text extraction feature T input, Norm 2 represents the second RMS normalization layer, W 6 Represents the sixth fully connected layer, Con 3 Represents the third convolutional layer, Silu5 Represents the fifth activation layer, SSM 3 Indicates the third SSM module, W 7 Represents the seventh fully connected layer, Silu 6 Indicates the sixth activation layer.

[0059] After the above operation, y t The filtered text feature, i.e., the text filtering feature, is obtained by fusing it with the input text extraction feature T through the following formula. The calculation formula is as follows:

[0060]

[0061] Among them, T intra represents the text filtering feature, W 8 Represents the eighth fully connected layer.

[0062] Furthermore, since the extracted visual extraction features V and text extraction features T are not aligned, the embodiment of the present invention proposes to construct a cross-modal selective alignment module to solve this problem, and the designed cross-modal selective alignment module is as follows: Figure 5 As shown, it includes a splicing layer, a third RMS normalization layer, a ninth fully connected layer to an eleventh fully connected layer, a fourth convolutional layer, a seventh activation layer, an eighth activation layer, a fourth SSM module and a splitting layer; wherein the visual extraction features and the text extraction features are input into the splicing layer, the output end of the splicing layer is connected to the input end of the third RMS normalization layer, the output end of the third RMS normalization layer is respectively connected to the input ends of the ninth fully connected layer and the tenth fully connected layer, the output end of the ninth fully connected layer is connected to the fourth convolutional layer, the seventh activation layer, and the fourth SSM module in sequence, the output end of the tenth fully connected layer is connected to the input end of the eighth activation layer, the output end of the eighth activation layer is point-multiplied with the output end of the fourth SSM module and then input into the eleventh fully connected layer, the output end of the eleventh fully connected layer is summed with the output end of the splicing layer and then input into the splitting layer, and the output data of the splitting layer is used as the output of the cross-modal selective alignment module. Specifically:

[0063] In the embodiment of the present invention, the visual extraction feature V and the text extraction feature T are input into the cross-modal selective alignment module to perform efficient selective cross-modal alignment. The calculation formula is as follows:

[0064] S = Cat(V, T);

[0065]

[0066]

[0067] Among them, V cross represents the visual alignment feature, T crossrepresents the text alignment feature, Cat represents the concatenation layer, which is used to concatenate the visual extraction feature V and the text extraction feature T in the sequence length dimension, and Norm 3 represents the third RMS normalization layer, W 9 Represents the ninth fully connected layer, Con 4 Represents the fourth convolutional layer, Silu 7 Indicates the seventh activation layer, SSM 4 Indicates the fourth SSM module, W 10 Represents the tenth fully connected layer, Silu 8 represents the eighth activation layer, W 11 represents the eleventh fully connected layer, and Split represents the splitting layer, which is used to perform splitting operations on the sequence length dimension according to the original sequence length.

[0068] Furthermore, in the calculation module, the output V of the visual selective filtering module is intra , T output by the text selective filtering module intra V output by the cross-modal selective alignment module cross , T cross Perform average pooling operations respectively to obtain the corresponding global embedding features: i intra ,t intra 、i cross ,t cross Finally, a fusion operation is used to combine these global embedding features from intra-modal and cross-modal interactions, and they are passed through a fully connected layer to obtain the final visual fusion feature and text fusion feature. The calculation formula is as follows:

[0069]

[0070] Among them, V' represents visual fusion features, T' represents text fusion features, and W V represents the fully connected layer after the fusion of the global embedding features of vision, W T The fully connected layer after the fusion of the global embedding features of the text. Furthermore, the similarity between the visual fusion features and the text fusion features is calculated by using a similarity function used to measure the similarity between the image features and the text features, and it is determined whether the calculated similarity meets the similarity threshold. The similarity threshold is set according to the actual situation. Finally, the image of the traffic event corresponding to the visual fusion feature that meets the similarity threshold is used as the recognition result of the traffic event to be identified, thereby realizing the cross-modal retrieval of the traffic event to be identified. For multiple images of traffic events that meet the similarity threshold, the image of the traffic event with the highest similarity is selected as the recognition result of the traffic event to be identified.

[0071] Furthermore, the embodiment of the present invention adopts the existing image-text matching training algorithm for training, such as adopting the idea of ​​contrastive learning for training. In the training process of the traffic event recognition network model, the loss function adopted by the embodiment of the present invention is the two-way triple contrast loss, which is expressed as follows:

[0072]

[0073] Where L represents the two-way triplet contrast loss, m represents the margin hyperparameter, [] + represents the max(0,·) operation, which means selecting the maximum value between 0 and another data. represents an image of any traffic event in the training sample set, Indicates the training sample set The corresponding text description, and Constitute a positive sample pair, Indicates that the training sample set Any image of a traffic incident outside Indicates that the training sample set The text description corresponding to any traffic event other than and and They constitute negative sample pairs respectively, and sim() represents the similarity function used to measure the similarity between image features and text features. The bidirectional triplet contrast loss adopted encourages matching image-text positive samples to contrast mismatched image-text negative sample pairs, thereby obtaining a higher similarity score.

[0074] It should be noted here that before recognition and training, the traffic event data needs to be cleaned, and image-text sample pair data needs to be constructed. The image-text sample pairs need to be paired with positive samples and negative samples to ensure the consistency of the traffic event images and their corresponding text descriptions, providing a high-quality data foundation for the recognition and training process.

[0075] It should be noted that the embodiment of the present invention can also add a verification process based on a verification data set. The verification process refers to the training process. Finally, the recognition results based on the training data set and the verification data set can be stored in a database. The data stored in the database is used to retrieve images of traffic events to be identified. In actual application, the image of the traffic event to be identified can be identified only based on the text description corresponding to the traffic event to be identified. The specific recognition process is: calculating the similarity between the text description corresponding to the traffic event to be identified and the images of all traffic events in the database, judging whether the similarity meets the similarity threshold, and taking the image of the traffic event that meets the similarity threshold as the recognition result of the traffic event to be identified. For multiple images of traffic events that meet the similarity threshold, select the image of the traffic event with the highest similarity as the recognition result of the traffic event to be identified.

[0076] In order to verify the effectiveness of the intelligent traffic event identification method of the large traffic model based on cross-modal retrieval provided by the embodiment of the present invention, the following experiments are conducted for verification.

[0077] 1. Experimental simulation parameters

[0078] The embodiment of the present invention is experimented on a Linux operating system with a central processor of Intel(R) Xeon(R) Gold 6248 CPU @ 2.50 GHz and a graphics processor of NVIDIA Tesla V100 GPU using the deep learning framework Pytorch-GPU 2.0.0. The graphic data of traffic events uses the traffic scene graphic data in the public data set Flickr30K and MS-COCO.

[0079] The methods compared in the experiment include: a multimodal cross attention network based on the self-attention mechanism, denoted as the MMCA method in the experiment, and the reference is "X. Wei, T. Zhang, Y. Li, Y. Zhang, and F. Wu, "Multi-modality cross attention network for image and sentence matching," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 10941-10950."; a method based on similarity graph reasoning and attention filtering, denoted as the SGRAF method in the experiment, and the reference is "H. Diao, Y. Zhang, L. Ma, and H. Lu, "Similarity reasoning and filtration for image-text matching," in Proceedings of the AAAI conference on artificial intelligence, vol.35, no.2, 2021, pp.1218-1226."; Hierarchical alignment network based on the classic Transformer, marked as HAT method in the experiment, the reference is "Y.Bin, H.Li, Y.Xu, X.Xu, Y.Yang, and HTShen, "Unifying two-stream encoders with transformers for cross-modal retrieval," in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp.3041-3050.".

[0080] In the experiment, the Recall@K index is used as the retrieval evaluation index to measure the performance of image-to-text retrieval (I2T) and text-to-image retrieval (T2I) tasks, referred to as R@K (K = 1, 5, 10). It measures the proportion of correct retrieval of the top K names in the retrieval results. In addition, Rsum is also used to evaluate the performance of the model, which is defined as follows:

[0081] Rsum=R@1 i2t +R@5 i2t +R@10 i2t +R@1 t2i +R@5t2i +R@10 t2i .

[0082] 2. Experimental simulation content

[0083] The retrieval evaluation indicators of the traffic scene image and text test datasets in the public datasets Flickr30K and MS-COCO 1K are calculated respectively, and compared with the retrieval evaluation indicators of the MMCA method, SGRAF method and HAT method. The results are shown in Tables 1 and 2.

[0084] Table 1 Retrieval evaluation indicators of Flickr30K dataset

[0085]

[0086] Table 2. MS-COCO 1K dataset retrieval evaluation indicators

[0087]

[0088] As can be seen from Tables 1 and 2, since the visual selective filtering module and the text selective filtering module adopted by the present invention can implicitly and selectively filter irrelevant visual information and unnecessary word embedding features (such as prepositions), and the cross-modal selective alignment module can selectively align key fine-grained features across modalities, extremely significant retrieval effects have been achieved on the public datasets Flickr30K and MS-COCO 1K, verifying the advanced nature of the method proposed in the present invention.

[0089] In summary, the intelligent traffic event recognition method based on cross-modal retrieval of a large traffic model proposed in the embodiment of the present invention introduces a visual selective filtering module, a text selective filtering module and a cross-modal selective alignment module into the designed traffic event recognition network model. By implicitly and selectively matching the image and text of the traffic event, the model can effectively filter out irrelevant information, strengthen the perception and alignment of key features, thereby improving the accuracy and speed of cross-modal retrieval, and greatly improving the ability to accurately identify and locate traffic events; at the same time, the representation learned by the model is optimized and updated, solving the problem of invalid semantic alignment and high computational complexity caused by global allocation in the traditional cross-modal attention mechanism. In general, the embodiment of the present invention is based on the multimodal data of traffic scene images and text descriptions, and uses the designed traffic event recognition network model to achieve high-precision and high-efficiency traffic event recognition and positioning, while greatly reducing the computational complexity, so that the method proposed in the present invention can be better used in the field of traffic monitoring and management, such as helping traffic management departments to monitor traffic conditions in real time, quickly respond to emergencies, optimize traffic flow, improve road safety, reduce the incidence of traffic accidents, promote the construction of intelligent transportation systems, improve the level of urban traffic management, and promote the development of smart cities.

[0090] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0091] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the specification and its drawings. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0092] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval, characterized in that: The method comprises: Collect images of traffic events to be identified and their corresponding text descriptions; The image of the traffic event to be identified and its corresponding text description are input into the trained traffic event identification network model to obtain the identification result of the traffic event to be identified; wherein, The trained traffic event recognition network model includes a visual feature extraction module, a text feature extraction module, a visual selective filtering module, a text selective filtering module, a cross-modal selective alignment module and a calculation module; the visual feature extraction module is used to extract visual features in the image of the traffic event to be identified and map the extracted visual features to a preset dimension to obtain visual extraction features; the text feature extraction module is used to extract text features in the text description corresponding to the traffic event to be identified and map the extracted text features to the preset dimension to obtain text extraction features; the visual selective filtering module is used to selectively filter irrelevant visual extraction features to obtain visual filtering features; the text selective filtering module, Used to selectively filter unnecessary text extraction features to obtain text filtering features; the cross-modal selective alignment module is used to perform cross-modal alignment processing on the visual extraction features and the text extraction features to obtain visual alignment features and text alignment features; the calculation module is used to fuse the visual filtering features and the visual alignment features to obtain visual fusion features, and fuse the text filtering features and the text alignment features to obtain text fusion features, calculate the similarity between the visual fusion features and the text fusion features, determine whether the similarity meets the similarity threshold, and use the image of the traffic event corresponding to the visual fusion feature that meets the similarity threshold as the recognition result of the traffic event to be recognized.

2. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: The visual feature extraction module is implemented using the Faster R-CNN network.

3. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: The text feature extraction module is implemented using the BERT encoder.

4. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: The visual selective filtering module includes a first RMS normalization layer, a first fully connected layer to a fifth fully connected layer, a first convolutional layer, a second convolutional layer, a first activation layer to a third activation layer, a first SSM module and a second SSM module; wherein, The visual extraction feature is input into the first RMS normalization layer, the output end of the first RMS normalization layer is connected to the input ends of the first fully connected layer and the second fully connected layer respectively, the output end of the first fully connected layer is connected to the first convolutional layer, the first activation layer, and the first SSM module in sequence, the output end of the second fully connected layer is connected to the input end of the second activation layer, the output end of the second activation layer is point multiplied with the output end of the first SSM module to obtain a first point multiplication result, the output data of the first RMS normalization layer is flipped and respectively input into the third fully connected layer and the fourth fully connected layer, the output end of the third fully connected layer is connected to the second convolutional layer, the third activation layer, and the second SSM module in sequence, the output end of the fourth fully connected layer is connected to the input end of the fourth activation layer, the output end of the fourth activation layer is point multiplied with the output end of the second SSM module to obtain a second point multiplication result, the second point multiplication result is flipped and summed with the first point multiplication result and the summed result is input into the fifth fully connected layer, the output end of the fifth fully connected layer is summed with the visual extraction feature as the output of the visual selective filtering module.

5. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: After the visual selective filtering module, a first average pooling layer is also included; the first average pooling layer is used to perform average pooling processing on the visual filtering features, and input the visual filtering features after the average pooling processing into the calculation module.

6. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: The text selective filtering module includes a second RMS normalization layer, a sixth fully connected layer to an eighth fully connected layer, a third convolutional layer, a fifth activation layer, a sixth activation layer and a third SSM module; wherein, The text extraction feature is input into the second RMS normalization layer, the output end of the second RMS normalization layer is respectively connected to the input ends of the sixth fully connected layer and the seventh fully connected layer, the output end of the sixth fully connected layer is connected to the third convolutional layer, the fifth activation layer, and the third SSM module in sequence, the output end of the seventh fully connected layer is connected to the input end of the sixth activation layer, the output end of the sixth activation layer is dot-multiplied with the output end of the third SSM module and then input into the eighth fully connected layer, and the output end of the eighth fully connected layer and the text extraction feature are summed as the output of the text selective filtering module.

7. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: A second average pooling layer is also included after the text selective filtering module; the second average pooling layer is used to perform average pooling processing on the text filtering features, and input the text filtering features after the average pooling processing into the calculation module.

8. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: The cross-modal selective alignment module includes a concatenation layer, a third RMS normalization layer, a ninth fully connected layer to an eleventh fully connected layer, a fourth convolutional layer, a seventh activation layer, an eighth activation layer, a fourth SSM module and a splitting layer; wherein, The visual extraction features and the text extraction features are input into the splicing layer, the output end of the splicing layer is connected to the input end of the third RMS normalization layer, the output end of the third RMS normalization layer is respectively connected to the input ends of the ninth fully connected layer and the tenth fully connected layer, the output end of the ninth fully connected layer is connected to the fourth convolutional layer, the seventh activation layer, and the fourth SSM module in sequence, the output end of the tenth fully connected layer is connected to the input end of the eighth activation layer, the output end of the eighth activation layer is dot-multiplied with the output end of the fourth SSM module and then input into the eleventh fully connected layer, the output end of the eleventh fully connected layer is summed with the output end of the splicing layer and then input into the splitting layer, and the output data of the splitting layer is used as the output of the cross-modal selective alignment module.

9. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: After the cross-modal selective alignment module, a third average pooling layer and a fourth average pooling layer are also included; the third average pooling layer is used to perform average pooling processing on the visual alignment features, and input the visual alignment features after the average pooling processing into the calculation module; the fourth average pooling layer is used to perform average pooling processing on the text alignment features, and input the text alignment features after the average pooling processing into the calculation module.

10. The method for intelligent traffic event identification based on a large traffic model of cross-modal retrieval according to claim 1 is characterized in that: In the training process of the traffic event recognition network model, the loss function used is the two-way triplet contrast loss, which is expressed as follows: Where L represents the two-way triplet contrast loss, m represents the margin hyperparameter, [] + represents the max(0,·) operation, which means selecting the maximum value between 0 and another data. represents an image of any traffic event in the training sample set, Indicates the training sample set The corresponding text description, and Constitute a positive sample pair, Indicates that the training sample set Any image of a traffic incident outside Indicates that the training sample set The text description corresponding to any traffic event other than and and They constitute negative sample pairs respectively, and sim() represents the similarity function used to measure the similarity between image features and text features.

Images