An intelligent traffic event recognition method based on large traffic model and cross-modal retrieval

Abstract

The present invention discloses a method for intelligent traffic event recognition based on a large traffic model using cross-modal retrieval. The method comprises the following steps: inputting an image of a traffic event to be recognized and its text description into a traffic event recognition network model; the traffic event recognition network model comprises: a visual feature extraction module for extracting visual features from the image; a text feature extraction module for extracting text features from the text description; a visual selective filtering module for selectively filtering irrelevant visual extraction features to obtain visual filtering features; a text selective filtering module for selectively filtering unnecessary text extraction features to obtain text filtering features; a cross-modal selective alignment module for cross-modally aligning visual and text extraction features to obtain visual alignment features and text alignment features; and a calculation module for fusing the visual filtering features and visual alignment features, as well as the text filtering features and text alignment features, to determine a recognition result. The present invention improves the accuracy and efficiency of retrieval and recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of large traffic models, and in particular relates to an intelligent traffic event recognition method for large traffic models based on cross-modal retrieval. Background Art

[0002] As urban transportation systems become increasingly complex, the frequency of traffic incidents poses significant challenges to traffic management and monitoring. However, accurate traffic incident recognition can effectively address these challenges and provide valuable reference information for traffic management departments. First, accurate traffic incident recognition can help traffic management departments better coordinate traffic resources and emergency response measures. By cross-modal retrieval, recognition, and localization of multimodal data, such as traffic scene images and text descriptions, management departments can understand road conditions in real time, rationally deploy personnel and rescue equipment, and flexibly adjust traffic control strategies. This not only ensures road safety and smooth traffic flow, but also maximizes the utilization of traffic resources. Second, through intelligent traffic incident recognition, traffic accidents, congestion, and violations can be promptly detected, allowing for swift action to reduce traffic delays and accidents. This refined management approach helps optimize traffic flow and enhance the public travel experience. Efficient traffic incident recognition reduces reliance on human monitoring, saving costs in rationally planning transportation infrastructure and optimizing resource allocation.

[0003] Most existing cross-modal retrieval technologies are based on deep learning and are mainly divided into coarse-grained matching and fine-grained matching. Among them, the fine-grained matching method can more accurately capture the subtle correspondence between images and texts. For example: a fine-grained matching method based on the attention mechanism, whose basic idea is: introduce the attention mechanism, dynamically pay attention to the key features in the image and text, and assign weights according to their importance to improve the accuracy of matching. In this way, the model can more effectively align the detailed information between the image and text; a fine-grained matching method based on a graph neural network, whose basic idea is: construct the feature representation of the image and text into a graph structure, and use the graph neural network to capture the complex relationship and structural information between multimodal data; by performing information propagation and feature aggregation on the graph, the model can more deeply understand and associate data of different modalities, thereby improving the performance of cross-modal retrieval. These methods perform well in handling the complexity and diversity of traffic events, and help improve the accuracy and efficiency of traffic event identification and positioning.

[0004] However, the above existing methods have shortcomings: in the fine-grained matching method based on the attention mechanism, not every image contains a valid area, and not every word in a sentence necessarily contributes to semantic alignment. Meaningless image areas and words (such as prepositions) may lead to redundant semantic associations and alignments, reducing the consistency of cross-modal representation alignment; in the fine-grained matching method based on graph neural networks, although the graph learning-based network can capture the complex relationship between modalities, it consumes a lot of computing resources and has low training efficiency when processing large-scale data. Summary of the Invention

[0005] To address the above-mentioned problems in the prior art, the present invention provides a method for intelligent traffic event recognition based on a large traffic model using cross-modal retrieval. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] An embodiment of the present invention provides a method for intelligent traffic event recognition based on a large traffic model using cross-modal retrieval, the method comprising:

[0007] Collect images of traffic events to be identified and their corresponding text descriptions;

[0008] The image of the traffic event to be identified and its corresponding text description are input into the trained traffic event recognition network model to obtain the recognition result of the traffic event to be identified; wherein,

[0009] The trained traffic event recognition network model includes a visual feature extraction module, a text feature extraction module, a visual selective filtering module, a text selective filtering module, a cross-modal selective alignment module and a calculation module; the visual feature extraction module is used to extract visual features in the image of the traffic event to be identified and map the extracted visual features to a preset dimension to obtain visual extraction features; the text feature extraction module is used to extract text features in the text description corresponding to the traffic event to be identified and map the extracted text features to the preset dimension to obtain text extraction features; the visual selective filtering module is used to selectively filter irrelevant visual extraction features to obtain visual filtering features; the text selective filtering module, Used to selectively filter unnecessary text extraction features to obtain text filtering features; the cross-modal selective alignment module is used to perform cross-modal alignment processing on the visual extraction features and the text extraction features to obtain visual alignment features and text alignment features; the calculation module is used to fuse the visual filtering features and the visual alignment features to obtain visual fusion features, and fuse the text filtering features and the text alignment features to obtain text fusion features, calculate the similarity between the visual fusion features and the text fusion features, determine whether the similarity meets the similarity threshold, and use the image of the traffic event corresponding to the visual fusion feature that meets the similarity threshold as the recognition result of the traffic event to be recognized.

[0010] In one embodiment of the present invention, the visual feature extraction module is implemented using a Faster R-CNN network.

[0011] In one embodiment of the present invention, the text feature extraction module is implemented using a BERT encoder.

[0012] In one embodiment of the present invention, the visual selective filtering module includes a first RMS normalization layer, a first fully connected layer to a fifth fully connected layer, a first convolutional layer, a second convolutional layer, a first activation layer to a third activation layer, a first SSM module and a second SSM module; wherein,

[0013] The visual extraction feature is input into the first RMS normalization layer, and the output end of the first RMS normalization layer is connected to the input ends of the first fully connected layer and the second fully connected layer respectively. The output end of the first fully connected layer is connected to the first convolutional layer, the first activation layer, and the first SSM module in sequence. The output end of the second fully connected layer is connected to the input end of the second activation layer. The output end of the second activation layer is dot-multiplied with the output end of the first SSM module to obtain a first dot product result. The output data of the first RMS normalization layer is flipped and input into the third fully connected layer and the fourth fully connected layer respectively. The output end of the third fully connected layer is connected to the second convolutional layer, the third activation layer, and the second SSM module in sequence. The output end of the fourth fully connected layer is connected to the input end of the fourth activation layer. The output end of the fourth activation layer is dot-multiplied with the output end of the second SSM module to obtain a second dot product result. The second dot product result is flipped and summed with the first dot product result, and the sum result is input into the fifth fully connected layer. The output end of the fifth fully connected layer is summed with the visual extraction feature as the output of the visual selective filtering module.

[0014] In one embodiment of the present invention, the visual selective filtering module further includes a first average pooling layer; the first average pooling layer is used to perform average pooling processing on the visual filtering features and input the visual filtering features after average pooling processing into the calculation module.

[0015] In one embodiment of the present invention, the text selective filtering module includes a second RMS normalization layer, a sixth fully connected layer to an eighth fully connected layer, a third convolutional layer, a fifth activation layer, a sixth activation layer and a third SSM module; wherein,

[0016] The text extraction feature is input into the second RMS normalization layer, the output end of the second RMS normalization layer is connected to the input ends of the sixth fully connected layer and the seventh fully connected layer respectively, the output end of the sixth fully connected layer is connected to the third convolutional layer, the fifth activation layer, and the third SSM module in sequence, the output end of the seventh fully connected layer is connected to the input end of the sixth activation layer, the output end of the sixth activation layer is dot-multiplied with the output end of the third SSM module and then input into the eighth fully connected layer, and the output end of the eighth fully connected layer and the text extraction feature are summed as the output of the text selective filtering module.

[0017] In one embodiment of the present invention, a second average pooling layer is further included after the text selective filtering module; the second average pooling layer is used to perform average pooling processing on the text filtering features and input the text filtering features after average pooling processing into the calculation module.

[0018] In one embodiment of the present invention, the cross-modal selective alignment module includes a splicing layer, a third RMS normalization layer, a ninth fully connected layer to an eleventh fully connected layer, a fourth convolutional layer, a seventh activation layer, an eighth activation layer, a fourth SSM module and a splitting layer; wherein,

[0019] The visual extraction features and the text extraction features are input into the splicing layer, the output end of the splicing layer is connected to the input end of the third RMS normalization layer, the output end of the third RMS normalization layer is connected to the input ends of the ninth fully connected layer and the tenth fully connected layer respectively, the output end of the ninth fully connected layer is connected to the fourth convolutional layer, the seventh activation layer, and the fourth SSM module in sequence, the output end of the tenth fully connected layer is connected to the input end of the eighth activation layer, the output end of the eighth activation layer is dot-multiplied with the output end of the fourth SSM module and then input into the eleventh fully connected layer, the output end of the eleventh fully connected layer and the output end of the splicing layer are summed and input into the splitting layer, and the output data of the splitting layer is used as the output of the cross-modal selective alignment module.

[0020] In one embodiment of the present invention, the cross-modal selective alignment module further includes a third average pooling layer and a fourth average pooling layer; the third average pooling layer is used to perform average pooling processing on the visual alignment features, and input the visual alignment features after the average pooling processing into the calculation module; the fourth average pooling layer is used to perform average pooling processing on the text alignment features, and input the text alignment features after the average pooling processing into the calculation module.

[0021] In one embodiment of the present invention, during the training of the traffic event recognition network model, the loss function used is a two-way triplet contrast loss, which is expressed as follows:

[0022] ;

[0023] in, represents the bidirectional triplet contrast loss, represents the margin hyperparameter, express operate, It means to select the maximum value between 0 and another data. represents an image of any traffic event in the training sample set, Indicates that the training sample set The corresponding text description, and Constitute a positive sample pair, Indicates that the training sample set Any image of a traffic incident, Indicates that the training sample set The text description of any traffic event other than and 、 and They constitute negative sample pairs respectively, Represents a similarity function used to measure the similarity between image features and text features.

[0024] Beneficial effects of the present invention:

[0025] The present invention proposes an intelligent traffic event recognition method for large-scale traffic models based on cross-modal retrieval. The designed traffic event recognition network model introduces a visual selective filtering module, a text selective filtering module, and a cross-modal selective alignment module. By performing implicit and selective fine-grained matching between traffic event images and text, the model can effectively filter out irrelevant information, strengthen the perception and alignment of key features, thereby improving the accuracy and speed of cross-modal retrieval, and significantly enhancing the ability to accurately identify and locate traffic events. At the same time, the model optimizes and updates the representation learned by the model, solving the problems of ineffective semantic alignment and high computational complexity caused by global allocation in traditional cross-modal attention mechanisms. In summary, the present invention uses a designed traffic event recognition network model based on multimodal data of traffic scene images and text descriptions to achieve high-precision and high-efficiency traffic event recognition and location, while significantly reducing computational complexity. This makes the proposed method more suitable for the field of traffic monitoring and management, such as helping traffic management departments monitor traffic conditions in real time, quickly respond to emergencies, optimize traffic flow, improve road safety, reduce traffic accident rates, promote the construction of intelligent transportation systems, enhance urban traffic management, and promote the development of smart cities.

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval provided by an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of the structure of the traffic incident recognition network model provided by an embodiment of the present invention;

[0029] Figure 3 is a schematic structural diagram of a visual selective filtering module provided by an embodiment of the present invention;

[0030] Figure 4 is a structural diagram of a text selective filtering module provided by an embodiment of the present invention;

[0031] Figure 5 3 is a schematic diagram of the structure of the cross-modal selective alignment module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0033] See Figure 1 The embodiment of the present invention provides a method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval, which specifically includes the following steps:

[0034] S10: Collect images of traffic events to be identified and their corresponding text descriptions.

[0035] S20, input the image of the traffic event to be identified and its corresponding text description into the trained traffic event recognition network model to obtain the recognition result of the traffic event to be identified; wherein the trained traffic event recognition network model includes a visual feature extraction module, a text feature extraction module, a visual selective filtering module, a text selective filtering module, a cross-modal selective alignment module and a calculation module; the visual feature extraction module is used to extract the visual features in the image of the traffic event to be identified and map the extracted visual features to a preset dimension to obtain visual extraction features; the text feature extraction module is used to extract the text features in the text description corresponding to the traffic event to be identified and map the extracted text features to a preset dimension to obtain text extraction features; the visual selective filtering module is used to extract the text features in the text description corresponding to the traffic event to be identified and map the extracted text features to a preset dimension to obtain text extraction features; A text selective filtering module is used to selectively filter irrelevant visual extraction features to obtain visual filtering features; a text selective filtering module is used to selectively filter unnecessary text extraction features to obtain text filtering features; a cross-modal selective alignment module is used to perform cross-modal alignment processing on visual extraction features and text extraction features to obtain visual alignment features and text alignment features; a calculation module is used to fuse the visual filtering features and the visual alignment features to obtain visual fusion features, and to fuse the text filtering features and the text alignment features to obtain text fusion features, calculate the similarity between the visual fusion features and the text fusion features, determine whether the similarity meets the similarity threshold, and use the image of the traffic event corresponding to the visual fusion feature that meets the similarity threshold as the recognition result of the traffic event to be recognized.

[0036] The traffic event recognition network model trained in S20 of the embodiment of the present invention is as follows: Figure 2 As shown. In the embodiment of the present invention, after the visual selective filtering module, it also includes a first average pooling layer; the first average pooling layer is used to perform average pooling processing on the visual filtering features, and input the visual filtering features after the average pooling processing into the calculation module; after the text selective filtering module in the embodiment of the present invention, it also includes a second average pooling layer; the second average pooling layer is used to perform average pooling processing on the text filtering features, and input the text filtering features after the average pooling processing into the calculation module; after the cross-modal selective alignment module in the embodiment of the present invention, it also includes a third average pooling layer and a fourth average pooling layer; the third average pooling layer is used to perform average pooling processing on the visual alignment features, and input the visual alignment features after the average pooling processing into the calculation module; the fourth average pooling layer is used to perform average pooling processing on the text alignment features, and input the text alignment features after the average pooling processing into the calculation module.

[0037] Furthermore, the visual feature extraction module in the embodiment of the present invention is implemented using the Faster R-CNN network. By using the pre-trained Faster R-CNN network combined with the bottom-up attention mechanism, the image of the traffic event to be identified is extracted. The visual features of the salient areas are used as visual extraction features: ,in, Represents the first Mean pooling convolution features of significant regions, dimension According to the actual situation, such as It can be 2048. Next, the embodiment of the present invention also introduces a fully connected layer to adjust these visually extracted features so that they can better interact with the subsequently extracted text features across modalities:

[0038] ;

[0039] in, express Visual extraction features adjusted by the fully connected layer, 、 Respectively represent the weights and bias values ​​of the fully connected layer introduced at this time. The visual extraction features adjusted by the fully connected layer are expressed as: , which will be The visual extraction feature map of dimension is Visual extraction features, > ,for example =1024.

[0040] Furthermore, in the embodiment of the present invention, the text feature extraction module is implemented using a BERT (Bidirectional Encoder Representations from Transformers) encoder. By using the BERT encoder as the text encoder, Text description of words , first add at the beginning of each sentence Mark, and add at the end This is the premise for using BERT as a text encoder. Next, by combining the word embedding, position embedding, and segment embedding of each word and using the pre-trained BERT encoder, we can obtain the embedding features of each word in the text description of the traffic event to be identified: , Indicates the number of words in the text description corresponding to the traffic event to be identified, Indicates the first character in the text description of the traffic event to be identified extracted by the BERT encoder. word embedding features, dimension According to the actual situation, such as It can be 768. In order to better interact with the visual extraction features across modalities, the embodiment of the present invention introduces a fully connected layer to map these embedded features to the same Dimensions:

[0041] ;

[0042] in, express Text extraction features after mapping by the fully connected layer, 、 They represent the weight and bias value of the fully connected layer introduced at this time. The text extraction feature after mapping by the fully connected layer is represented as: .

[0043] Furthermore, since the visual extraction feature representation sequence lacks inherent sequence information, that is, the visual extraction feature is non-sequence information, the embodiment of the present invention designs a visual selective filtering module for the visual extraction feature, and uses the visual selective filtering module to selectively filter the irrelevant visual extraction features to obtain the visual filtering features. The specifically designed visual selective filtering module is as follows: Figure 3 As shown, it includes the first RMS (Root Mean Square) normalization layer, the first to fifth fully connected layers, the first convolutional layer, the second convolutional layer, the first to third activation layers, the first SSM (State Space Model, state space model) module and the second SSM module; wherein, the visual extraction feature is input into the first RMS normalization layer, the output end of the first RMS normalization layer is connected to the input end of the first fully connected layer and the second fully connected layer respectively, the output end of the first fully connected layer is connected to the first convolutional layer, the first activation layer, and the first SSM module in sequence, the output end of the second fully connected layer is connected to the input end of the second activation layer, the output end of the second activation layer is multiplied by the output end of the first SSM module to obtain the first dot product result, the output data of the first RMS normalization layer is flipped and input into the third fully connected layer and the fourth fully connected layer respectively, the output end of the third fully connected layer is connected to the second convolutional layer, the third activation layer, and the second SSM module in sequence, the output end of the fourth fully connected layer is connected to the input end of the fourth activation layer, the output end of the fourth activation layer is multiplied by the output end of the second SSM module to obtain the second dot product result, the second dot product result is flipped and summed with the first dot product result and the summed result is input into the fifth fully connected layer, and the output end of the fifth fully connected layer is summed with the visual extraction feature as the output of the visual selective filtering module. Specifically:

[0044] The state-space model (SSM) is a linear, time-invariant system model that models the dynamic behavior of a system through the relationship between input signals and hidden states. It can handle time-varying and nonlinear systems, making it highly versatile and particularly common in control theory, signal processing, and linear systems. The SSM model was chosen in this embodiment of the present invention because its core advantages lie in its modeling of temporal relationships and its application in dynamic systems. This embodiment of the present invention utilizes both forward and backward state-space models (SSMs) to construct a visual selective filtering module.

[0045] For visual feature extraction , which is input into the state space model SSM from both the forward and reverse directions, thereby implicitly and selectively filtering out irrelevant visual extraction information. Specifically:

[0046] For the forward direction, the visual features are extracted Input into the state space model SSM, the calculation formula is as follows:

[0047] ;

[0048] in, Represents visual extraction features The forward input is to the output corresponding to the state space model SSM, represents the first RMS normalization layer, represents the first fully connected layer, represents the first convolutional layer, represents the first activation layer, Indicates the first SSM module. represents the second fully connected layer, Represents the second activation layer. The first activation layer and the second activation layer are both Activation function, the first convolution layer uses 1-D convolution operation, and the first SSM module uses the state space model.

[0049] For the backward direction, the visual features are extracted Input into the state space model SSM, the calculation formula is as follows:

[0050] ;

[0051] in, Represents visual extraction features The forward input is to the output corresponding to the state space model SSM, Indicates the first sequence reversal, that is, the original input sequence is reversed to the inverted sequence. represents the third fully connected layer, represents the second convolutional layer, represents the third activation layer, Indicates the second SSM module. represents the fourth fully connected layer, represents the fourth activation layer, Indicates the second sequence flip, that is, flipping the current output to the form of the original input sequence.

[0052] After the above operations, we get the forward and reverse outputs respectively: and , and finally it is combined with the input visual extraction features through the following formula The filtered visual features obtained by fusion are visual filtering features, and the calculation formula is as follows:

[0053] ;

[0054] in, represents the visual filtering feature, Represents the fifth fully connected layer.

[0055] Furthermore, for text extraction features, considering the temporal nature of the text representation sequence, the embodiment of the present invention uses a unidirectional (forward) state space model to construct a text selective filtering module, and the constructed text selective filtering module is as follows: Figure 4 As shown, it includes the second RMS normalization layer, the sixth to eighth fully connected layers, the third convolutional layer, the fifth activation layer, the sixth activation layer and the third SSM module; wherein, the text extraction feature is input into the second RMS normalization layer, the output end of the second RMS normalization layer is connected to the input ends of the sixth and seventh fully connected layers respectively, the output end of the sixth fully connected layer is connected to the third convolutional layer, the fifth activation layer and the third SSM module in sequence, the output end of the seventh fully connected layer is connected to the input end of the sixth activation layer, the output end of the sixth activation layer is dot-multiplied with the output end of the third SSM module and then input into the eighth fully connected layer, and the output end of the eighth fully connected layer is summed with the text extraction feature as the output of the text selective filtering module. Specifically:

[0056] The embodiment of the present invention extracts features from text Input into the unidirectional state space model SSM, thereby implicitly and selectively filtering unnecessary word embedding features (such as prepositions). The calculation formula is as follows:

[0057] ;

[0058] in, Represents text extraction features The output corresponding to the input one-way state space model SSM, represents the second RMS normalization layer, represents the sixth fully connected layer, represents the third convolutional layer, represents the fifth activation layer, Indicates the third SSM module. represents the seventh fully connected layer, Indicates the sixth activation layer.

[0059] After the above operations, The following formula is used to extract features from the input text The filtered text features obtained by fusion are the text filtering features, and the calculation formula is as follows:

[0060] ;

[0061] in, Represents text filtering features, Represents the eighth fully connected layer.

[0062] Furthermore, due to the extracted visual features and text extraction features There is a problem of misalignment. The embodiment of the present invention proposes to construct a cross-modal selective alignment module to solve this problem, and the designed cross-modal selective alignment module is as follows: Figure 5 As shown, it includes a splicing layer, a third RMS normalization layer, a ninth fully connected layer to an eleventh fully connected layer, a fourth convolutional layer, a seventh activation layer, an eighth activation layer, a fourth SSM module, and a splitting layer; wherein, the visual extraction features and the text extraction features are input into the splicing layer, the output of the splicing layer is connected to the input of the third RMS normalization layer, the output of the third RMS normalization layer is connected to the input of the ninth fully connected layer and the tenth fully connected layer respectively, the output of the ninth fully connected layer is connected to the fourth convolutional layer, the seventh activation layer, and the fourth SSM module in sequence, the output of the tenth fully connected layer is connected to the input of the eighth activation layer, the output of the eighth activation layer is dot-multiplied with the output of the fourth SSM module and then input into the eleventh fully connected layer, the output of the eleventh fully connected layer is summed with the output of the splicing layer and then input into the splitting layer, and the output data of the splitting layer is used as the output of the cross-modal selective alignment module. Specifically:

[0063] The embodiment of the present invention extracts visual features and text extraction features Input into the cross-modal selective alignment module for efficient selective cross-modal alignment. The calculation formula is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] in, represents the visual alignment feature, represents the text alignment feature, Represents the concatenation layer, which is used to extract visual features and text extraction features Perform concatenation operations on the sequence length dimension, represents the third RMS normalization layer, represents the ninth fully connected layer, represents the fourth convolutional layer, represents the seventh activation layer, Indicates the fourth SSM module. represents the tenth fully connected layer, represents the eighth activation layer, represents the eleventh fully connected layer, Represents the splitting layer, which is used to perform splitting operations on the sequence length dimension according to the original sequence length.

[0068] Furthermore, in the calculation module, the output of the visual selective filtering module is , output of the text selective filtering module With the output of the cross-modal selective alignment module 、 Perform average pooling operations respectively to obtain the corresponding global embedding features: 、 、 、 Finally, a fusion operation is used to combine these global embedding features from intra-modal and cross-modal interactions, and pass them through a fully connected layer to obtain the final visual fusion feature and text fusion feature. The calculation formula is as follows:

[0069] ;

[0070] ;

[0071] in, Represents the visual fusion feature, represents the text fusion feature, Represents the fully connected layer after the fusion of the global embedding features of vision, The fully connected layer after the fusion of the global embedding features representing the text. Furthermore, a similarity function, used to measure the similarity between image and text features, is used to calculate the similarity between the visual fusion features and the text fusion features. A determination is made as to whether the calculated similarity meets a similarity threshold, which is set based on the actual situation. Finally, the image of the traffic event corresponding to the visual fusion features that meet the similarity threshold is used as the recognition result for the traffic event to be identified, enabling cross-modal retrieval of the traffic event to be identified. For multiple traffic event images that meet the similarity threshold, the image of the traffic event with the highest similarity is selected as the recognition result for the traffic event to be identified.

[0072] Furthermore, the embodiment of the present invention adopts existing image-text matching training algorithms for training, such as contrastive learning. In the training process of the traffic event recognition network model, the loss function adopted by the embodiment of the present invention is the two-way triple contrast loss, which is expressed as follows:

[0073] ;

[0074] in, represents the bidirectional triplet contrast loss, represents the margin hyperparameter, express operate, It means to select the maximum value between 0 and another data. represents an image of any traffic event in the training sample set, Indicates that the training sample set The corresponding text description, and Constitute a positive sample pair, Indicates that the training sample set Any image of a traffic incident, Indicates that the training sample set The text description of any traffic event other than and 、 and They constitute negative sample pairs respectively, Represents a similarity function used to measure the similarity between image features and text features. The bidirectional triplet contrast loss adopted encourages matching image-text positive samples to be stronger than mismatched image-text negative samples, achieving higher similarity scores.

[0075] It should be noted here that before recognition and training, the traffic event data needs to be cleaned, and image-text sample pair data needs to be constructed. The image-text sample pairs are then paired with positive samples and negative samples to ensure the consistency of the traffic event images and their corresponding text descriptions, providing a high-quality data foundation for the recognition and training process.

[0076] It should be noted that the embodiment of the present invention can also add a verification process based on a verification data set. The verification process refers to the training process. Finally, the recognition results based on the training data set and the verification data set can be stored in a database. The data stored in the database is used to retrieve images of traffic events to be identified. In actual application, the image of the traffic event to be identified can be identified only based on the text description corresponding to the traffic event to be identified. The specific recognition process is: calculating the similarity between the text description corresponding to the traffic event to be identified and the images of all traffic events in the database, judging whether the similarity meets the similarity threshold, and using the image of the traffic event that meets the similarity threshold as the recognition result of the traffic event to be identified. For multiple images of traffic events that meet the similarity threshold, the image of the traffic event with the highest similarity is selected as the recognition result of the traffic event to be identified.

[0077] In order to verify the effectiveness of the intelligent traffic event recognition method based on large-scale traffic model and cross-modal retrieval provided by the embodiment of the present invention, the following experiments were conducted for verification.

[0078] 1. Experimental simulation parameters

[0079] Experiments in this paper were conducted using the deep learning framework Pytorch-GPU 2.0.0 on a Linux operating system equipped with an Intel(R) Xeon(R) Gold 6248 CPU at 2.50GHz and an NVIDIA Tesla V100 GPU. Traffic event image and text data were obtained from the public datasets Flickr30K and MS-COCO.

[0080] The methods compared in the experiment include: a multimodal cross attention network based on the self-attention mechanism, denoted as the MMCA method in the experiment, and the reference is "X. Wei, T. Zhang, Y. Li, Y. Zhang, and F. Wu, "Multi-modality cross attention network for image and sentence matching," inProceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 10 941–10 950."; a method based on similarity graph reasoning and attention filtering, denoted as the SGRAF method in the experiment, and the reference is "H. Diao, Y. Zhang, L. Ma, and H. Lu, "Similarityreasoning and filtration for image-text matching," inProceedings of the AAAIconference on artificial intelligence, vol. 35, no. 2, 2021, pp. 1218–1226."; a hierarchical alignment network based on the classic Transformer, denoted as the HAT method in the experiment, and the reference is "Y. Bin, H.Li, Y. Xu, X. Xu, Y. Yang, and HT Shen, “Unifying two-stream encoders with transformers for cross-modal retrieval,” in Proceedings of the 31st ACMInternational Conference on Multimedia, 2023, pp. 3041–3050.”

[0081] Used in the experiment Indicators are used as retrieval evaluation indicators by using Metrics to measure image-to-text retrieval ( ) and text-to-image retrieval ( ) task performance, referred to as It measures the top The correct retrieval ratio of the name. To evaluate the performance of the model, it is defined as follows:

[0082] .

[0083] 2. Experimental simulation content

[0084] The retrieval evaluation indicators of the traffic scene image and text test datasets in the public datasets Flickr30K and MS-COCO 1K are calculated respectively, and compared with the retrieval evaluation indicators of the MMCA method, SGRAF method and HAT method. The results are shown in Tables 1 and 2.

[0085] Table 1 Retrieval evaluation indicators of Flickr30K dataset

[0086]

[0087] Table 2. MS-COCO 1K dataset retrieval evaluation metrics

[0088]

[0089] As can be seen from Tables 1 and 2, the visual selective filtering module and text selective filtering module adopted by the present invention can implicitly and selectively filter irrelevant visual information and unnecessary word embedding features (such as prepositions), and the cross-modal selective alignment module can selectively align key fine-grained features across modalities. Therefore, extremely significant retrieval results are achieved on the public datasets Flickr30K and MS-COCO 1K, verifying the effectiveness of the proposed method.

[0090] In summary, the embodiment of the present invention proposes an intelligent traffic event recognition method for a large traffic model based on cross-modal retrieval. The designed traffic event recognition network model introduces a visual selective filtering module, a text selective filtering module, and a cross-modal selective alignment module. By performing implicit and selective fine-grained matching between the image and text of the traffic event, the model can effectively filter out irrelevant information, strengthen the perception and alignment of key features, thereby improving the accuracy and speed of cross-modal retrieval, and significantly enhancing the ability to accurately identify and locate traffic events. At the same time, the representation learned by the model is optimized and updated, solving the problems of invalid semantic alignment and high computational complexity caused by global allocation in the traditional cross-modal attention mechanism. In general, the embodiment of the present invention uses the designed traffic event recognition network model based on multimodal data of traffic scene images and text descriptions to achieve high-precision and high-efficiency traffic event recognition and location, while greatly reducing computational complexity. This makes the method proposed by the present invention more suitable for the field of traffic monitoring and management, such as helping traffic management departments to monitor traffic conditions in real time, quickly respond to emergencies, optimize traffic flow, improve road safety, reduce traffic accident rates, promote the construction of intelligent transportation systems, enhance urban traffic management, and promote the development of smart cities.

[0091] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0092] Although the present invention is described herein in conjunction with various embodiments, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the specification and accompanying drawings in the process of implementing the claimed invention. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components or steps. The fact that certain measures are described in different embodiments does not mean that these measures cannot be combined to produce good results.

[0093] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for intelligent traffic event recognition based on a large traffic model of cross-modal retrieval, characterized by: The method comprises: Collect text descriptions corresponding to the traffic events to be identified; Based on the trained traffic event recognition network model, the similarity between the text description of the traffic event to be recognized and each traffic event image stored in the database is calculated; whether the similarity meets the similarity threshold is determined, and the image of the traffic event with the highest similarity is selected from all traffic event images that meet the similarity threshold as the recognition result of the traffic event to be recognized; wherein, The trained traffic event recognition network model includes a visual feature extraction module, a text feature extraction module, a visual selective filtering module, a text selective filtering module, a cross-modal selective alignment module and a calculation module; the visual feature extraction module is used to extract visual features from each traffic event image in the database and map the extracted visual features to a preset dimension to obtain corresponding visual extraction features; the text feature extraction module is used to extract text features from the text description corresponding to the traffic event to be identified and map the extracted text features to the preset dimension to obtain text extraction features; the visual selective filtering module is used to selectively filter irrelevant visual extraction features to obtain visual filtering features; the text selective filtering module is used to selectively filter the text extraction features to obtain text filtering features; the cross-modal selective alignment module is used to perform cross-modal alignment processing on the visual extraction features and the text extraction features to obtain visual alignment features and text alignment features; the calculation module is used to fuse the visual filtering features and the visual alignment features to obtain visual fusion features, and fuse the text filtering features and the text alignment features to obtain text fusion features, and calculate the similarity between the visual fusion features and the text fusion features; The visual selective filtering module includes a first RMS normalization layer, a first fully connected layer to a fifth fully connected layer, a first convolutional layer, a second convolutional layer, a first activation layer to a third activation layer, a first SSM module and a second SSM module; the text selective filtering module includes a second RMS normalization layer, a sixth fully connected layer to an eighth fully connected layer, a third convolutional layer, a fifth activation layer, a sixth activation layer and a third SSM module.

2. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1, characterized in that: The visual feature extraction module is implemented using the Faster R-CNN network.

3. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1 is characterized in that: The text feature extraction module is implemented using the BERT encoder.

4. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1 is characterized in that: In the visual selective filtering module: The visual extraction feature is input into the first RMS normalization layer, and the output end of the first RMS normalization layer is connected to the input ends of the first fully connected layer and the second fully connected layer respectively. The output end of the first fully connected layer is connected to the first convolutional layer, the first activation layer, and the first SSM module in sequence. The output end of the second fully connected layer is connected to the input end of the second activation layer. The output end of the second activation layer is dot-multiplied with the output end of the first SSM module to obtain a first dot product result. The output data of the first RMS normalization layer is flipped and input into the third fully connected layer and the fourth fully connected layer respectively. The output end of the third fully connected layer is connected to the second convolutional layer, the third activation layer, and the second SSM module in sequence. The output end of the fourth fully connected layer is connected to the input end of the fourth activation layer. The output end of the fourth activation layer is dot-multiplied with the output end of the second SSM module to obtain a second dot product result. The second dot product result is flipped and summed with the first dot product result, and the sum result is input into the fifth fully connected layer. The output end of the fifth fully connected layer is summed with the visual extraction feature as the output of the visual selective filtering module.

5. The method for intelligent traffic event identification based on a large traffic model and cross-modal retrieval according to claim 1 is characterized in that: After the visual selective filtering module, a first average pooling layer is also included; the first average pooling layer is used to perform average pooling processing on the visual filtering features and input the visual filtering features after average pooling processing into the calculation module.

6. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1 is characterized in that: In the text selective filtering module: The text extraction feature is input into the second RMS normalization layer, the output end of the second RMS normalization layer is connected to the input ends of the sixth fully connected layer and the seventh fully connected layer respectively, the output end of the sixth fully connected layer is connected to the third convolutional layer, the fifth activation layer, and the third SSM module in sequence, the output end of the seventh fully connected layer is connected to the input end of the sixth activation layer, the output end of the sixth activation layer is dot-multiplied with the output end of the third SSM module and then input into the eighth fully connected layer, and the output end of the eighth fully connected layer and the text extraction feature are summed as the output of the text selective filtering module.

7. The method for intelligent traffic event identification based on a large traffic model and cross-modal retrieval according to claim 1 is characterized in that: After the text selective filtering module, a second average pooling layer is also included; the second average pooling layer is used to perform average pooling processing on the text filtering features and input the text filtering features after average pooling processing into the calculation module.

8. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1 is characterized in that: The cross-modal selective alignment module includes a splicing layer, a third RMS normalization layer, a ninth to eleventh fully connected layers, a fourth convolutional layer, a seventh activation layer, an eighth activation layer, a fourth SSM module, and a splitting layer; wherein, The visual extraction features and the text extraction features are input into the splicing layer, the output end of the splicing layer is connected to the input end of the third RMS normalization layer, the output end of the third RMS normalization layer is connected to the input ends of the ninth fully connected layer and the tenth fully connected layer respectively, the output end of the ninth fully connected layer is connected to the fourth convolutional layer, the seventh activation layer, and the fourth SSM module in sequence, the output end of the tenth fully connected layer is connected to the input end of the eighth activation layer, the output end of the eighth activation layer is dot-multiplied with the output end of the fourth SSM module and then input into the eleventh fully connected layer, the output end of the eleventh fully connected layer and the output end of the splicing layer are summed and input into the splitting layer, and the output data of the splitting layer is used as the output of the cross-modal selective alignment module.

9. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1 is characterized in that: After the cross-modal selective alignment module, a third average pooling layer and a fourth average pooling layer are also included; the third average pooling layer is used to perform average pooling processing on the visual alignment features, and input the visual alignment features after the average pooling processing into the calculation module; the fourth average pooling layer is used to perform average pooling processing on the text alignment features, and input the text alignment features after the average pooling processing into the calculation module.

10. The method for intelligent traffic event identification based on a large traffic model using cross-modal retrieval according to claim 1, characterized in that: During the training process of the traffic incident recognition network model, the loss function used is the two-way triplet contrast loss, which is expressed as follows: ； in, represents the bidirectional triplet contrast loss, represents the margin hyperparameter, express operate, It means to select the maximum value between 0 and another data. represents an image of any traffic event in the training sample set, Indicates that the training sample set The corresponding text description, and Constitute a positive sample pair, Indicates that the training sample set Any image of a traffic incident, Indicates that the training sample set The text description of any traffic event other than and 、 and They constitute negative sample pairs respectively, Represents a similarity function used to measure the similarity between image features and text features.

Images