Multimodal traffic perception method based on UNITER-Yolov5 model

Abstract

The present application relates to a multimodal traffic perception method based on the UNITER‑Yolov5 model. The method comprises: obtaining image data and text description data in a traffic scene; inputting the image data into the Yolov5 module of the trained multimodal traffic perception model based on UNITER‑Yolov5 for preprocessing, and obtaining image convolution features of K target areas; inputting the text description data and the image data into the UNITER module of the multimodal traffic perception model for preprocessing, and obtaining the semantic features of the image; inputting the semantic features of the image and the image convolution features of the K target areas into the improved Yolov5 module of the multimodal traffic perception model for perception, and outputting the perception results of the traffic scene. It can effectively integrate image information and natural language information, cope with the target perception of the current multimodal environment and complex traffic environment, provide a more comprehensive target understanding, and thus improve the accuracy of detection.

Description

Technical Field

[0001] The present application relates to the technical field of target detection and multimodal fusion, and in particular to a multimodal traffic perception method based on the UNITER-Yolov5 model. Background Art

[0002] The application of object detection technology in the field of transportation covers many aspects, bringing significant changes to the fields of traffic management, intelligent transportation systems and autonomous driving. It is widely used in vehicle detection, pedestrian detection, intersection management, traffic sign and signal detection, intelligent traffic light control, traffic accident prevention, etc. These applications have jointly promoted the development of intelligence and automation in the field of transportation, and provided more efficient and safer solutions for urban transportation systems. With the continuous advancement of technology, the application of object detection in the field of transportation is expected to be further expanded and optimized.

[0003] Deep learning methods dominate the field of target detection. In particular, the successful application of convolutional neural networks (CNNs), such as Faster R-CNN, YOLO, SSD, etc., has become a classic algorithm in the field of target detection. However, multimodal fusion perception is still a challenge. Data of different perception modalities have different representations and structures, so it is necessary to design a model that can handle heterogeneous data. However, the current traffic perception method based on multimodal fusion has a poor effect in extracting semantic information from images, which makes the accuracy of the multimodal fusion traffic perception method low. Summary of the invention

[0004] Based on this, it is necessary to provide a multimodal traffic perception method based on the UNITER-Yolov5 model that can improve the accuracy of the traffic perception method of multimodal fusion in response to the above technical problems.

[0005] A multimodal traffic perception method based on UNITER-Yolov5 model, the method comprising:

[0006] Obtain image data and text description data in traffic scenes;

[0007] Input the image data into the trained Yolov5 module of the multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas;

[0008] Inputting the text description data and the image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain semantic features of the image;

[0009] The semantic features of the image and the image convolution features of K target areas are input into the improved Yolov5 module of the multimodal traffic perception model for perception, and the perception result of the traffic scene is output.

[0010] In one embodiment, the improved Yolov5 module includes: a convolutional layer, a maximum pooling operation, a fully connected layer, a splicing operation and a fusion layer;

[0011] The semantic features of the image are input into the fully connected layer for feature extraction to obtain text features;

[0012] The image convolution features of the K target areas are input into the convolution layer for processing to obtain preliminary image features, the preliminary image features are subjected to a maximum pooling operation to reduce feature map vectors, and image features are output;

[0013] The text features and the image features are spliced ​​through a splicing operation, and then input into a fusion layer for processing, and the perception result of the traffic scene is output.

[0014] In one embodiment, the multimodal traffic perception model is trained as follows:

[0015] Get a sample set of image-text pairs,

[0016] Dividing the image-text pair samples in the image-text pair sample set to obtain a training set and a verification set, wherein the image-text pair samples consist of image samples and corresponding text description samples;

[0017] The multimodal traffic perception model is trained using the training set, and the loss is determined according to a set multimodal comprehensive loss function. The expression of the multimodal comprehensive loss function L is:

[0018] L=αL I +(1-α)L T ,

[0019] L I =L loc +L obj +L cls ,

[0020] L T =ω1L M +ω2L C +ω3L D ,

[0021] Among them, α is the weight of the image feature, L I is the loss function of the YOLOv5 module, L T is the loss function of the UNITER module, L loc is the positioning loss, Lobj is the confidence loss, L cls is the classification loss, ω1 is the weight of the image-text matching loss, L M is the image-text matching loss, ω2 is the weight of the image-text contrast loss, L C is the image-text contrast loss, ω3 is the weight of the language modeling loss, and L D Modeling loss for language;

[0022] According to the loss, a back propagation algorithm of the model is used to perform gradient calculation to determine a gradient, and a network parameter of the multimodal traffic perception model is updated according to the gradient until the multimodal comprehensive loss function reaches convergence, thereby obtaining a preliminarily trained multimodal traffic perception model;

[0023] The verification set is input into the preliminarily trained multimodal traffic perception model for verification. When the verification result meets the accuracy requirement, the trained multimodal traffic perception model based on UNITER-Yolov5 is obtained. If the verification result does not meet the accuracy requirement, the training is continued until the verification result meets the accuracy requirement, and the trained multimodal traffic perception model based on UNITER-Yolov5 is obtained.

[0024] In one embodiment, the expression for gradient calculation is:

[0025]

[0026] in, is the gradient of the multimodal comprehensive loss function L with respect to the network parameters θ.

[0027] In one embodiment, the inputting the text description data and the image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image includes:

[0028] The text description data is divided into tokens to obtain the tokenized text, and then the tokenized text is mapped to a unique integer using bert-base-chinese and bert-base-cased, and then the integer is converted into an embedding vector;

[0029] Then, a start tag and an end tag are added to the text sequence in the segmented text, and finally an attention mask is generated to indicate the position of the real text, thereby obtaining the preprocessed text description data.

[0030] The preprocessed text description data and the image data are input into the UNITER module of the multimodal traffic perception model for preprocessing to obtain semantic features of the image.

[0031] In one embodiment, the image data is input into a Yolov5 module of a trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas, including:

[0032] The image data is input into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain convolutional features and K target areas. Each target area Among them, the target area border coordinates b i =(x min ,y min ,x max ,y max ), (x min ,y min ) is the coordinate of the upper left corner of the target area border, (x max ,y max ) is the coordinate of the lower right corner of the target area border, p i is the probability that the target belongs to a certain category, c i is the category label to which the target belongs; and the convolution features of each target area in the convolution features are cropped according to the coordinates of the target area border to obtain K target areas and image convolution features of the K target areas.

[0033] In one embodiment, the convolution function of the convolution layer is:

[0034]

[0035] Among them, Y i,j,k is the element in the output preliminary image feature, W m,n,c,k is the weight of the convolution kernel, X i+m,j+n,c is the element in the convolution feature of the input image, b k is the bias term, m is the vertical offset of the convolution kernel, n is the horizontal offset of the convolution kernel, and c is the number of channels of the input image convolution feature.

[0036] In one embodiment, the expression of the maximum pooling operation is:

[0037] Y i,j,c =Max(X (i·s+m),(j·s+n),c )

[0038] Among them, Y i,j,c is the element in the image feature output after the maximum pooling operation, located in the i-th row, j-th column, and c-th channel, X (i·s+m),(j·s+n),cis the element in the pooling window, (i·s+m) represents the offset of the pooling window in the vertical direction; (j·s+n) is the offset of the pooling window in the horizontal direction, and c is the number of channels.

[0039] The multimodal traffic perception method based on the UNITER-Yolov5 model obtains image data and text description data in the traffic scene; inputs the image data into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas; inputs the text description data and image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image; inputs the semantic features of the image and the image convolution features of the K target areas into the improved Yolov5 module of the multimodal traffic perception model for perception, and outputs the perception results of the traffic scene. It can effectively integrate image information and natural language information, cope with the target perception in the current multimodal environment and complex traffic environment, provide a more comprehensive target understanding, and thus improve the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 1 is a flow chart of a multimodal traffic perception method based on the UNITER-Yolov5 model in one embodiment;

[0041] Figure 2 Schematic diagram of a multimodal traffic perception method based on UNITER-Yolov5 model in one embodiment;

[0042] Figure 3 A schematic diagram of performing word segmentation and encoding on text description data in one embodiment;

[0043] Figure 4 It is a schematic diagram of the structure of an improved Yolov5 module in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, Figure 1 and Figure 2 As shown, a multimodal traffic perception method based on the UNITER-Yolov5 model is provided, comprising the following steps:

[0046] Step S220, acquiring image data and text description data in the traffic scene.

[0047] Among them, traffic scenes can be traffic scenes such as automatic driving assisted recognition and intelligent navigation.

[0048] Among them, in the automatic driving assisted recognition of traffic scenes, the vehicle's camera can capture road images (i.e. image data), while the text description data can come from map data or voice assistant instructions. Through the multimodal traffic perception method based on the UNITER-Yolov5 model, the system can more accurately identify targets such as traffic signs, pedestrians, and vehicles.

[0049] Among them, in intelligent navigation traffic scenarios, the road images (i.e., image data) obtained by the camera are combined with the text description of the navigation instructions (i.e., text description data), which can help artificial intelligence better understand the environment and perform tasks.

[0050] Step S240: input the image data into the trained Yolov5 module of the multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas.

[0051] Among them, the Yolov5 model is used as the Yolov5 module in the multimodal traffic perception model based on UNITER-Yolov5. The Yolov5 model is a network model for object detection tasks designed in computer vision mode.

[0052] In one embodiment, the image data is input into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas, including:

[0053] The image data is input into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain convolutional features and K target areas. Each target area Among them, the target area border coordinates b i =(x min ,y min ,x max ,y max ), (x min ,y min ) is the coordinate of the upper left corner of the target area border, (x max ,y max ) is the coordinate of the lower right corner of the target area border, p i is the probability that the target belongs to a certain category, c i is the category label to which the target belongs; and the convolution features of each target area in the convolution feature are cropped according to the coordinates of the target area border to obtain K target areas and image convolution features of K target areas.

[0054] The cropping operation is as follows:

[0055] cropped_features=conv_features[:,:,int(y):int(y+h),int(x):int(x+w)].

[0056] Among them, y: represents the ordinate of the top border of the target area border, which specifies the starting ordinate of the feature map to be cropped; h: represents the height of the target area border, which determines the height of the feature map to be cropped; x: represents the abscissa of the left border of the target area border, which specifies the starting abscissa of the feature map to be cropped; w: represents the width of the target area border, which determines the width of the feature map to be cropped.

[0057] Step S260: input the text description data and the image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image.

[0058] Among them, the UNITER model is used as the UNITER module in the multimodal traffic perception model based on UNITER-Yolov5. The UNITER (Universal Image-Text Representation Learning) model is a deep learning model that realizes multimodal understanding and representation learning between images and texts.

[0059] In one embodiment, the text description data and the image data are input into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image, including:

[0060] The text description data is divided into words to obtain the segmented text, and then the segmented text is mapped to a unique integer using bert-base-chinese and bert-base-cased, and then the integer is converted into an embedding vector; then the start tag and the end tag are added to the text sequence in the segmented text, and finally an attention mask is generated to indicate the position of the real text to obtain the preprocessed text description data; the preprocessed text description data and image data are input into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image.

[0061] Among them, the schematic diagram of word segmentation and encoding of text description data is as follows Figure 3 shown.

[0062] Before the image data is input into the UNITER module of the multimodal traffic perception model, the image data can also be converted into a format acceptable to the UNITER module. First, use the Image.open method in the PIL library to open the image file of the specified path and create an Image object. Then load the image file, convert it to RGB mode, and resize it to ensure that the image has a fixed size so that it can be input into the UNITER module.

[0063] Among them, the semantic features of the image input by the UNITER module are a PyTorch tensor of size (224, 224).

[0064] Step S280, inputting the semantic features of the image and the image convolution features of the K target areas into the improved Yolov5 module of the multimodal traffic perception model for perception, and outputting the perception result of the traffic scene.

[0065] Among them, the improved Yolov5 module is an improved network model based on the existing Yolov5 model.

[0066] In one embodiment, Figure 4 As shown, the improved Yolov5 module includes: convolution layer, maximum pooling operation, fully connected layer, splicing operation and fusion layer;

[0067] The semantic features of the image are input into the fully connected layer for feature extraction to obtain text features;

[0068] The image convolution features of K target regions are input into the convolution layer for processing to obtain preliminary image features. The preliminary image features are subjected to the maximum pooling operation to reduce the feature map vector and output the image features.

[0069] After the text features and image features are spliced ​​through the splicing operation, they are input into the fusion layer for processing and the perception results of the traffic scene are output.

[0070] In one embodiment, the convolution function of the convolution layer is:

[0071]

[0072] Among them, Y i,j,k is the element in the output preliminary image feature, W m,n,c,k is the weight of the convolution kernel, X i+m,j+n,c is the element in the convolution feature of the input image, b k is the bias term, m is the vertical offset of the convolution kernel, n is the horizontal offset of the convolution kernel, and c is the number of channels of the input image convolution feature.

[0073] In one embodiment, the expression of the maximum pooling operation is:

[0074] Y i,j,c =Max(X (i·s+m),(j·s+n),c )

[0075] Among them, Y i,j,c is the element in the image feature output after the maximum pooling operation, located in the i-th row, j-th column, and c-th channel, X (i·s+m),(j·s+n),c is the element in the pooling window, (i·s+m) represents the offset of the pooling window in the vertical direction; (j·s+n) is the offset of the pooling window in the horizontal direction, and c is the number of channels.

[0076] Among them, the improved Yolov5 module is improved as follows:

[0077] 3.1) Define a PyTorch model class that inherits from nn.Module. This class is used to customize the input layer of the YOLOv5 module to simultaneously accept the image convolution features of K target regions and the semantic features of the image from the UNITER module as input.

[0078] 3.2) Define the class constructor to initialize the instance of the above model class. It accepts two parameters, representing the number of channels of the image and the dimension size of the text embedding. Call the constructor of the parent class nn.Module to ensure that the model is initialized correctly.

[0079] 3.3) Locate the convolution layer of YOLOv5 and create another convolution layer (nn.Conv2d) to process the input image convolution features. This includes defining the number of channels of the input image convolution features, the number of output channels of the convolution layer, the size of the convolution kernel, the step size of the convolution kernel sliding on the image convolution features, and the number of layers of zero values ​​added around the input image convolution features to maintain the output size.

[0080] 3.4) Create a fully connected layer to process the text input and define the dimensions of the input features and the output features of the fully connected layer.

[0081] 3.5) Use PyTorch’s nn.ModuleList or nn.Sequential to concatenate the two inputs as a new YOLOv5 input layer.

[0082] Among them, the image processing layer (i.e. convolution layer + maximum pooling operation):

[0083] O I =Conv(I,W I )

[0084] Text processing layer (i.e. fully connected layer):

[0085] O T=Linear(T,W T )

[0086] Fusion layer:

[0087] O C =F(Con(O I ,O T ),W F )

[0088] Among them, the image input is I, and the output of the image processing layer is O I , the text input is T, and the output of the text processing layer is O T . Set the weight of the convolutional layer to W I , the weight of the fully connected layer is W T , the weight of the fusion layer is W F Conv represents the convolution operation, Linear represents the fully connected operation, Con represents the connection operation, and F represents the operation of the fusion layer.

[0089] Among them, the weight of the fusion layer is determined by using the feature weight fusion method to merge the semantic information from UNITER with the features of YOLOv5. The weight should be determined from both qualitative and quantitative aspects.

[0090] Among them, qualitatively: object detection contributes more to images, while in the image description generation task, the contribution of text is more significant. Tasks with greater contributions are assigned greater weights and then fine-tuned. If it is a joint task, further quantitative operations are required.

[0091] Among them, quantitative: design an ablation experiment, gradually remove one of the modes, then retrain the model, and observe the changes in model performance.

[0092] Follow these steps:

[0093] Experiment 1: Use the UNITER model alone, train it and evaluate its performance.

[0094] Experiment 2: Use the YOLOv5 model alone, train and evaluate the performance.

[0095] Experiment 3: Use the joint model (UNITER+YOLOv5), train and evaluate the performance.

[0096] Among them, the comprehensive accuracy is used as the evaluation index to calculate the relative accuracy, and the weight of the combined features is introduced into the calculation of the relative accuracy gain. The formula for the relative accuracy gain combined with the weight is as follows:

[0097]

[0098] C=α×I+(1-α)×T

[0099] Among them: A is the comprehensive accuracy, α is the weight of the image feature, A I is the accuracy of using image features alone, A C To measure the accuracy of the combined use of image features and text features, C is the comprehensive feature, I is the image feature, T is the text feature, P is the precision, which measures the accuracy of the model's prediction of positive examples, and R is the recall rate, which measures the model's ability to find all positive examples.

[0100] After matching the image and text, a fused feature map will be generated. The number of channels of this feature map will change, assuming it is C. Assuming that the size of the matched feature map is M×N×C, the size of the convolution kernel will become K×L×C. The size of the output feature map is P×Q×K. After matching the image and text data, the weights and bias items of the convolution layer will need to adapt to the new number of channels. Therefore, the convolution function (Conv) is:

[0101]

[0102] Among them, Y i,j,k is the element in the output preliminary image feature, W m,n,c,k is the weight of the convolution kernel, X i+m,j+n,c is the element in the convolution feature of the input image, b k is the bias term, m is the vertical offset of the convolution kernel, n is the horizontal offset of the convolution kernel, and c is the number of channels of the input image convolution feature.

[0103] Among them, the improved Yolov5 module realizes cross-modal connection and information transmission mechanism by adding fully connected layers, splicing operations and fusion layers in the network, so that the semantic understanding information of the UNITER module can be transmitted to the target detection part.

[0104] Among them, after the image convolution feature input passes through the above-mentioned convolution layer and maximum pooling operation, the output image feature is ready to be fused with the text feature. The image feature is also reshaped through the view operation to ensure that its dimension is consistent with the text feature. The semantic feature input of the image is first processed through the fully connected layer and fused with the image feature. Then, the image feature and the text feature are connected together (connected in the first dimension) through torch.cat. Finally, the connected features are passed to the fusion layer for processing to obtain the final fusion output.

[0105] The multimodal traffic perception method based on the UNITER-Yolov5 model obtains image data and text description data in the traffic scene; inputs the image data into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas; inputs the text description data and image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image; inputs the semantic features of the image and the image convolution features of the K target areas into the improved Yolov5 module of the multimodal traffic perception model for perception, and outputs the perception results of the traffic scene. It can effectively integrate image information and natural language information, cope with the target perception in the current multimodal environment and complex traffic environment, provide a more comprehensive target understanding, and thus improve the accuracy of detection.

[0106] In one embodiment, the multimodal traffic perception model is trained as follows:

[0107] Get a sample set of image-text pairs,

[0108] The image-text pair samples in the image-text pair sample set are divided to obtain a training set and a validation set, wherein the image-text pair samples are composed of image samples and corresponding text description samples;

[0109] The multimodal traffic perception model is trained using the training set, and the loss is determined according to the set multimodal comprehensive loss function. The expression of the multimodal comprehensive loss function L is:

[0110] L=αL I +(1-α)L T ,

[0111] L I =L loc +L obj +L cls ,

[0112] L T =ω1L M +ω2L C +ω3L D ,

[0113] Among them, α is the weight of the image feature, L I is the loss function of the YOLOv5 module, L T is the loss function of the UNITER module, L loc is the positioning loss, L obj is the confidence loss, L cls is the classification loss, ω1 is the weight of the image-text matching loss, L Mis the image-text matching loss, ω2 is the weight of the image-text contrast loss, L C is the image-text contrast loss, ω3 is the weight of the language modeling loss, and L D Modeling loss for language;

[0114] According to the loss, the back propagation algorithm of the model is used to calculate the gradient and determine the gradient. The network parameters of the multimodal traffic perception model are updated according to the gradient until the multimodal comprehensive loss function reaches convergence, and a preliminarily trained multimodal traffic perception model is obtained.

[0115] The verification set is input into the preliminarily trained multimodal traffic perception model for verification. When the verification result meets the accuracy requirement, the trained multimodal traffic perception model based on UNITER-Yolov5 is obtained. If the verification result does not meet the accuracy requirement, the training is continued until the verification result meets the accuracy requirement, and the trained multimodal traffic perception model based on UNITER-Yolov5 is obtained.

[0116] The image-text pair samples in the image-text pair sample set can be obtained by searching various objects on Flickr from COCO data. Each image has at least 3 objects with text annotations. VG (Visual Genome) divides the images obtained from the COCO dataset into multiple regions, each of which has a corresponding text description. The overlapping data in the dataset is excluded. Subsequently, the image-text pairs are created using the remaining images and corresponding descriptions as image-text pair samples.

[0117] In one embodiment, the expression for gradient calculation is:

[0118]

[0119] in, is the gradient of the multimodal comprehensive loss function L with respect to the network parameters θ.

[0120] in,

[0121] Among them, a joint loss function is designed to consider the goals of object detection and semantic understanding at the same time. During the joint training process, the weights of the two are ensured to be balanced so as to fully utilize their advantages in the task.

[0122] The loss function is defined as follows. The contribution of the two modes is calculated, and the comprehensive loss function is determined based on the contribution. The loss function of the YOLOv5 module includes the positioning loss (L loc ), confidence loss (L obj ), classification loss (L cls ), the weights of the individual losses are determined by hyperparameters.

[0123] L I =L loc +L obj +L cls

[0124] Among them, the loss function of the UNITER module includes the image text matching task loss (L M ), image-text contrast task loss (L C ), language modeling task loss (L D ).

[0125] L T =ω1L M +ω2L C +ω3L D

[0126] Among them, ω1, ω2, and ω3 are the weights of each loss, which are also determined by the hyperparameters in the training process.

[0127] After further clarifying the contribution rates of the YOLOv5 module and the UNITER module in comprehensive training, they are applied to determine the multimodal comprehensive loss function:

[0128] L=αL I +(1-α)L T

[0129] Among them, the modification of the convolutional layer will change the model's ability to extract features from the input data, which in turn affects the feature representation of each modality. The gradient is calculated through the back propagation algorithm of the model to optimize the comprehensive loss function during training.

[0130] Compute the gradient:

[0131]

[0132] Use the gradient descent algorithm to update the model parameters according to the gradient.

[0133]

[0134] Among them, η is the learning rate, which controls the step size of parameter update, and θ is the model parameter (including the network layer parameters of the two modes).

[0135] When the comprehensive loss function reaches convergence, the final comprehensive loss function is obtained.

[0136] Among them, the comprehensive accuracy of the multimodal traffic perception model based on UNITER-Yolov5 is calculated to determine whether the accuracy requirement is met. When calculating the comprehensive accuracy of the multimodal traffic perception model based on UNITER-Yolov5, the basic indicators of model performance must be calculated first: accuracy (P) and recall (R). The specific calculation is as follows:

[0137]

[0138]

[0139] Among them, TA is the number of correctly detected bounding boxes, FP is the number of incorrectly detected bounding boxes, and FN is the number of undetected bounding boxes.

[0140] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multimodal traffic perception method based on UNITER-Yolov5 model, characterized in that: The method comprises: Obtain image data and text description data in traffic scenes; Input the image data into the trained Yolov5 module of the multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas; Inputting the text description data and the image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain semantic features of the image; Inputting the semantic features of the image and the image convolution features of the K target areas into the improved Yolov5 module of the multimodal traffic perception model for perception, and outputting the perception result of the traffic scene; The improved Yolov5 module includes: a convolutional layer, a maximum pooling operation, a fully connected layer, a splicing operation and a fusion layer; The semantic features of the image are input into the fully connected layer for feature extraction to obtain text features; The image convolution features of the K target areas are input into the convolution layer for processing to obtain preliminary image features, the preliminary image features are subjected to a maximum pooling operation to reduce feature map vectors, and image features are output; The text features and the image features are spliced ​​through a splicing operation, and then input into a fusion layer for processing, and the perception result of the traffic scene is output.

2. The method according to claim 1, characterized in that The training method of the multimodal traffic perception model is: Get a sample set of image-text pairs, Dividing the image-text pair samples in the image-text pair sample set to obtain a training set and a verification set, wherein the image-text pair samples consist of image samples and corresponding text description samples; The multimodal traffic perception model is trained using the training set, and the loss is determined according to a set multimodal comprehensive loss function, wherein the multimodal comprehensive loss function The expression is: ， ， ， in, is the weight of the image feature, is the loss function of the YOLOv5 module, is the loss function of the UNITER module, For the positioning loss, is the confidence loss, is the classification loss, is the weight of the image-text matching loss, is the image-text matching loss, is the weight of the image-text contrast loss, is the image-text contrast loss, is the weight for the language modeling loss, Modeling loss for language; According to the loss, a back propagation algorithm of the model is used to perform gradient calculation to determine a gradient, and a network parameter of the multimodal traffic perception model is updated according to the gradient until the multimodal comprehensive loss function reaches convergence, thereby obtaining a preliminarily trained multimodal traffic perception model; The verification set is input into the preliminarily trained multimodal traffic perception model for verification. When the verification result meets the accuracy requirement, the trained multimodal traffic perception model based on UNITER-Yolov5 is obtained. If the verification result does not meet the accuracy requirement, the training is continued until the verification result meets the accuracy requirement, and the trained multimodal traffic perception model based on UNITER-Yolov5 is obtained.

3. The method according to claim 2, characterized in that The expression for the gradient calculation is: ， in, is the multimodal comprehensive loss function L Relative to network parameters gradient.

4. The method according to claim 1, characterized in that The step of inputting the text description data and the image data into the UNITER module of the multimodal traffic perception model for preprocessing to obtain the semantic features of the image includes: The text description data is divided into tokens to obtain the tokenized text, and then the tokenized text is mapped to a unique integer using bert-base-chinese and bert-base-cased, and then the integer is converted into an embedding vector; Then, a start tag and an end tag are added to the text sequence in the segmented text, and finally an attention mask is generated to indicate the position of the real text, thereby obtaining the preprocessed text description data. The preprocessed text description data and the image data are input into the UNITER module of the multimodal traffic perception model for preprocessing to obtain semantic features of the image.

5. The method according to claim 1, characterized in that The image data is input into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain image convolution features of K target areas, including: The image data is input into the Yolov5 module of the trained multimodal traffic perception model based on UNITER-Yolov5 for preprocessing to obtain convolutional features and K target areas. , each target area , where the target area border coordinates , is the coordinate of the upper left corner of the target area border, is the coordinate of the lower right corner of the target area border, is the probability that the target belongs to a certain category, is the category label to which the target belongs; and the convolution features of each target area in the convolution features are cropped according to the coordinates of the target area border to obtain K target areas and image convolution features of the K target areas.

6. The method according to claim 1, characterized in that The convolution function of the convolution layer is: ， in, is the element in the output preliminary image feature, is the weight of the convolution kernel, is the element in the input image convolution feature, is the bias term, is the vertical offset of the convolution kernel, is the horizontal offset of the convolution kernel, The number of channels of the input image convolution feature.

7. The method according to claim 6, characterized in that The expression of the maximum pooling operation is: ， in, is the element in the image feature output after the maximum pooling operation, located at i Row, No. j Column, No. c channels, is the element in the pooling window, Indicates the vertical offset of the pooling window; is the horizontal offset of the pooling window, and c is the number of channels.

Images