A multi-modal feature fusion traffic prediction method, device and medium

Abstract

The application discloses a multi-modal feature fusion traffic prediction method, device and medium, and relates to the technical field of traffic prediction. The method comprises the following steps: unmanned aerial vehicle video collection and preprocessing are performed on an interlaced area to obtain a target area video; a multi-target detection and tracking algorithm is used to extract traffic flow parameters from the target area video, wherein the traffic flow parameters comprise spatial average speed, spatial occupancy rate and vehicle interlacing conflict index; a deep learning model is used to process an image sequence of the target area video, and potential semantic features are extracted, wherein the deep learning model comprises a variational autoencoder; a dual attention mechanism is used to perform weighted processing on the potential semantic features; and the weighted potential semantic features and the traffic flow parameters are spliced into a multi-modal feature vector for traffic state prediction. The application improves the accuracy and robustness of interlaced area traffic state prediction through the above method, and provides more accurate data support and decision basis for dynamic traffic management of complex road sections.

Description

Technical Field

[0001] This application relates to the field of traffic prediction technology, and in particular to a traffic prediction method, device and medium based on multimodal feature fusion. Background Technology

[0002] As a key area where urban expressway mainlines and ramps merge and diverge, the weaving zone has complex traffic flow and is prone to congestion and conflicts. Accurate prediction of its traffic conditions is of great significance for dynamic traffic management and intelligent signal control.

[0003] In existing technologies, some methods attempt to fuse multimodal features. However, these methods suffer from several drawbacks: first, they fail to fully incorporate the temporal evolution of traffic flow during image feature extraction, resulting in weak correlation between latent semantic features and actual traffic conditions; second, multimodal feature fusion often employs simple concatenation without dynamically weighting key features through attention mechanisms, making it difficult to adapt to sudden changes in traffic flow in weaving areas; and third, they do not consider the uncertainty of model predictions when determining traffic state levels, leading to insufficient robustness in classification results. These issues collectively constrain the accuracy and reliability of traffic state prediction in weaving areas.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0005] In existing technologies for traffic prediction in weaving areas, multimodal feature fusion is insufficient, semantic feature extraction is weakly correlated with traffic conditions, and further prediction accuracy is low. Summary of the Invention

[0006] This application provides a traffic prediction method, device, and medium based on multimodal feature fusion, which can solve the problems of insufficient multimodal feature fusion, weak correlation between semantic feature extraction and traffic state, and low prediction accuracy in traffic prediction of weaving areas in the prior art.

[0007] In a first aspect, embodiments of this application provide a traffic prediction method based on multimodal feature fusion. The method includes: acquiring and preprocessing UAV video of weaving areas to obtain target area video; extracting traffic flow parameters from the target area video based on a multi-object detection and tracking algorithm, the traffic flow parameters including spatial average speed, space occupancy rate, and vehicle weaving conflict index; processing the image sequence of the target area video using a deep learning model to extract latent semantic features, the deep learning model including a variational autoencoder; weighting the latent semantic features using a dual attention mechanism; and concatenating the weighted latent semantic features with the traffic flow parameters to form a multimodal feature vector for traffic state prediction.

[0008] In one implementation of this application, UAV video acquisition and preprocessing are performed on the weaving area to obtain the target area video. Specifically, this includes: selecting a road segment in the weaving area, using a UAV to hover and capture one-way traffic flow video of the road segment, with the video covering the merging point to the diverging point, to obtain the original video; and performing spatial cropping on the original video based on the road structure information of the weaving area to remove irrelevant areas, thereby obtaining the target area video.

[0009] In one implementation of this application, traffic flow parameters are extracted from the target area video based on a multi-object detection and tracking algorithm. Specifically, this includes: performing multi-object detection and tracking on the target area video using a combination of YOLO and DeepSORT algorithms to extract detection boxes for vehicles in different frames, each detection box including a center point; determining the lateral and longitudinal displacements of vehicles between consecutive frames based on the positional changes of the center point; calculating the average speed of each frame based on the lateral and longitudinal displacements and the conversion factor between pixel distance and actual distance in each frame; and smoothing the spatial average speed of all frames, taking the average value as the spatial average speed.

[0010] In one implementation of this application, the method further includes: based on the vehicle's tracking trajectory in consecutive frames, counting the number of intersections of different vehicle tracking trajectories per unit time; calculating the duration and number of times the actual distance between vehicles is less than a preset safety distance; and weighting and summing the number of intersections, duration, and number of intersections to obtain a vehicle weaving conflict index.

[0011] In one implementation of this application, a deep learning model is used to process the image sequence of a target region video to extract latent semantic features. Specifically, this includes: modeling the image sequence of the target region video using ConvLSTM to output a temporal feature map; flattening the temporal feature map and inputting it into a fully connected layer to calculate the mean and variance of the latent distribution; introducing a semantic supervision mechanism at the end of the variational autoencoder, selecting random values ​​from the standard normal distribution, and combining the mean and variance to generate a latent vector; and connecting an auxiliary classifier at the end of the variational autoencoder, using the auxiliary classifier to predict the traffic state level based on the latent vector, and calculating the cross-entropy loss as a semantic supervision signal.

[0012] In one implementation of this application, the weighted latent semantic features are concatenated with traffic flow parameters to form a multimodal feature vector for traffic state prediction. Specifically, this includes: performing standardized linear mapping on the weighted latent semantic features to obtain normalized image semantic features; concatenating the image semantic features with traffic flow parameters to construct a multimodal feature vector; inputting the multimodal feature vector into an MLP for traffic state regression prediction to obtain prediction results for spatial average speed and occupancy; and inputting the prediction results into a trained random forest classifier for traffic state level discrimination.

[0013] In one implementation of this application, the method further includes: when each decision tree of the random forest outputs a classification result, calculating the out-of-bag error of the result and using the error as an uncertainty measure; and determining the level by weighting the classification results of each decision tree according to the inverse proportion of the uncertainty measure.

[0014] In one implementation of this application, image semantic features are concatenated with traffic flow parameters to construct a multimodal feature vector. Specifically, this includes: aligning the standardized image semantic features with the normalized traffic flow parameters using timestamps; employing a feature dimension expansion strategy to perform high-dimensional mapping on the traffic flow parameters, converting them into feature vectors with the same dimension as the image semantic features, and performing element-wise multiplication with the image semantic features to obtain fused interactive features; and concatenating the image semantic features and fused interactive features along the channel dimension to obtain the multimodal feature vector.

[0015] Secondly, embodiments of this application also provide a traffic prediction device based on multimodal feature fusion. The device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to: acquire and preprocess UAV video of a weaving area to obtain a target area video; extract traffic flow parameters from the target area video based on a multi-target detection and tracking algorithm, the traffic flow parameters including spatial average speed, space occupancy, and vehicle weaving conflict index; process the image sequence of the target area video using a deep learning model to extract latent semantic features, the deep learning model including a variational autoencoder; weight the latent semantic features using a dual attention mechanism; and concatenate the weighted latent semantic features with the traffic flow parameters to form a multimodal feature vector for traffic state prediction.

[0016] Thirdly, this application also provides a non-volatile computer storage medium for traffic prediction based on multimodal feature fusion, storing computer-executable instructions. These instructions are configured to: acquire and preprocess UAV video of the weaving area to obtain a target area video; extract traffic flow parameters from the target area video based on a multi-target detection and tracking algorithm, including spatial average speed, spatial occupancy, and vehicle weaving conflict index; process the image sequence of the target area video using a deep learning model to extract latent semantic features, the deep learning model including a variational autoencoder; weight the latent semantic features using a dual attention mechanism; and concatenate the weighted latent semantic features with the traffic flow parameters to form a multimodal feature vector for traffic state prediction.

[0017] This application provides a multimodal feature fusion-based traffic prediction method, device, and medium that overcomes the coverage limitations of ground detectors, ensuring the comprehensiveness of traffic flow parameters such as spatial average speed, spatial occupancy, and vehicle weaving conflict index. It employs a variational autoencoder combining ConvLSTM and a semantic supervision mechanism to extract latent semantic features, strengthening the semantic correlation between image features and traffic states. Through a dual attention mechanism, latent features are dynamically weighted, and a multimodal feature vector is constructed via refined steps, achieving the highlighting of key features and deep fusion of cross-modal information. Finally, a random forest classifier with uncertainty estimation is combined to improve the reliability of traffic state level discrimination. In summary, this method significantly improves the accuracy and robustness of traffic state prediction in weaving areas, providing more accurate data support and decision-making basis for dynamic traffic management in complex road sections. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A flowchart illustrating a traffic prediction method based on multimodal feature fusion, provided as an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the internal structure of a traffic prediction device based on multimodal feature fusion, provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application provides a traffic prediction method, device, and medium based on multimodal feature fusion, which solves the problems of insufficient multimodal feature fusion, weak correlation between semantic feature extraction and traffic state, and low prediction accuracy in traffic prediction in weaving areas in the prior art.

[0023] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart illustrating a traffic prediction method based on multimodal feature fusion, as provided in an embodiment of this application. Figure 1As shown in the embodiment of this application, a traffic prediction method based on multimodal feature fusion specifically includes the following steps:

[0025] Step 10: Perform UAV video acquisition and preprocessing on the interlacing area to obtain video of the target area;

[0026] As an optional embodiment, UAV video acquisition and preprocessing are performed on the interlacing area to obtain video of the target area, which may specifically include:

[0027] Step 101: Select the weaving section of the road and use a drone to hover and capture one-way traffic video of the section, covering the merging point to the diverging point, to obtain the original video;

[0028] In this step, a weaving section of an urban expressway is selected. This section includes a continuous structure of merging and diverging traffic between the mainline and ramps. A drone is used to hover and capture a one-way traffic flow video of this section. The video range should cover the entire area from the merging point to the diverging point. The original video collected is denoted as V. raw .

[0029] Step 102: Based on the road structure information of the interlacing area, perform spatial cropping on the original video to remove irrelevant areas and obtain the target area video.

[0030] In this step, based on the road structure information of the weaving area, the original video V is processed. raw Spatial cropping is performed to remove irrelevant areas, retaining only the image content of the main passage area in the interlacing region. The cropped video is denoted as the region of interest video V. roi .

[0031] Step 20: Extract traffic flow parameters from the target area video based on the multi-target detection and tracking algorithm. The traffic flow parameters include spatial average speed, space occupancy rate, and vehicle weaving conflict index.

[0032] As an optional embodiment, extracting traffic flow parameters from video of the target area based on a multi-target detection and tracking algorithm may specifically include:

[0033] Step 201: Combine YOLO and DeepSORT algorithms to perform multi-target detection and tracking on the target area video, and extract the detection boxes of the vehicle in different frames. The detection boxes include the center point.

[0034] In this step, the YOLO and DeepSORT algorithms are combined to analyze the region of interest video V. roi The system performs multi-target detection and tracking on vehicles and extracts the coordinate information of the detection boxes of vehicles in different frames.

[0035] Step 202: Determine the lateral and longitudinal displacements of the vehicle between consecutive frames based on the changes in the position of the center point;

[0036] Step 203: Calculate the average velocity of each frame based on the horizontal and vertical displacements and the conversion factor between the pixel distance in each frame and the actual distance;

[0037] In this step, the actual distance the vehicle travels between consecutive frames is estimated based on the positional changes of the detection box's center point, thus calculating the spatial average velocity for each frame. The calculation formula is as follows:

[0038]

[0039] In the formula: s f Let f be the spatial average velocity of the f-th frame. Let f be the number of vehicles in the f-th frame of the image. represents the horizontal and vertical displacement of the center point of vehicle i detection box from frame f to frame f+1, respectively, and k is the conversion factor between pixel distance and actual distance.

[0040] Simultaneously, the space occupancy rate is calculated based on the pixel proportion of the lane-occupied area in the image. The formula is as follows:

[0041]

[0042] In the formula: o f Where is the frame occupancy, n is the number of lanes in the video, and L is the lane pixel length. and These are the x-coordinates of the left and right boundaries of the vehicle i detection box in frame f.

[0043] Step 204: Smooth the spatial average velocity of all frames and take the average value as the spatial average velocity.

[0044] In this step, the spatial average velocity and spatial occupancy of all frames per second are smoothed, and their average value is taken as the traffic flow parameter representing each second.

[0045]

[0046] In the formula: s t and o t These represent the average spatial velocity and spatial occupancy rate at second t, respectively.

[0047] As an optional embodiment, the method may further include: step 205: based on the vehicle's tracking trajectory in consecutive frames, count the number of intersections of different vehicle tracking trajectories per unit time; step 206: calculate the duration and number of times the actual distance between vehicles is less than a preset safety distance, and sum the number of intersections, duration and number of times by weighting to obtain the vehicle weaving conflict index.

[0048] This step focuses on the lateral interaction behavior of vehicles. In the weaving zone, lane changes, overtaking, and other actions cause the movement trajectories of different vehicles to intersect. The more frequent the intersections, the more frequent the lateral interference between vehicles, and the higher the potential risk of scrapes and congestion. By statistically analyzing the number of intersections per unit time, the dynamic chaos level of the weaving zone can be directly reflected, which is one of the core indicators for measuring the likelihood of conflict. The next step focuses on the longitudinal and lateral distance risks of vehicles. The preset safe distance is a safety threshold determined based on road conditions and vehicle speed. When the actual distance between vehicles is less than this threshold, the risk of collision increases significantly. Statistically analyzing the duration and frequency of such violations can accurately characterize the severity of the conflict. For example, short-term, occasional close proximity may be normal following, while prolonged or frequent close proximity indicates high risk.

[0049] By weighting and integrating the number of trajectory intersections with the duration and frequency of violations of safe distance, the resulting "vehicle weaving conflict index" can comprehensively quantify the conflict risk level of weaving areas. This index compensates for the limitations of spatial average speed and space occupancy. Traditional parameters can only reflect macroscopic traffic flow and density, while this index can capture microscopic interactive danger states.

[0050] Therefore, embodiments of this application may further include using the conflict index as an important component of traffic flow parameters, and participating in multimodal feature fusion together with spatial average speed and spatial occupancy. By incorporating micro-conflict risks into the feature vector, the prediction model can simultaneously perceive macro-traffic flow status and micro-hazard trends, thereby more accurately predicting the evolution of traffic status in weaving areas.

[0051] Step 30: Use a deep learning model to process the image sequence of the target region video and extract latent semantic features. The deep learning model includes a variational autoencoder.

[0052] As an optional embodiment, a deep learning model is used to process the image sequence of the target region video to extract latent semantic features. Specifically, it may include: Step 301: Modeling the image sequence of the target region video using ConvLSTM to output a temporal feature map; Step 302: Flattening the temporal feature map and inputting it into a fully connected layer to calculate the mean and variance of the latent distribution.

[0053] In this step, ConvLSTM is used to analyze the video V of the region of interest. roi The image sequence is modeled to output temporal features per second. These temporal features are then flattened and input into two fully connected layers to calculate the mean and variance of the latent distribution. Based on the VAE model, latent features are learned and generated. The specific calculation formula is as follows:

[0054] μ t =W μ ·flatten(gt )+b μ

[0055]

[0056] In the formula: For time series feature maps, d C d H d W These represent the number of channels, height, and width, respectively. `flatten(.)` means flattening the 3D feature map into a vector according to channel priority. `Wμ,Wμ` and `bμ,bσ` are learnable linear transformation parameters.

[0057] Step 303: Based on the introduction of a semantic supervision mechanism at the end of the variational autoencoder, random values ​​are selected from the standard normal distribution, and the latent vector is generated by combining the mean and variance;

[0058] In this step, the latent vector is then generated from the standard normal distribution using a reparameterization technique, calculated as follows:

[0059] z t =μ t +σ t ·ε t

[0060] In the formula: ε represents the latent features of the image frame at second t. t ~N(0,I d ) indicates from d z Noise sampled from a standard normal distribution, I d It is the unit covariance matrix.

[0061] Step 304: Connect an auxiliary classifier to the end of the variational autoencoder. The auxiliary classifier predicts the traffic state level based on the latent vector and calculates the cross-entropy loss as a semantic supervision signal.

[0062] In this step, a semantic supervision mechanism is introduced at the end of the encoder. This mechanism is implemented by connecting an auxiliary classifier module. This classifier takes latent features as input, predicts the traffic state level, and calculates the cross-entropy loss as the semantic supervision signal. The calculation formula is as follows:

[0063]

[0064] In the formula: L sem Let C represent the semantic supervision loss, and C be the number of traffic state categories. For the one-hot encoded label of class i, W cls With b cls The softmax function is a learnable parameter that assists the classifier. It outputs the probability that the current frame belongs to each category.

[0065] Step 40: Employ a dual attention mechanism to weight the latent semantic features;

[0066] In this step, a temporal attention mechanism is introduced to perform context-weighted updates on latent image features, enhancing the model's sensitivity to temporal changes. The specific calculation formula is as follows:

[0067]

[0068] In the formula: This represents the latent features of an image after time-attention enhancement. The query vector constructed for the current frame. The key vector and value vector constructed for the j-th frame of history, d k Let T be the dimension of the attention subspace, and T be the length of the historical window used for attention modeling.

[0069] (2) Then, a feature dimension attention mechanism is introduced to weight the importance of each dimension of the feature representation. The calculation is as follows:

[0070] e t =W γ ·z′ t +b γ

[0071]

[0072] z″ t =γ t ☉z′ t

[0073] In the formula: These are the learnable linear transformation parameters in the feature dimension attention mechanism. The scoring vector represents the feature dimension, and the attention weights are y. t,m ∈(0,1) represents the importance of the m-th dimension, satisfying the normalization constraint. Combine the attention weights of all dimensions into a vector. For latent feature vectors Perform dimension-wise weighting. ⊙ indicates that element-wise multiplication is used here, ultimately yielding the weighted latent feature representation. This means that while preserving information about traffic evolution trends, the model can enhance its ability to focus on key dimensions.

[0074] Step 50: Concatenate the weighted latent semantic features with traffic flow parameters to form a multimodal feature vector for traffic state prediction.

[0075] As an optional implementation, the weighted latent semantic features are concatenated with traffic flow parameters to form a multimodal feature vector for traffic state prediction. Specifically, this may include:

[0076] Step 501: Perform standardized linear mapping on the weighted latent semantic features to obtain normalized image semantic features;

[0077] In this step, based on the attention-weighted latent representation of the feature dimensions, a standardized linear mapping operation is introduced to normalize the features, calculated as follows:

[0078]

[0079] This represents the latent feature representation of an image frame after processing with an improved VAE encoder and attention mechanism. For learnable linear transformation parameters, Let represent the mean and variance of the current sample in each dimension, respectively, and ξ be a stability term to prevent division by zero errors, which is set to 10 in this paper. -5 , This represents the normalized semantic features of the image.

[0080] Step 502: Concatenate the image semantic features with traffic flow parameters to construct a multimodal feature vector;

[0081] Subsequently, the latent features of the image Traffic flow parameters Concatenate the modal feature vectors:

[0082]

[0083] In the formula: F t Let d be the multimodal feature vector at second t, with dimension d. z +2.

[0084] As an optional embodiment, image semantic features are concatenated with traffic flow parameters to construct a multimodal feature vector. Specifically, this may include: Step 5021: Timestamp alignment of the standardized image semantic features and the normalized traffic flow parameters; Step 5022: Using a feature dimension expansion strategy, the traffic flow parameters are mapped to a high dimension, converted into a feature vector with the same dimension as the image semantic features, and element-wise multiplication is performed with the image semantic features to obtain fused interactive features; Step 5023: The image semantic features and fused interactive features are concatenated along the channel dimension to obtain a multimodal feature vector.

[0085] In this step, the embodiments of this application may further include: high-dimensional mapping solves the problem of modal dimension mismatch. Traffic flow parameters are low-dimensional values, while image semantic features are high-dimensional vectors. Direct concatenation would dilute the low-dimensional features. High-dimensional mapping converts traffic flow parameters into vectors of the same dimension as image features, allowing the two modalities to be fused in the same dimensional space. Element-level multiplication strengthens cross-modal correlation. Calculating the product of image semantic features and high-dimensional mapped traffic flow features dimension by dimension highlights the dimension with higher values ​​in both features. For example, a certain dimension may represent dense vehicles in the image but extremely low speed in the traffic flow. After multiplication, the feature value of that dimension will be amplified, thereby strengthening the strongly indicative correlation features of traffic conditions such as congestion and filtering irrelevant or weakly correlated information. This preserves the unique information in image semantic features that is not covered by traffic flow parameters and incorporates the fused features generated by the interaction of the two modalities.

[0086] Step 503: Input the multimodal feature vector into the MLP to perform traffic state regression prediction and obtain the prediction results of spatial average speed and occupancy rate;

[0087] In this step, the multimodal feature vectors are input into a multilayer perceptron (MLP) for traffic state prediction. The MLP consists of a two-layer linear mapping structure. The first layer performs nonlinear feature extraction, and the second layer outputs the predicted spatial average speed and occupancy rate, as shown in the following formula:

[0088]

[0089] For the predicted results, Let W1, W2, b1, and b2 represent the ReLU activation function, and W1, W2, b1, and b2 represent the weight matrix and bias term of the two linear transformations in the prediction module, respectively.

[0090] Step 504: Input the prediction results into the trained random forest classifier to determine the traffic state level.

[0091] In this step, after prediction, a trained random forest classifier is used to determine the traffic state. The random forest consists of P decision trees, each making independent predictions for the input samples. The final classification result is determined by majority vote, calculated as follows:

[0092]

[0093] In the formula: Let represent the predicted traffic state category at second t (classifier output), c be the candidate category, C be the set of all categories, and 8(.) be the equality judgment function used to simulate voting and counting behavior.

[0094] As an optional embodiment, the method may further include: step 505: when each decision tree of the random forest outputs a classification result, calculate the out-of-bag error of the result and use the error as an uncertainty measure; step 506: determine the level by weighting the classification results of each decision tree according to the inverse proportion of the uncertainty measure.

[0095] In this step, the random forest consists of multiple independently trained decision trees. Each tree randomly selects a subset of samples during training; the samples not selected are the out-of-bag data, which can be understood as test data that the tree has not seen before. By validating the tree's predictions using the out-of-bag data and calculating the error, the reliability of the tree's predictions for unknown data can be measured. The smaller the error, the stronger the tree's generalization ability, the more stable the prediction, and the lower the uncertainty; conversely, the larger the error, the higher the uncertainty. Each tree is scored to quantify its credibility, avoiding the limitation of equal weights for all trees in traditional random forests.

[0096] Based on the uncertainty measure obtained in step 505, weights are assigned to each tree: the smaller the error, the larger the weight; the larger the error, the smaller the weight. The final traffic state level is determined by a weighted vote of the classification results of all trees according to these weights.

[0097] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a traffic prediction device for multimodal feature fusion, the structure of which is as follows: Figure 2 As shown.

[0098] Figure 2 This is a schematic diagram of the internal structure of a traffic prediction device based on multimodal feature fusion, provided as an embodiment of this application. Figure 2 As shown, the device includes:

[0099] At least one processor 201;

[0100] And a memory 202 that is communicatively connected to at least one processor;

[0101] The memory 202 stores instructions executable by at least one processor, which are executed by at least one processor 201 to enable at least one processor 201 to:

[0102] Unmanned aerial vehicle (UAV) video was collected and preprocessed in the weaving area to obtain video of the target area. Traffic flow parameters, including spatial average speed, space occupancy rate, and vehicle weaving conflict index, were extracted from the video of the target area based on a multi-object detection and tracking algorithm. The image sequence of the video of the target area was processed using a deep learning model, including a variational autoencoder, to extract latent semantic features. A dual attention mechanism was used to weight the latent semantic features. The weighted latent semantic features and traffic flow parameters were concatenated to form a multimodal feature vector for traffic state prediction.

[0103] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for traffic prediction based on multimodal feature fusion, storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0104] Unmanned aerial vehicle (UAV) video was collected and preprocessed in the weaving area to obtain video of the target area. Traffic flow parameters, including spatial average speed, space occupancy rate, and vehicle weaving conflict index, were extracted from the video of the target area based on a multi-object detection and tracking algorithm. The image sequence of the video of the target area was processed using a deep learning model, including a variational autoencoder, to extract latent semantic features. A dual attention mechanism was used to weight the latent semantic features. The weighted latent semantic features and traffic flow parameters were concatenated to form a multimodal feature vector for traffic state prediction.

[0105] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0106] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0113] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A traffic prediction method based on multimodal feature fusion, characterized in that, The method includes: The video of the target area is obtained by drone video acquisition and preprocessing in the interlacing area. Traffic flow parameters are extracted from the video of the target area based on a multi-target detection and tracking algorithm. The traffic flow parameters include spatial average speed, spatial occupancy rate, and vehicle weaving conflict index. The image sequence of the video of the target region is processed using a deep learning model to extract latent semantic features. The deep learning model includes a variational autoencoder. The latent semantic features are weighted using a dual attention mechanism; The weighted latent semantic features are concatenated with the traffic flow parameters to form a multimodal feature vector for traffic state prediction, specifically including: The weighted latent semantic features are subjected to standardized linear mapping to obtain normalized image semantic features; The image semantic features are concatenated with the traffic flow parameters to construct a multimodal feature vector; The multimodal feature vectors are input into the MLP for traffic state regression prediction to obtain the predicted results of spatial average speed and occupancy. The prediction results are input into a trained random forest classifier to determine the traffic state level.

2. The traffic prediction method based on multimodal feature fusion according to claim 1, characterized in that, The process of acquiring and preprocessing UAV video of the interlacing area to obtain video of the target area specifically includes: Select a section of road in the weaving zone and use a drone to hover and capture video of the one-way traffic flow in that section. The video covers the merging point to the diverging point to obtain the original video. Based on the road structure information of the interlacing area, the original video is spatially cropped to remove irrelevant areas, thus obtaining the target area video.

3. The traffic prediction method based on multimodal feature fusion according to claim 1, characterized in that, The extraction of traffic flow parameters from the target area video based on the multi-target detection and tracking algorithm specifically includes: The YOLO and DeepSORT algorithms are combined to perform multi-target detection and tracking on the target area video, and the detection boxes of vehicles in different frames are extracted. The detection boxes include the center point. Based on the position change of the center point, determine the lateral and longitudinal displacement of the vehicle between consecutive frames; The average speed of each frame is calculated based on the horizontal and vertical displacements and the conversion factor between the pixel distance in each frame and the actual distance. The spatial average velocity of all frames is smoothed, and the average value is taken as the spatial average velocity.

4. The traffic prediction method based on multimodal feature fusion according to claim 3, characterized in that, The method further includes: Based on the vehicle's tracking trajectory in the consecutive frames, the number of intersections of different vehicle tracking trajectories per unit time is counted. The duration and number of times the actual distance between vehicles is less than the preset safety distance are calculated. The number of intersections, duration, and number of intersections are weighted and summed to obtain the vehicle weaving conflict index.

5. The traffic prediction method based on multimodal feature fusion according to claim 1, characterized in that, The process of using a deep learning model to process the image sequence of the target region video and extract latent semantic features specifically includes: The target region video image sequence is modeled using ConvLSTM to output a temporal feature map; The temporal feature map is flattened and then input into a fully connected layer to calculate the mean and variance of the latent distribution. Based on the semantic supervision mechanism introduced at the end of the variational autoencoder, random values ​​are selected from the standard normal distribution, and latent vectors are generated by combining the mean and variance. An auxiliary classifier is connected to the end of the variational autoencoder. The auxiliary classifier predicts the traffic state level based on the latent vector and calculates the cross-entropy loss as a semantic supervision signal.

6. The traffic prediction method based on multimodal feature fusion according to claim 1, characterized in that, The method further includes: When each decision tree of the random forest outputs a classification result, the out-of-bag error of the result is calculated, and the error is used as a measure of uncertainty. The classification results of each decision tree are used to determine the level by weighted voting inversely proportional to the uncertainty metric.

7. The traffic prediction method based on multimodal feature fusion according to claim 1, characterized in that, The step of concatenating the image semantic features with the traffic flow parameters to construct a multimodal feature vector specifically includes: The standardized image semantic features are aligned with the normalized traffic flow parameters using timestamps. A feature dimension expansion strategy is adopted to perform high-dimensional mapping on the traffic flow parameters, converting them into feature vectors with the same dimension as the image semantic features, and then performing element-wise multiplication with the image semantic features to obtain fused interactive features. The image semantic features and fused interactive features are concatenated along the channel dimension to obtain a multimodal feature vector.

8. A traffic prediction device based on multimodal feature fusion, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Perform the steps of the traffic prediction method based on multimodal feature fusion as described in any one of claims 1-7.

9. A non-volatile computer storage medium for traffic prediction based on multimodal feature fusion, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Perform the steps of the traffic prediction method based on multimodal feature fusion as described in any one of claims 1-7.

Images