Highway traffic flow prediction method based on structured semantics and expert network

Abstract

The application relates to a highway traffic flow prediction method based on structured semantics and expert networks, which first converts a road condition image into low-bandwidth structured text at the edge of a highway and uploads the text to the cloud; the cloud associates multiple-source heterogeneous data according to the structured text; then, a general expert network is dynamically selected for feature extraction according to the text semantics through a gated routing network, and global text and video image features are extracted by using a global text and video image expert network, and multiple features are weighted and fused; then, the text semantics and the fused features are input into a prediction decoder to obtain a prediction result, and the model is trained through a joint loss; finally, the trained prediction model is used for highway traffic flow prediction. The application can reduce data transmission bandwidth, realize adaptive fusion of multiple-source heterogeneous information, and effectively improve the accuracy and reliability of highway traffic flow prediction in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of traffic flow prediction and control technology, specifically to a method for predicting highway traffic flow based on structured semantics and expert networks. Background Technology

[0002] Traffic flow prediction on highways is crucial for the intelligent implementation of highway traffic management. Accurate prediction helps optimize road network scheduling, alleviate congestion, and improve travel efficiency. Traditional prediction schemes mainly rely on two types of data: first, full video streams transmitted in real time from roadside monitoring cameras, analyzed in the cloud; and second, cross-sectional traffic flow data collected by fixed sensors such as microwave radar and coils. However, existing technologies have significant drawbacks: 1. Full video transmission puts enormous pressure on communication bandwidth and cloud storage. In edge road sections with poor network signals, it is difficult to guarantee the real-time performance and stability of data transmission, affecting the timeliness of predictions. 2. Traditional sensors can only provide numerical information such as traffic flow and speed, and cannot perceive rich visual and environmental semantic information such as "road icing," "traffic accidents," and "vehicle queue length," resulting in insufficient model perception of complex traffic conditions. 3. Multi-source data such as meteorological, holiday, POI (Point of Interest), and historical traffic flow vary greatly in format, dimension, and semantics. Existing models struggle to perform effective deep semantic fusion, limiting further improvements in prediction accuracy.

[0003] Therefore, there is an urgent need for a new method for predicting highway traffic flow that can reduce transmission burden, integrate multi-source heterogeneous information, and make full use of environmental semantics. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a highway traffic flow prediction method based on structured semantics and expert networks. This method converts high-bandwidth video into low-bandwidth structured text at the edge and introduces a hybrid expert network in the cloud for dynamic routing and multimodal feature extraction and fusion, thereby achieving high-precision, adaptive highway traffic flow prediction under low bandwidth requirements.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for predicting highway traffic flow based on structured semantics and expert networks, comprising the following steps:

[0006] Step S1: At the edge of the highway, acquire road condition images, generate structured text describing road condition information using a pre-trained multimodal model, and then upload it to the cloud;

[0007] Step S2: In the cloud, acquire multi-source heterogeneous data of structured text-related road segments, and convert the multi-source heterogeneous data into a semantic vector sequence with unified feature dimensions;

[0008] Step S3: Based on the semantic vector sequence of structured text, the general expert network required for this highway traffic flow prediction is dynamically selected through a gated routing network, and the required multi-source heterogeneous data is input into each general expert network to obtain the semantic feature sequence of heterogeneous multi-source information; at the same time, the text data and video image data involved in the multi-source heterogeneous data are input into the global text feature extraction expert network and the global video image feature extraction expert network respectively to obtain global text features and global video image features;

[0009] Step S4: Perform weighted fusion of the semantic feature sequence of multi-source information, global text features, and global video image features to obtain a fused heterogeneous information feature vector sequence;

[0010] Step S5: Input the structured text semantic vector sequence and the fused heterogeneous information feature vector sequence into the prediction decoder for decoding to obtain the highway traffic flow prediction result; construct the total loss based on the error between the prediction result and the actual traffic flow and the auxiliary loss of the gated routing network, train the overall model, and obtain the trained prediction model.

[0011] Step S6: Use the trained prediction model to predict highway traffic flow and generate corresponding prediction results.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] Step S11: Capture a frame of road condition image at regular intervals using edge devices equipped with high-performance cameras and computing units deployed on the highway. After standardization processing, a standardized road condition image is obtained. ;

[0014]

[0015] in, For normalized functions;

[0016] Step S12: Convert the standardized road condition image Input a pre-trained multimodal model deployed in a computing unit to generate structured text describing road condition information. And upload it to the cloud;

[0017]

[0018] in, This is a pre-trained multimodal model for generating structured text based on road condition images. Prompt words generated for pre-defined structured guidance text.

[0019] Furthermore, step S2 specifically includes the following steps:

[0020] Step S21: In the cloud, obtain multi-source heterogeneous data sequences of structured text-related road segments. :

[0021]

[0022] ,

[0023] in, This is a function for obtaining multi-source heterogeneous data sequences based on road segment location information. To associate the location information of road segments with structured text, Let L be the i-th multi-source heterogeneous data of the structured text associated road segment, and L be the number of multi-source heterogeneous data.

[0024] Step S22: Transform the acquired multi-source heterogeneous data sequences into a set of semantic vector sequences with unified feature dimensions. ,in Let i be the semantic vector sequence of the i-th multi-source heterogeneous data;

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in, This is a function that transforms text data into a sequence of semantic vectors. This is a function that transforms image data into a sequence of semantic vectors. This is a function that transforms video data into a sequence of semantic vectors. This indicates word embedding operations. This indicates that the input data is positionally encoded. This indicates that the image data is divided into blocks. This indicates that convolution operations are performed on image blocks. This indicates the extraction of keyframes from video data. This represents the Mth word vector in a sequence of semantic vectors from multi-source heterogeneous data.

[0031] Furthermore, step S3 specifically includes the following steps:

[0032] Step S31: Generate structured text from the highway edge side Encoding is performed to obtain a sequence of structured text semantic vectors. ;

[0033]

[0034]

[0035] in, This is the nth word vector in the structured text semantic vector sequence;

[0036] Step S32: Obtain the structured text semantic vector sequence Input gated routing network The indices and weights of the top K general expert networks, the weights of the two global vision text and video image feature extraction expert networks, and the auxiliary loss of the gated routing network were calculated.

[0037]

[0038] in, This is the index sequence of the top K general expert networks selected by the gated routing network based on the structured text semantic vector sequence. For the selected top K general expert networks, Weights for the expert network are extracted to extract global text features. The weights of the expert network for global video image feature extraction are given. The auxiliary loss in a gated routing network is calculated as follows:

[0039]

[0040] in, Let j be the call frequency of the j-th general expert network. is the average routing probability for the j-th general expert network in a training batch, where N is the total number of general expert networks;

[0041] Step S33: Based on the index sequence Select the corresponding general expert network and index the sequence. Mapping to sequence The required multi-source heterogeneous data is input into the topK general expert networks for feature extraction, thereby obtaining the semantic features of heterogeneous multi-source information.

[0042]

[0043] in, This represents the semantic feature sequence of heterogeneous multi-source information extracted by the top K general expert networks; This represents the semantic features of the heterogeneous multi-source information extracted by the k-th general expert network; This indicates that feature extraction is performed using the k-th general expert network; Represents a set of semantic vector sequences from multi-source heterogeneous data. The set of multi-source heterogeneous data semantic vector sequences extracted for the k-th general expert network. ;

[0044] All text-type data used by the general expert network in this route selection Input Global Text Feature Extraction Expert Network Feature extraction is performed to obtain the global text features of this route. ;

[0045]

[0046] All video and image data used by the general expert network in this routing selection Input global video image feature extraction expert network Feature extraction is performed to obtain the global video image features of this route. ;

[0047] .

[0048] Furthermore, in step S33, each general expert network It includes an L1 attention layer, and each layer contains a layer normalization module, a multi-head self-attention module, and a multilayer perceptron module;

[0049]

[0050]

[0051]

[0052] in, This represents the input to the c-th attention layer. MSA represents the first normalization module in the c-th attention layer, and MSA represents the multi-head self-attention module in the c-th attention layer. Indicates input The intermediate features output after processing by the first-layer normalization module and the multi-head self-attention module. This represents the output feature of the c-th attention layer. The second normalization module in the c-th attention layer is represented by MLP, and MLP represents the multilayer perceptron module in the c-th attention layer; the input is a general expert network. The data is processed sequentially through the L1 attention layer to obtain the semantic features of the heterogeneous multi-source information. ;

[0053] Global Text Feature Extraction Expert Network It contains L2 recursive evolution layers, and each recursive evolution layer is defined as follows:

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] in, For time steps Input features, For time steps The implicit memory tensor, For time steps The implicit memory tensor, It represents the Hadamah accumulation. An adaptive forgetting gate operator used to calculate the historical information retention rate under time-varying parameters. An intensity operator is injected into the information to control the intensity at which the current input features are written into the memory tensor. The input projection operator is used to reconstruct the semantic features at the current time step. , For learnable time-scale projection weights, For time scale offset. For predefined structured state parameters, For smoothing nonlinear activation functions, For input projection weights, For input bias, To output the projected weights, For output bias, the input of the first recursive evolution layer For expert network Input , No. Output of the recursive evolution layer For expert network Output ;

[0060] Global Video Image Feature Extraction Expert Network It includes an L3 spline topological association layer, and each spline topological association layer uses attention aggregation computation based on B-spline basis functions for feature extraction:

[0061]

[0062]

[0063]

[0064] in, and These are the input and output feature tensors within the layer, respectively. For via nonlinear mapping The generated query tensor, key tensor, and value tensor For tensor Dimensions The first eigenvector of the output eigenvector for nonlinear mapping One portion, The first normalized input feature vector is the... One portion, Let be the nonlinear mapping function on the connecting edge. For the p-th order B-spline basis function, Let be the control point coefficients learned during network training, representing the th control point coefficients learned using the p-th order B-spline basis function on the normalized input feature vector. The i-th component is calculated to obtain the i-th eigenvector of the output feature vector. The control point coefficients along this calculation path for each component, where i is the input feature. The index of the i-th vector component, where j is the output feature. The index of the j-th vector component, where p is the index of the p-th B-spline basis function. The spline mesh granularity is used to approximate the complex nonlinear spatiotemporal evolution of traffic flow videos by adjusting the control point coefficients. Based on the basic weight parameter matrix, for Activation function For layer normalization; input to the first spline topological correlation layer For expert network Input , No. Output of the spline topological association layer for .

[0065] Furthermore, step S4 specifically includes the following steps:

[0066] Step S41: Fuse the semantic features of heterogeneous multi-source information extracted by the top K general expert networks, the global text features extracted by the global text feature extraction expert network, and the global video image features extracted by the global video image feature extraction expert network to obtain a feature vector sequence. ;

[0067]

[0068] in, Let be the weight of the k-th general expert network;

[0069] Step S42: Combine the fused feature vector sequence After the input to the feedforward neural network undergoes nonlinear transformation, a sequence of fused heterogeneous information feature vectors S for traffic flow prediction is obtained.

[0070]

[0071] in, This is the first feedforward neural network. for Activation function This is the second feedforward neural network.

[0072] Furthermore, step S5 specifically includes the following steps:

[0073] Step S51: Transform the structured text semantic vector sequence and fusion of heterogeneous information feature vector sequences Input Predictor Decoder This yields the final predicted vector sequence. ;

[0074]

[0075]

[0076] in, This is the vector of the nth word in the predicted vector sequence;

[0077] Predictive Decoder Depend on It consists of stacked decoding layers, each of which contains a mask multi-head self-attention module, a multi-head cross-attention module, a layer normalization module, and a multi-layer perception module.

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] in, For the input of the h-th decoding layer, For the first layer normalization module of the h-th decoding layer, The result is the input of the h-th decoding layer after being normalized by the normalization module of the first input layer. Let be the first key projection matrix of the h-th decoding layer. Let be the first key feature matrix of the h-th decoding layer. Let be the projection matrix of the first value of the h-th decoding layer. Let be the first value feature matrix of the h-th decoding layer. This is the first query projection matrix of the h-th decoding layer. Let H be the first query feature matrix of the h-th decoding layer, where H is the number of stacked decoding layers.

[0084]

[0085] in, For the mask multi-head self-attention module of the h-th decoding layer, The output of the mask multi-head self-attention module for the h-th decoding layer and the input of the h-th decoding layer. The result of performing residual connection;

[0086]

[0087]

[0088]

[0089] in, This is the normalization module for the second layer of the h-th decoding layer. This is the second query projection matrix of the h-th decoding layer. Let S be the second query feature matrix of the h-th decoding layer, and let S be the sequence of fused heterogeneous information feature vectors used for traffic flow prediction. Let be the second key projection matrix of the h-th decoding layer. This is the second key feature matrix of the h-th decoding layer. The projection matrix of the second value of the h-th decoding layer. This is the second value feature matrix of the h-th decoding layer;

[0090]

[0091] in, For the multi-head cross-attention module of the h-th decoding layer, The output of the multi-head cross-attention module of the h-th decoding layer is concatenated with the residual and then the output of the mask multi-head self-attention module. The result of performing residual connection;

[0092]

[0093] in, This is the normalization module for the third layer of the h-th decoding layer. For the h-th decoding layer of the multi-layer perception module, This is the output of the h-th decoding layer;

[0094]

[0095] in, This is the final predicted vector sequence. For the last layer normalization module of the predictive decoder, This is the last multilayer perceptron module in the prediction decoder. This is the output of the H-th decoding layer, where H is the number of stacked predictive decoder layers;

[0096] Step S52: Use the final predicted vector sequence Traffic flow is predicted to obtain highway traffic flow prediction results;

[0097]

[0098] in, This is a multilayer perceptron module, where Y represents the predicted highway traffic flow, and t is the prediction time step. Number of lanes;

[0099] Step S53: Calculate the total loss of the overall model. ;

[0100]

[0101]

[0102] in, The mean squared error loss for a training batch. For the i-th sample in the batch, This refers to the highway traffic flow predicted using the i-th sample in the batch. The number of samples in a training batch;

[0103] Step S54: Total loss based on the overall model Perform backpropagation, update the model parameters, and train to obtain the final prediction model.

[0104] Furthermore, step S6 specifically includes the following steps:

[0105] Step S61: Periodically acquire multi-source heterogeneous data from highway monitoring points and store it in the database;

[0106] Step S62: Capture real-time road condition images at regular intervals on the edge of the highway, generate structured text on the edge, and upload it to the cloud;

[0107] Step S63: Retrieve the latest multi-source heterogeneous data set from the database. ,in This is the latest L-th multi-source heterogeneous data from the monitoring point;

[0108] Step S64: Combine multi-source heterogeneous data sets The predicted traffic flow results for highways are obtained by combining the predicted model with structured text input. .

[0109] The present invention also provides a computer device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which implement the above-described method when executed by the processor.

[0110] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the above-described method.

[0111] Compared with the prior art, the present invention has the following beneficial effects:

[0112] (1) This invention greatly reduces the network transmission bandwidth requirement while preserving the rich semantic information of the scene. By using lightweight image-text conversion technology at the edge, the video stream that originally required high bandwidth transmission is converted into KB-level structured text. This method breaks the dependence of traditional video backhaul on high bandwidth, enabling low-latency and high-stability real-time data backhaul even on highway edge sections with poor signal. At the same time, compared with traditional sensors that only transmit traffic flow values, the text can preserve key environmental details such as accidents, weather, and queuing conditions.

[0113] (2) This invention constructs a hybrid expert system based on semantic routing and global routing information perception mechanisms, achieving adaptive and accurate prediction of normal and sudden traffic conditions. The system innovatively introduces two experts focused on global routing information: a global text expert and a global video image expert. Combined with an intelligent gated routing network, the system can dynamically calculate the fusion weights of general sparse experts and dedicated global experts based on real-time traffic conditions. Under normal conditions, it emphasizes semantic experts such as conventional historical traffic flow and road network information; in severe weather or sudden accidents, it automatically switches to semantic experts focusing on environmental impact. This dynamic mechanism effectively solves the problem of prediction failure of traditional models under non-stationary traffic flow, significantly improving prediction accuracy in complex scenarios. Attached Figure Description

[0114] Figure 1 This is a flowchart illustrating the implementation of the highway traffic flow prediction method based on structured semantics and expert networks provided in this embodiment of the invention.

[0115] Figure 2 This is a schematic diagram illustrating the use of a trained prediction model to predict highway traffic flow in an embodiment of the present invention. Detailed Implementation

[0116] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0117] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0118] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0119] like Figure 1 As shown in the figure, this embodiment provides a method for predicting highway traffic flow based on structured semantics and expert networks, which is implemented according to the following steps.

[0120] Step S1: At the edge of the highway, acquire road condition images, generate structured text describing the road condition information using a pre-trained multimodal model, and then upload it to the cloud. The implementation steps are as follows.

[0121] Step S11: Capture a frame of road condition image every 10 minutes using edge devices equipped with high-performance cameras and computing units deployed on the highway. After standardization processing, a standardized road condition image is obtained. ;

[0122]

[0123] in, For normalization functions, , The tensor dimension of the normalized input image.

[0124] Step S12: Convert the standardized road condition image Input a pre-trained multimodal model deployed in a computing unit to generate structured text describing road condition information. ;

[0125]

[0126] in, A pre-trained multimodal model for generating structured text based on road condition images; Prompt words generated for pre-defined structured guidance text. Prompt words Example as follows:

[0127] "You are a road condition analysis expert. Please describe the current highway conditions in detail and provide the corresponding JSON text, including: 1) Lane occupancy status; 2) Vehicle density level (smooth / slow / congested); 3) Estimated types and numbers of visible vehicles; 4) Weather and visibility conditions; 5) Whether there are accidents or construction; 6) Road facility status."

[0128] Structured text After adding the road segment number and timestamp, upload it to the cloud. The final uploaded structured text example is as follows:

[0129] {

[0130] "Road Section Number": "Highway_Monitor_S01_Cam04",

[0131] "Timestamp": "2025-05-20T14:30:00+08:00",

[0132] Lane information: {

[0133] Total number of lanes: 4

[0134] "First Lane (Overtaking Lane)": {

[0135] Status: "Unimpeded"

[0136] Vehicle density: Medium density

[0137] Average distance between vehicles: 30-50 meters

[0138] Main vehicle types: Small passenger vehicles

[0139] },

[0140] "Second Lane": {

[0141] Status: Slowing down

[0142] Vehicle density: Medium to high density

[0143] Average vehicle distance: 20-30 meters

[0144] "Main vehicle types": "Mixed (freight trucks / buses)"

[0145] },

[0146] "Third lane": {

[0147] Status: Slowing down

[0148] Vehicle density: Medium density

[0149] Average vehicle distance: 30-40 meters

[0150] Main vehicle types: Large trucks

[0151] },

[0152] Emergency Lane: {

[0153] Status: "Idle"

[0154] "Abnormal situation": "None"

[0155] }

[0156] },

[0157] "Vehicle Statistics": {

[0158] "Number of visible vehicles": 9,

[0159] "Percentage of small passenger vehicles": 0.44

[0160] "Trucks as a percentage": 0.44

[0161] "Other": 0.12,

[0162] Note: Includes one minivan / commercial vehicle.

[0163] },

[0164] "Environmental Information": {

[0165] Weather: Foggy / Rainy

[0166] Visibility: Poor (<200 meters)

[0167] Road surface condition: "Slippery / Waterlogged"

[0168] },

[0169] "Abnormal Events": {

[0170] "Accident": None

[0171] Construction: None

[0172] "Other": "Due to the heavy fog, traffic speeds are generally low."

[0173] },

[0174] "Road Facility Status": {

[0175] "Electronic Information Board": {

[0176] Status: "On",

[0177] Display content: "Control vehicle speed",

[0178] Color: Red

[0179] },

[0180] "Guardrail": "Normal"

[0181] "Markings": "Clearly visible"

[0182] }

[0183] }

[0184] Step S2: In the cloud, acquire multi-source heterogeneous data of structured text-related road segments, and convert the multi-source heterogeneous data into a semantic vector sequence with unified feature dimensions. The implementation steps are as follows.

[0185] Step S21: In the cloud, obtain multi-source heterogeneous data sequences of structured text-related road segments. :

[0186]

[0187] ,

[0188] in, This is a function for obtaining multi-source heterogeneous data sequences based on road segment location information. To associate the location information of road segments with structured text, Let L be the i-th multi-source heterogeneous data of the structured text associated road segment, and L be the number of multi-source heterogeneous data.

[0189] In this embodiment, L=7. Examples of seven heterogeneous data points in a multi-source heterogeneous data sequence are as follows:

[0190]

[0191] Step S22: Transform the acquired multi-source heterogeneous data sequences into a set of semantic vector sequences with unified feature dimensions. ,in Let i be the semantic vector sequence of the i-th multi-source heterogeneous data;

[0192]

[0193]

[0194]

[0195]

[0196]

[0197] in, This is a function that transforms text data into a sequence of semantic vectors. This is a function that transforms image data into a sequence of semantic vectors. This is a function that transforms video data into a sequence of semantic vectors. This indicates word embedding operations. This indicates that the input data is positionally encoded. This indicates that the image data is divided into blocks. This indicates that convolution operations are performed on image blocks. This indicates the extraction of keyframes from video data. This represents the Mth word vector in a semantic vector sequence of multi-source heterogeneous data. , The tensor dimension for each word vector.

[0198] Step S3: Based on the semantic vector sequence of structured text, a gated routing network is used to dynamically select the general expert network required for this highway traffic flow prediction. The required multi-source heterogeneous data is then input into each general expert network to obtain the semantic feature sequence of the heterogeneous multi-source information. Simultaneously, the text data and video image data involved in the multi-source heterogeneous data are input into the global text feature extraction expert network and the global video image feature extraction expert network, respectively, to obtain global text features and global video image features. The implementation steps are as follows.

[0199] Step S31: Generate structured text from the highway edge side Encoding is performed to obtain a sequence of structured text semantic vectors. ;

[0200]

[0201]

[0202] in, It is the nth word vector in the structured text semantic vector sequence.

[0203] Step S32: Obtain the structured text semantic vector sequence Input gated routing network The indices and weights of the top K general expert networks, the weights of the two global vision text and video image feature extraction expert networks, and the auxiliary loss of the gated routing network were calculated.

[0204]

[0205] in, This is the index sequence of the top K general expert networks selected by the gated routing network based on the structured text semantic vector sequence. For the selected top K general expert networks, Weights for the expert network are extracted to extract global text features. The weights of the expert network for global video image feature extraction are given. This is an auxiliary loss for gated routing networks.

[0206] Auxiliary loss The calculation method in a gated routing network is as follows:

[0207]

[0208] in, Let j be the call frequency of the j-th general expert network. Let N be the average routing probability for the j-th general expert network in a training batch, where N is the total number of general expert networks.

[0209] In this embodiment, N=6, indicating that there are a total of 6 general expert networks; topK is 4, indicating that 4 general expert networks are selected for each routing iteration, and the index of the general expert network selected in this routing iteration is... for .

[0210] Six examples of general expert networks are as follows:

[0211]

[0212] The following are examples of general expert networks selected:

[0213]

[0214] In this embodiment, the weights of the general expert network expert0 The weights of the general expert network expert2 are 0.2. The weights of the general expert network expert4 are 0.15. The weights of the general expert network expert5 are 0.25. The weights of the global text feature extraction expert network are 0.2. The weights of the global video image feature extraction expert network are set to 0.1. It is 0.1.

[0215] Step S33: Based on the index sequence Select the corresponding general expert network and index the sequence. Mapping to sequence The required multi-source heterogeneous data is input into the topK general expert networks for feature extraction, thereby obtaining the semantic features of heterogeneous multi-source information.

[0216]

[0217] in, This represents the semantic feature sequence of heterogeneous multi-source information extracted by the top K general expert networks; This represents the semantic features of the heterogeneous multi-source information extracted by the k-th general expert network; This indicates that feature extraction is performed using the k-th general expert network; Represents a set of semantic vector sequences from multi-source heterogeneous data. The set of multi-source heterogeneous data semantic vector sequences extracted for the k-th general expert network. .

[0218] In this embodiment, each general expert network It includes an L1 attention layer, and each layer contains a layer normalization module, a multi-head self-attention module, and a multilayer perceptron module;

[0219]

[0220]

[0221]

[0222] in, This represents the input to the c-th attention layer. MSA represents the first normalization module in the c-th attention layer, and MSA represents the multi-head self-attention module in the c-th attention layer. Indicates input The intermediate features output after processing by the first-layer normalization module and the multi-head self-attention module. This represents the output feature of the c-th attention layer. The second normalization module in the c-th attention layer is represented by MLP, and MLP represents the multilayer perceptron module in the c-th attention layer; the input is a general expert network. The data is processed sequentially through the L1 attention layer to obtain the semantic features of the heterogeneous multi-source information. .

[0223] All text-type data used by the general expert network in this route selection Input Global Text Feature Extraction Expert Network Feature extraction is performed to obtain the global text features of this route. ;

[0224]

[0225] In this embodiment, a global text feature extraction expert network It contains L2 recursive evolution layers, and each recursive evolution layer is defined as follows:

[0226]

[0227]

[0228]

[0229]

[0230]

[0231] in, For time steps Input features, For time steps The implicit memory tensor, For time steps The implicit memory tensor, It represents the Hadamah accumulation. An adaptive forgetting gate operator used to calculate the historical information retention rate under time-varying parameters. An intensity operator is injected into the information to control the intensity at which the current input features are written into the memory tensor. The input projection operator is used to reconstruct the semantic features at the current time step. , For learnable time-scale projection weights, For time scale offset. For predefined structured state parameters, For smoothing nonlinear activation functions, For input projection weights, For input bias, To output the projected weights, This is the output bias. The input to the first recursive evolution layer. For expert network Input , No. Output of the recursive evolution layer For expert network Output .

[0232] All video and image data used by the general expert network in this routing selection Input global video image feature extraction expert network Feature extraction is performed to obtain the global video image features of this route. ;

[0233]

[0234] In this embodiment, a global video image feature extraction expert network It includes an L3 spline topological association layer, and each spline topological association layer uses attention aggregation computation based on B-spline basis functions for feature extraction:

[0235]

[0236]

[0237]

[0238] in, and These are the input and output feature tensors within the layer, respectively. For via nonlinear mapping The generated query tensor, key tensor, and value tensor For tensor Dimensions The first eigenvector of the output eigenvector for nonlinear mapping One portion, The first normalized input feature vector is the... One portion, Let be the nonlinear mapping function on the connecting edge. For the p-th order B-spline basis function, Let be the control point coefficients learned during network training, representing the th control point coefficients learned using the p-th order B-spline basis function on the normalized input feature vector. The i-th component is calculated to obtain the i-th eigenvector of the output feature vector. The control point coefficients along this calculation path for each component, where i is the input feature. The index of the i-th vector component, where j is the output feature. The index of the j-th vector component, where p is the index of the p-th B-spline basis function. The spline mesh granularity is used to approximate the complex nonlinear spatiotemporal evolution of traffic flow videos by adjusting the control point coefficients. Based on the basic weight parameter matrix, for Activation function For layer normalization. Input to the first spline topological association layer. For expert network Input , No. Output of the spline topological association layer for .

[0239] Step S4: Perform weighted fusion of the semantic feature sequence of multi-source information, global text features, and global video image features to obtain a fused heterogeneous information feature vector sequence. The implementation steps are as follows.

[0240] Step S41: Fuse the semantic features of heterogeneous multi-source information extracted by the top K general expert networks, the global text features extracted by the global text feature extraction expert network, and the global video image features extracted by the global video image feature extraction expert network to obtain a feature vector sequence. ;

[0241]

[0242] in, Let be the weight of the k-th general expert network.

[0243] In this embodiment, the general expert index for routing selection for The weights of the general expert network expert0 The weights of the general expert network expert2 are 0.2. The weights of the general expert network expert4 are 0.15. The weights of the general expert network expert5 are 0.25. The weight of the global text feature extraction expert network is 0.2. The weights of the global video image feature extraction expert network are set to 0.1. The value is 0.1, therefore, the characteristic vector sequence The calculation formula is as follows:

[0244]

[0245] Step S42: Combine the fused feature vector sequence After the input to the feedforward neural network undergoes nonlinear transformation, a sequence of fused heterogeneous information feature vectors S for traffic flow prediction is obtained.

[0246]

[0247] in, This is the first feedforward neural network. for Activation function This is the second feedforward neural network.

[0248] Step S5: Input the structured text semantic vector sequence and the fused heterogeneous information feature vector sequence into the prediction decoder for decoding to obtain the highway traffic flow prediction result; construct the total loss based on the error between the prediction result and the actual traffic flow, as well as the auxiliary loss of the gated routing network, and train the overall model to obtain the trained prediction model. The implementation steps are as follows.

[0249] Step S51: Transform the structured text semantic vector sequence and fusion of heterogeneous information feature vector sequences Input Predictor Decoder This yields the final predicted vector sequence. ;

[0250]

[0251]

[0252] in, This is the prediction vector for the nth word in the vector sequence.

[0253] Predictive Decoder Depend on It consists of stacked decoding layers, each of which contains a mask multi-head self-attention module, a multi-head cross-attention module, a layer normalization module, and a multi-layer perception module.

[0254]

[0255]

[0256]

[0257]

[0258]

[0259] in, For the input of the h-th decoding layer, For the first layer normalization module of the h-th decoding layer, The result is the input of the h-th decoding layer after being normalized by the normalization module of the first input layer. Let be the first key projection matrix of the h-th decoding layer. Let be the first key feature matrix of the h-th decoding layer. Let be the projection matrix of the first value of the h-th decoding layer. Let be the first value feature matrix of the h-th decoding layer. This is the first query projection matrix of the h-th decoding layer. Let H be the first query feature matrix of the h-th decoding layer, where H is the number of stacked decoding layers.

[0260]

[0261] in, For the mask multi-head self-attention module of the h-th decoding layer, The output of the mask multi-head self-attention module for the h-th decoding layer and the input of the h-th decoding layer. The result of performing residual connection;

[0262]

[0263]

[0264]

[0265] in, This is the normalization module for the second layer of the h-th decoding layer. This is the second query projection matrix of the h-th decoding layer. Let S be the second query feature matrix of the h-th decoding layer, and let S be the sequence of fused heterogeneous information feature vectors used for traffic flow prediction. Let be the second key projection matrix of the h-th decoding layer. This is the second key feature matrix of the h-th decoding layer. The projection matrix of the second value of the h-th decoding layer. This is the second value feature matrix of the h-th decoding layer;

[0266]

[0267] in, For the multi-head cross-attention module of the h-th decoding layer, The output of the multi-head cross-attention module of the h-th decoding layer is concatenated with the residual and then the output of the mask multi-head self-attention module. The result of performing residual connection;

[0268]

[0269] in, This is the normalization module for the third layer of the h-th decoding layer. For the h-th decoding layer of the multi-layer perception module, This is the output of the h-th decoding layer;

[0270]

[0271] in, This is the final predicted vector sequence. For the last layer normalization module of the predictive decoder, This is the last multilayer perceptron module in the prediction decoder. This is the output of the H-th decoding layer, where H is the number of stacked predictive decoder layers.

[0272] Step S52: Use the final predicted vector sequence Traffic flow is predicted to obtain highway traffic flow prediction results;

[0273]

[0274] in, Y represents the predicted traffic flow of the highway, which is a multilayer perceptron module. t is the prediction time step. This refers to the number of lanes.

[0275] Step S53: Calculate the total loss of the overall model. ;

[0276]

[0277]

[0278] in, The mean squared error loss for a training batch. For the i-th sample in the batch, This refers to the highway traffic flow predicted using the i-th sample in the batch. The number of samples in a training batch.

[0279] Step S54: Total loss based on the overall model Perform backpropagation, update the model parameters, and train to obtain the final prediction model.

[0280] Step S6: Use the trained prediction model to predict highway traffic flow and generate corresponding prediction results.

[0281] like Figure 2 As shown, the implementation steps for predicting highway traffic flow using a trained prediction model are as follows.

[0282] Step S61: At regular intervals, acquire multi-source heterogeneous data from the monitoring points of interest on the highway, including POI data, meteorological data, holiday information data, historical traffic flow data, and surveillance video data, and store them in the database.

[0283] Step S62: Capture real-time road condition images at regular intervals on the edge of the highway, generate structured text on the edge, and upload it to the cloud.

[0284] Step S63: Retrieve the latest multi-source heterogeneous data set from the database. ,in This is the latest Lth multi-source heterogeneous data from the monitoring point.

[0285] Step S64: Combine multi-source heterogeneous data sets The predicted traffic flow results for highways are obtained by combining the predicted model with structured text input. .

[0286] This embodiment also provides a computer device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which implement the above-described method when executed by the processor.

[0287] This embodiment also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method.

[0288] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0289] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0290] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0291] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0292] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for predicting highway traffic flow based on structured semantics and expert networks, characterized in that, Includes the following steps: Step S1: At the edge of the highway, acquire road condition images, generate structured text describing road condition information using a pre-trained multimodal model, and then upload it to the cloud; Step S2: In the cloud, acquire multi-source heterogeneous data of structured text-related road segments, and convert the multi-source heterogeneous data into a semantic vector sequence with unified feature dimensions; Step S3: Based on the semantic vector sequence of structured text, the general expert network required for this highway traffic flow prediction is dynamically selected through a gated routing network, and the required multi-source heterogeneous data is input into each general expert network to obtain the semantic feature sequence of heterogeneous multi-source information; at the same time, the text data and video image data involved in the multi-source heterogeneous data are input into the global text feature extraction expert network and the global video image feature extraction expert network respectively to obtain global text features and global video image features; Step S4: Perform weighted fusion of the semantic feature sequence of multi-source information, global text features, and global video image features to obtain a fused heterogeneous information feature vector sequence; Step S5: Input the structured text semantic vector sequence and the fused heterogeneous information feature vector sequence into the prediction decoder for decoding to obtain the highway traffic flow prediction result; construct the total loss based on the error between the prediction result and the actual traffic flow and the auxiliary loss of the gated routing network, train the overall model, and obtain the trained prediction model. Step S6: Using the trained prediction model, predict the traffic flow on the highway and generate the corresponding prediction results; Step S1 specifically includes the following steps: Step S11: Capture a frame of road condition image at regular intervals using edge devices equipped with high-performance cameras and computing units deployed on the highway. After standardization processing, a standardized road condition image is obtained. ; in, For normalized functions; Step S12: Convert the standardized road condition image Input a pre-trained multimodal model deployed in a computing unit to generate structured text describing road condition information. And upload it to the cloud; in, This is a pre-trained multimodal model for generating structured text based on road condition images. Prompt words generated for pre-defined structured guidance text.

2. The highway traffic flow prediction method based on structured semantics and expert networks according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S21: In the cloud, obtain multi-source heterogeneous data sequences of structured text-related road segments. : ， in, This is a function for obtaining multi-source heterogeneous data sequences based on road segment location information. To associate the location information of road segments with structured text, Let L be the i-th multi-source heterogeneous data of the structured text associated road segment, and L be the number of multi-source heterogeneous data. Step S22: Transform the acquired multi-source heterogeneous data sequences into a set of semantic vector sequences with unified feature dimensions. ,in Let i be the semantic vector sequence of the i-th multi-source heterogeneous data; in, This is a function that transforms text data into a sequence of semantic vectors. This is a function that transforms image data into a sequence of semantic vectors. This is a function that transforms video data into a sequence of semantic vectors. This indicates word embedding operations. This indicates that the input data is positionally encoded. This indicates that the image data is divided into blocks. This indicates that convolution operations are performed on image blocks. This indicates the extraction of keyframes from video data. This represents the Mth word vector in a sequence of semantic vectors from multi-source heterogeneous data.

3. The highway traffic flow prediction method based on structured semantics and expert networks according to claim 1, characterized in that, Step S3 specifically includes the following steps: Step S31: Generate structured text from the highway edge side Encoding is performed to obtain a sequence of structured text semantic vectors. ; in, This is the nth word vector in the structured text semantic vector sequence; Step S32: Obtain the structured text semantic vector sequence Input gated routing network The indices and weights of the top K general expert networks, the weights of the two global vision text and video image feature extraction expert networks, and the auxiliary loss of the gated routing network were calculated. in, This is the index sequence of the top K general expert networks selected by the gated routing network based on the structured text semantic vector sequence. For the selected top K general expert networks, Weights for the expert network are extracted to extract global text features. The weights of the expert network for global video image feature extraction are given. The auxiliary loss in a gated routing network is calculated as follows: in, Let j be the call frequency of the j-th general expert network. is the average routing probability for the j-th general expert network in a training batch, where N is the total number of general expert networks; Step S33: Based on the index sequence Select the corresponding general expert network and index the sequence. Mapping to sequence The required multi-source heterogeneous data is input into the topK general expert networks for feature extraction, thereby obtaining the semantic features of heterogeneous multi-source information. in, This represents the semantic feature sequence of heterogeneous multi-source information extracted by the top K general expert networks; This represents the semantic features of the heterogeneous multi-source information extracted by the k-th general expert network; This indicates that feature extraction is performed using the k-th general expert network; Represents a set of semantic vector sequences from multi-source heterogeneous data. The set of multi-source heterogeneous data semantic vector sequences extracted for the k-th general expert network. ; All text-type data used by the general expert network in this route selection Input Global Text Feature Extraction Expert Network Feature extraction is performed to obtain the global text features of this route. ; All video and image data used by the general expert network in this routing selection Input global video image feature extraction expert network Feature extraction is performed to obtain the global video image features of this route. ; 。 4. The highway traffic flow prediction method based on structured semantics and expert networks according to claim 3, characterized in that, In step S33, each general expert network It includes an L1 attention layer, and each layer contains a layer normalization module, a multi-head self-attention module, and a multilayer perceptron module; in, This represents the input to the c-th attention layer. MSA represents the first normalization module in the c-th attention layer, and MSA represents the multi-head self-attention module in the c-th attention layer. Indicates input The intermediate features output after processing by the first-layer normalization module and the multi-head self-attention module. This represents the output feature of the c-th attention layer. The second normalization module in the c-th attention layer is represented by MLP, and MLP represents the multilayer perceptron module in the c-th attention layer; the input is a general expert network. The data is processed sequentially through the L1 attention layer to obtain the semantic features of the heterogeneous multi-source information. ; Global Text Feature Extraction Expert Network It contains L2 recursive evolution layers, and each recursive evolution layer is defined as follows: in, For time step Input features, For time step The implicit memory tensor, For time step The implicit memory tensor, It represents the Hadamah accumulation. An adaptive forgetting gate operator used to calculate the historical information retention rate under time-varying parameters. An intensity operator is injected into the information to control the intensity at which the current input features are written into the memory tensor. The input projection operator is used to reconstruct the semantic features at the current time step. , For learnable time-scale projection weights, For time scale offset. For predefined structured state parameters, For smoothing nonlinear activation functions, For input projection weights, For input bias, To output the projected weights, For output bias; input of the first recursive evolution layer For expert network Input , No. Output of the recursive evolution layer For expert network Output ; Global Video Image Feature Extraction Expert Network It includes an L3 spline topological association layer, and each spline topological association layer uses attention aggregation computation based on B-spline basis functions for feature extraction: in, and These are the input and output feature tensors within the layer, respectively. For via nonlinear mapping The generated query tensor, key tensor, and value tensor For tensor Dimensions The first eigenvector of the output eigenvector for nonlinear mapping One portion, The first normalized input feature vector is the... One portion, Let be the nonlinear mapping function on the connecting edge. For the p-th order B-spline basis function, Let be the control point coefficients learned during network training, representing the th control point coefficients learned using the p-th order B-spline basis function on the normalized input feature vector. The i-th component is calculated to obtain the i-th eigenvector of the output feature vector. The control point coefficients along this calculation path for each component, where i is the input feature. The index of the i-th vector component, where j is the output feature. The index of the j-th vector component, where p is the index of the p-th B-spline basis function. The spline mesh granularity is used to approximate the complex nonlinear spatiotemporal evolution of traffic flow videos by adjusting the control point coefficients. Based on the basic weight parameter matrix, for Activation function For layer normalization; input to the first spline topological correlation layer For expert network Input , No. Output of the spline topological association layer for .

5. The highway traffic flow prediction method based on structured semantics and expert networks according to claim 3, characterized in that, Step S4 specifically includes the following steps: Step S41: Fuse the semantic features of heterogeneous multi-source information extracted by the top K general expert networks, the global text features extracted by the global text feature extraction expert network, and the global video image features extracted by the global video image feature extraction expert network to obtain a feature vector sequence. ; in, Let be the weight of the k-th general expert network; Step S42: Combine the fused feature vector sequence After the input to the feedforward neural network undergoes nonlinear transformation, a sequence of fused heterogeneous information feature vectors S for traffic flow prediction is obtained. in, This is the first feedforward neural network. for Activation function This is the second feedforward neural network.

6. The highway traffic flow prediction method based on structured semantics and expert networks according to claim 3, characterized in that, Step S5 specifically includes the following steps: Step S51: Transform the structured text semantic vector sequence and fusion of heterogeneous information feature vector sequences Input Predictor Decoder This yields the final predicted vector sequence. ; in, This is the vector of the nth word in the predicted vector sequence; Predictive Decoder Depend on It consists of stacked decoding layers, each of which contains a mask multi-head self-attention module, a multi-head cross-attention module, a layer normalization module, and a multi-layer perception module. in, This is the input to the h-th decoding layer. For the first layer normalization module of the h-th decoding layer, The result is the input of the h-th decoding layer after being normalized by the normalization module of the first input layer. Let be the first key projection matrix of the h-th decoding layer. Let be the first key feature matrix of the h-th decoding layer. Let be the projection matrix of the first value of the h-th decoding layer. Let be the first value feature matrix of the h-th decoding layer. This is the first query projection matrix of the h-th decoding layer. Let H be the first query feature matrix of the h-th decoding layer, where H is the number of stacked decoding layers. in, For the mask multi-head self-attention module of the h-th decoding layer, The output of the mask multi-head self-attention module for the h-th decoding layer and the input of the h-th decoding layer. The result of performing residual connection; in, This is the normalization module for the second layer of the h-th decoding layer. This is the second query projection matrix of the h-th decoding layer. Let S be the second query feature matrix of the h-th decoding layer, and let S be the sequence of fused heterogeneous information feature vectors used for traffic flow prediction. Let be the second key projection matrix of the h-th decoding layer. This is the second key feature matrix of the h-th decoding layer. The projection matrix of the second value of the h-th decoding layer. This is the second value feature matrix of the h-th decoding layer; in, For the multi-head cross-attention module of the h-th decoding layer, The output of the multi-head cross-attention module of the h-th decoding layer is concatenated with the residual and then the output of the mask multi-head self-attention module. The result of performing residual connection; in, This is the normalization module for the third layer of the h-th decoding layer. For the h-th decoding layer of the multi-layer perception module, This is the output of the h-th decoding layer; in, This is the final predicted vector sequence. For the last layer normalization module of the predictive decoder, This is the last multilayer perceptron module in the prediction decoder. This is the output of the H-th decoding layer, where H is the number of stacked predictive decoder layers; Step S52: Use the final predicted vector sequence Traffic flow is predicted to obtain highway traffic flow prediction results; in, For a multilayer perceptron module, Y represents the predicted highway traffic flow, and t represents the prediction time step. Number of lanes; Step S53: Calculate the total loss of the overall model. ; in, The mean squared error loss for a training batch. For the i-th sample in the batch, This refers to the highway traffic flow predicted using the i-th sample in the batch. The number of samples in a training batch; Step S54: Total loss based on the overall model Perform backpropagation, update the model parameters, and train to obtain the final prediction model.

7. The highway traffic flow prediction method based on structured semantics and expert networks according to claim 1, characterized in that, Step S6 specifically includes the following steps: Step S61: Periodically acquire multi-source heterogeneous data from highway monitoring points and store it in the database; Step S62: Capture real-time road condition images at regular intervals on the edge of the highway, generate structured text on the edge, and upload it to the cloud; Step S63: Retrieve the latest multi-source heterogeneous data set from the database. ,in This is the latest L-th multi-source heterogeneous data from the monitoring point; Step S64: Combine multi-source heterogeneous data sets The predicted traffic flow results for highways are obtained by combining the predicted model with structured text input. .

8. A computer device, characterized in that, include: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method described in any one of claims 1-7 is implemented.

Images