Multi-source data processing method and device

Abstract

According to the multi-source data processing method and device provided by the embodiment of the invention, the abnormal and missing correction of the multi-source data is realized through innovatively constructing a data preprocessing mechanism and through exponential smoothing and local regression. And designing an attention-based feature extraction network, and establishing a fusion representation strategy of time sequence features in combination with query-key-value calculation and normalization processing. A deep optimization mechanism is introduced, and accurate training of a prediction model is realized through nonlinear transformation and gradient descent. According to the method, the defects of the traditional technology in the aspects of data processing, feature extraction, model optimization and the like are effectively overcome, and the accuracy and reliability of traffic flow prediction are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a multi-source data processing method and device. Background Art

[0002] Existing traffic flow forecasting methods have significant shortcomings. Traditional systems lack systematicity in processing multi-source data, making it difficult to effectively handle missing and anomalies in traffic flow, weather, and road section data, which affects forecast accuracy.

[0003] Furthermore, existing technologies face bottlenecks in feature extraction. Most systems fail to fully leverage the advantages of the Transformer model and lack temporal feature extraction based on attention mechanisms, resulting in suboptimal capture of long-term dependencies.

[0004] Existing systems have technical shortcomings in model optimization. They lack the ability to perform deep nonlinear transformations on features, making it difficult to improve prediction accuracy through gradient descent and loss optimization, which impacts the final results. Addressing these issues is crucial for improving traffic flow prediction. Summary of the Invention

[0005] In response to the problems in the existing technology, this application provides a multi-source data processing method and device, which can effectively solve the shortcomings of traditional technologies in data processing, feature extraction and model optimization, and significantly improve the accuracy and reliability of traffic flow prediction.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a multi-source data processing method, comprising: Collect historical traffic sequence data from traffic monitoring devices, obtain weather data from meteorological monitoring stations, and obtain road section data from road network management systems; fill missing values ​​in the historical traffic sequence data using exponential smoothing; supplement missing values ​​in the weather data and the road section data using linear interpolation of adjacent time data; calculate the mean and standard deviation of the historical traffic sequence data; mark data that deviates from the mean by more than a multiple of the standard deviation as outliers; smooth the outliers using a local weighted regression algorithm; construct a feature matrix from the historical traffic sequence data based on a sliding window method; perform numerical encoding conversion on the weather data and the road section data; and divide the preprocessed data into a training data set and a validation data set according to time series; Constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps to generate a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; The fused feature vector is input into a fully connected neural network, and nonlinear transformation and dimensionality reduction processing are performed on the features. An activation function is used to perform nonlinear mapping on the transformed features. The prediction error is calculated based on the mean square error loss function. The network parameters are iteratively optimized using the gradient descent method to generate a traffic flow prediction model. The verification data set is input into the traffic flow prediction model for prediction verification, and the traffic flow prediction result is output.

[0007] Furthermore, the method further includes: acquiring historical traffic sequence data, weather data, and road section data from a traffic monitoring device, a meteorological monitoring station, and a road network management system based on a data acquisition interface, marking the historical traffic sequence data, the weather data, and the road section data according to timestamps, constructing a time series index table, performing format conversion and data standardization processing on the historical traffic sequence data, and mapping the weather data and the road section data to the time series index table; Missing values ​​in the historical traffic sequence data are filled with predicted values ​​calculated based on the exponential decay coefficient, and missing values ​​in the weather data and the road section data are supplemented by the weighted average of the data at adjacent moments. The statistical characteristics of the historical traffic sequence data, including the mean, standard deviation and coefficient of variation, are calculated, and an outlier identification threshold is set based on the statistical characteristics. Data points that deviate from the mean by more than a multiple of the standard deviation are marked as outliers.

[0008] Furthermore, the method further includes: constructing a local time window for data points marked as outliers, calculating the Euclidean distance of the data points in the window, determining a weight coefficient based on the distance value, performing a weighted summation of the weight coefficient and the data points in the window, performing smoothing correction on the outliers, performing a sliding segmentation on the historical traffic sequence data based on a preset window length, constructing the traffic data in each time window into a feature vector, and performing a normalization process on the feature vector to generate a feature matrix; A category mapping dictionary is established for the weather data, and the weather type is converted into a numerical identifier. A hash mapping table is established for the road section data, and the road section number is converted into a continuous numerical identifier. The feature matrix, the weather data and the road section data are aligned in the time dimension, and the aligned data are divided into a training data set and a validation data set according to a preset ratio.

[0009] Furthermore, the method further includes: constructing a multi-head attention feature extraction network based on the training data set, dividing the feature matrix into equal lengths according to timestamps, performing a linear transformation on the divided feature sequence using three independent weight matrices to generate a query vector sequence, a key vector sequence, and a value vector sequence, respectively, performing a matrix transposition operation on the query vector sequence and the key vector sequence, and performing a dot product operation on the transposed matrices to generate a feature correlation matrix; The feature correlation matrix is ​​dimensionally scaled, and the scaled matrix is ​​input into the softmax function for normalization to generate an attention weight matrix, where each element of the attention weight matrix represents the correlation strength between features at different time steps. The attention weight matrix is ​​sparsified to filter out feature associations with weight values ​​below a threshold.

[0010] Furthermore, the method further includes: performing a matrix multiplication operation on the attention weight matrix and the value vector sequence to obtain a weighted feature representation, constructing a multilayer perceptron network for the weighted feature representation, performing a nonlinear transformation on the features through two fully connected layers, adding an activation function between the two fully connected layers to enhance the expression ability of the network, and performing a residual connection on the transformed features and the original input features; The mean and variance of the concatenated features are calculated, the features are standardized based on the mean and the variance, learnable scaling parameters and translation parameters are introduced to adaptively adjust the standardized features, a fused feature vector is generated, a dimensionality reduction mapping is performed on the fused feature vector, a feature importance score vector is constructed, and the fused features are screened based on the feature importance score vector.

[0011] Furthermore, the method further includes: inputting the fused feature vector into a multi-layer fully connected neural network, wherein the fully connected neural network includes an input layer, a hidden layer, and an output layer, constructing a weight matrix and a bias vector between adjacent layers, performing a linear transformation on the input features, applying batch normalization processing to the hidden layer to eliminate feature distribution offset, fusing feature representations of different levels through a skip connection structure, and introducing a dropout mechanism at the output end of each hidden layer to prevent overfitting; A modified linear unit function is used to perform nonlinear mapping on the transformed features, and the mapping result is input into the output layer to generate a predicted value. The difference between the predicted value and the true value is calculated, and the difference is squared and the mean is obtained to obtain the mean square error loss value. The objective function is constructed based on the mean square error loss value, and the gradient of the objective function with respect to the network parameters is calculated.

[0012] Furthermore, the method further includes: constructing a parameter update equation based on the objective function gradient, iteratively updating the network parameters using an adaptive moment estimation optimization algorithm, calculating first-order moment estimates and second-order moment estimates of the parameters in each iteration, adaptively adjusting the learning rate based on historical gradient information, performing gradient clipping on the updated parameters to prevent gradient explosion, stopping the optimization when the objective function converges or reaches a preset number of iterations, and generating a traffic flow prediction model; The validation data set is input into the traffic flow prediction model in chronological order, the mean absolute error and root mean square error of the prediction results are calculated, the optimal model parameters are screened based on the validation indicators, the screened model is applied to real-time traffic flow data, rolling predictions are made for traffic flows at different time scales, and the prediction results are output.

[0013] In a second aspect, the present application provides a multi-source data processing device, comprising: a data set determination module, configured to collect historical traffic sequence data from a traffic monitoring device, obtain weather data from a meteorological monitoring station, and obtain road section data from a road network management system; fill missing values ​​in the historical traffic sequence data using an exponential smoothing method; supplement missing values ​​in the weather data and the road section data using linear interpolation of adjacent time data; calculate the mean and standard deviation of the historical traffic sequence data; mark data that deviates from the mean by more than a multiple of the standard deviation as outliers; smooth the outliers using a local weighted regression algorithm; construct a feature matrix from the historical traffic sequence data based on a sliding window method; perform numerical encoding conversion on the weather data and the road section data; and divide the preprocessed data into a training data set and a validation data set according to time series; a feature fusion module for constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps to generate a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; The traffic flow prediction module is used to input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

[0014] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-source data processing method when executing the program.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the multi-source data processing method when executed by a processor.

[0016] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the multi-source data processing method when executed by a processor.

[0017] As can be seen from the above technical solutions, the present application provides a multi-source data processing method and device, which innovatively constructs a data preprocessing mechanism, realizes the correction of abnormal missing multi-source data through exponential smoothing and local regression. Designs an attention-based feature extraction network, combines query-key-value calculation and normalization processing, and establishes a fusion representation strategy for time series features. Introduces a deep optimization mechanism, and realizes accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction, and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 Schematic diagram of the process of multi-source data processing in an embodiment of the present application; Figure 2 This is a structural diagram of a multi-source data processing device in an embodiment of the present application; Figure 3 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0020] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] The acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant laws and regulations.

[0023] Taking into account the problems existing in the prior art, the present application provides a multi-source data processing method and device, which innovatively constructs a data preprocessing mechanism, realizes the correction of abnormal missing multi-source data through exponential smoothing and local regression. Designs an attention-based feature extraction network, combines query-key-value calculation and normalization processing, and establishes a fusion representation strategy for time series features. Introduces a deep optimization mechanism, and realizes accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction.

[0024] In order to effectively solve the deficiencies of traditional technologies in data processing, feature extraction, and model optimization, and significantly improve the accuracy and reliability of traffic flow prediction, this application provides an embodiment of a multi-source data processing method, see Figure 1 , the multi-source data processing method specifically includes the following contents: Step S101: historical traffic sequence data is collected from a traffic monitoring device, weather data is obtained from a meteorological monitoring station, and road section data is obtained from a road network management system. Missing values ​​in the historical traffic sequence data are filled using an exponential smoothing method. Missing values ​​in the weather data and the road section data are supplemented using linear interpolation of adjacent time data. The mean and standard deviation of the historical traffic sequence data are calculated. Data that deviates from the mean by more than a multiple of the standard deviation is marked as an outlier. The outliers are smoothed using a local weighted regression algorithm. A feature matrix is ​​constructed from the historical traffic sequence data based on a sliding window method. The weather data and the road section data are converted into numerical codes. The preprocessed data is divided into a training data set and a validation data set according to time series. Optionally, this embodiment innovatively designs a pre-processing solution based on multi-source data fusion to address issues such as unstable data quality and insufficient feature expression in traffic flow prediction. This embodiment first constructs a data acquisition framework and achieves comprehensive capture of traffic conditions through multi-level data cleaning. The system designs a data quality assessment formula: Quality_Score = α Completeness +β Consistency - γ × Anomaly_Factor, where Completeness represents data completeness, Consistency represents data consistency, Anomaly_Factor represents the anomaly factor, and α, β, and γ are dynamic adjustment coefficients. In traffic forecasting scenarios, this multi-dimensional quality assessment method can effectively ensure data availability.

[0025] This embodiment deeply optimizes the data collection strategy. In view of the multi-source characteristics of traffic data, a collection mechanism based on time alignment is designed. Through precise timestamp marking, synchronous collection of data from different sources is achieved. Special attention is paid to the timeliness of the data. When a data flow interruption or delay is detected, the system will perform supplementary collection through the backup channel. For example, when processing traffic data on urban main roads, multi-source collection can simultaneously obtain key information such as traffic flow, weather conditions, and road section status, which is crucial for building a complete prediction feature.

[0026] This embodiment innovatively implements a missing value processing mechanism. In view of the missing characteristics of different types of data, the system constructs a filling framework based on exponential smoothing. By analyzing the time series correlation of the data, accurate prediction of missing values ​​is achieved. Special attention is paid to the continuity of the data, and the rationality of the filling results is ensured by designing an adaptive smoothing strategy. This time series-based filling method can effectively maintain the time series characteristics of the data. This embodiment adopts the exponential smoothing prediction formula: Y(t) = α×X(t-1) + (1-α)×Y(t-1), where Y(t) represents the predicted value at the current moment, X(t-1) represents the actual value at the previous moment, Y(t-1) represents the predicted value at the previous moment, and α is the smoothing coefficient.

[0027] This embodiment deeply optimizes the outlier handling strategy. The system builds an anomaly detection framework based on standard deviation and accurately identifies outliers through dynamic threshold adjustment. It pays special attention to the distribution characteristics of the data and improves the accuracy of outlier handling by designing a local weighted regression mechanism. This comprehensive outlier handling approach provides a reliable data foundation for feature extraction.

[0028] This example achieves structured data representation through feature construction. The system builds a sliding window-based feature extraction framework, combining it with multi-dimensional feature encoding for comprehensive modeling. Special attention is paid to the expressive power of features, and a mapping dictionary mechanism is established to achieve effective conversion of categorical features. This systematic feature construction approach provides standardized data support for subsequent analysis.

[0029] The innovative design of this embodiment not only addresses data quality issues in traditional methods but also establishes a sustainable and optimized preprocessing framework. Through multi-level data cleaning and feature construction, the system is able to extract effective predictive features from complex raw data. This multi-source fusion-based processing mechanism ensures that the system maintains effective feature expression capabilities even in complex traffic environments. In traffic flow forecasting, this intelligent preprocessing solution significantly improves prediction accuracy.

[0030] This embodiment achieves intelligent analysis upgrades for traffic forecasting by establishing a complete data processing chain. The system dynamically adjusts processing strategies based on real-time data quality, avoiding the limitations of traditional fixed-rule solutions. Multi-dimensional data cleaning and feature construction significantly improve the quality and reliability of feature representation, providing reliable data support for subsequent traffic forecasting. This intelligent processing mechanism demonstrates strong adaptability and optimization effects in traffic forecasting.

[0031] This embodiment not only improves data processing accuracy but also establishes a continuously evolving preprocessing system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate feature representations for subsequent predictions. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction effectiveness, providing reliable technical support for traffic management.

[0032] Step S102: constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps, generating a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; Optionally, this embodiment innovatively designs a feature extraction solution based on the attention mechanism to address the problems of insufficient extraction of time series features and inaccurate feature association modeling in traffic flow prediction. This embodiment first constructs a feature processing framework and implements in-depth analysis of traffic characteristics through multi-level attention calculations. The system designs an attention calculation formula: Attention(Q,K,V) = softmax(QK^T / √d)V, where Q represents the query vector, K represents the key vector, V represents the value vector, d represents the feature dimension, and √d is the scaling factor. In the traffic flow prediction scenario, this attention-based feature evaluation method can effectively capture the dynamic correlation between time series features.

[0033] This embodiment deeply optimizes the time series segmentation strategy. In view of the time series characteristics of traffic flow data, a timestamp-based feature segmentation mechanism is designed. Through precise time series alignment, effective division of historical traffic characteristics is achieved. Special attention is paid to the continuity of the data. When a time series breakpoint is detected, the system will smooth it through a supplementary sampling method. For example, when analyzing the traffic flow on urban main roads, time series segmentation can accurately distinguish the characteristic patterns of different time periods such as morning and evening peaks and off-peak periods, which is crucial for building an accurate prediction model.

[0034] This embodiment innovatively implements a feature transformation mechanism. In view of the multi-dimensional attributes of traffic features, the system constructs a feature transformation framework based on linear mapping. Through three independent weight matrices, multi-angle transformation of input features is achieved. Special attention is paid to the orthogonality of the transformation, and the diversity of feature expression is ensured by designing independent mapping spaces. This feature processing method based on multi-head mapping can effectively extract temporal features from different perspectives. This embodiment adopts the feature importance evaluation formula: Feature_Weight = α Temporal_Correlation +β Spatial_Relevance, where Temporal_Correlation represents temporal correlation, Spatial_Relevance represents spatial correlation, and α and β are dynamic weight coefficients.

[0035] This embodiment deeply optimizes the attention calculation strategy. The system builds a feature correlation framework based on dot product operations and accurately calculates attention weights through dimensional scaling. It pays special attention to the distribution characteristics of weights and improves the stability of attention scores by designing a normalization mechanism. This comprehensive attention calculation method provides a reliable weight foundation for feature fusion.

[0036] This embodiment achieves effective integration of multi-source information through feature fusion. The system constructs a weighted feature fusion framework and combines it with layer normalization for feature optimization. It pays special attention to the statistical properties of features and establishes an adaptive normalization mechanism to effectively control feature distribution. This systematic fusion solution provides high-quality feature representation for subsequent predictions.

[0037] The innovative design of this embodiment not only solves the feature extraction problem in traditional methods but also establishes a sustainably optimized feature learning framework. Through multi-level attention calculation and feature fusion, the system is able to extract effective time series features from complex traffic data. This attention-based learning mechanism ensures that the system maintains effective feature expression capabilities even in complex traffic scenarios. In traffic flow prediction, this intelligent feature extraction solution significantly improves prediction accuracy.

[0038] This embodiment achieves an intelligent upgrade for traffic flow prediction by establishing a complete feature processing chain. The system dynamically adjusts processing strategies based on real-time data characteristics, avoiding the limitations of traditional fixed feature solutions. Through multi-dimensional feature extraction and fusion, the quality and reliability of feature expression are significantly improved, providing reliable feature support for subsequent traffic flow prediction. This intelligent processing mechanism demonstrates strong adaptability and optimization effects in traffic flow prediction.

[0039] This embodiment not only improves the accuracy of feature processing but also establishes an evolving learning system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate feature representations for subsequent predictions. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction results, providing reliable technical support for traffic management.

[0040] Step S103: Input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

[0041] Optionally, this embodiment innovatively designs a prediction optimization solution based on deep learning to address the problems of insufficient model expression ability and unstable prediction accuracy in traffic flow prediction. This embodiment first constructs a prediction model framework and realizes deep modeling of traffic characteristics through multi-level nonlinear transformations. The system designs a prediction error evaluation formula: Loss = (1 / n)∑(y_pred - y_true)^2 + λ×Regularization_Term, where y_pred represents the predicted value, y_true represents the true value, n represents the number of samples, λ is the regularization coefficient, and Regularization_Term represents the regularization term. In the traffic flow prediction scenario, this error evaluation method based on mean square error can effectively guide the optimization direction of the model.

[0042] This embodiment deeply optimizes the feature transformation strategy. In view of the complex distribution characteristics of the fusion features, a nonlinear transformation mechanism based on a multi-layer perceptron is designed. Through a deep network structure, a full exploration of the feature space is achieved. Special attention is paid to the expressive power of the network. When changes in the feature distribution are detected, the system will enhance the fitting ability of the model through adaptive layer width adjustment. For example, when processing urban traffic flow forecasts, nonlinear transformations can accurately capture the complex patterns of traffic changes, including the influence of multiple factors such as emergencies and seasonal changes, which is crucial to improving prediction accuracy.

[0043] This embodiment innovatively implements a parameter optimization mechanism. In response to the training problem of deep networks, the system constructs an optimization framework based on gradient descent. By analyzing the gradient information of the loss function, precise adjustment of network parameters is achieved. Special attention is paid to the stability of the optimization, and the effectiveness of parameter updates is ensured by designing an adaptive learning rate strategy. This gradient-based optimization method can effectively improve the convergence performance of the model. This embodiment adopts the parameter update formula: θ_new = θ_old -η×(∂Loss / ∂θ), where θ represents the network parameters, η represents the learning rate, and ∂Loss / ∂θ represents the gradient of the loss function with respect to the parameters.

[0044] This embodiment deeply optimizes the prediction validation strategy. The system builds a model evaluation framework based on the validation dataset, comprehensively evaluating prediction effectiveness through multi-dimensional performance metrics. Special attention is paid to prediction stability, and a rolling prediction mechanism is designed to improve the model's reliability in long-term forecasts. This comprehensive validation approach provides a reliable evaluation basis for model selection.

[0045] This example achieves continuous model improvement through iterative optimization. The system builds a training framework based on early stopping and combines it with validation metrics for model screening. It pays special attention to overfitting and effectively controls model complexity by establishing a regularization mechanism. This systematic optimization approach provides high-quality model support for prediction tasks.

[0046] The innovative design of this embodiment not only solves the prediction accuracy issues of traditional methods but also establishes a continuously optimized prediction framework. Through multi-level feature transformation and parameter optimization, the system is able to learn effective prediction models from complex traffic data. This deep learning-based prediction mechanism ensures that the system maintains effective prediction capabilities when faced with diverse traffic scenarios. In traffic flow prediction, this intelligent optimization solution significantly improves the accuracy and stability of predictions.

[0047] This embodiment achieves an intelligent forecast upgrade for traffic flow by establishing a complete forecast processing chain. The system dynamically adjusts optimization strategies based on real-time forecast results, avoiding the limitations of traditional fixed model solutions. Through multi-dimensional model optimization and verification, the quality and reliability of forecasts are significantly improved, providing reliable data support for traffic management decisions. This intelligent forecasting mechanism demonstrates strong adaptability and optimization effects in traffic flow analysis.

[0048] This embodiment not only improves prediction accuracy but also establishes a continuously evolving prediction system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate prediction results for traffic management. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction effectiveness, providing reliable decision-making support for traffic management.

[0049] From the above description, it can be seen that the multi-source data processing method provided in the embodiment of the present application can realize the correction of abnormal missing of multi-source data through innovative construction of data preprocessing mechanism, exponential smoothing and local regression. Design an attention-based feature extraction network, combine query-key-value calculation and normalization processing, and establish a fusion representation strategy for time series features. Introduce a deep optimization mechanism, and realize accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction.

[0050] In one embodiment of the multi-source data processing method of the present application, the following contents may also be specifically included: Step S201: Based on the data acquisition interface, historical traffic sequence data, weather data, and road segment data are respectively obtained from the traffic monitoring device, the meteorological monitoring station, and the road network management system, the historical traffic sequence data, the weather data, and the road segment data are marked according to timestamps, a time series index table is constructed, the format of the historical traffic sequence data is converted and the data is normalized, and the weather data and the road segment data are mapped to the time series index table; Step S202: Fill the missing values ​​in the historical traffic sequence data with predicted values ​​calculated based on the exponential decay coefficient, supplement the missing values ​​in the weather data and the road section data with the weighted average of the data at adjacent moments, calculate the statistical characteristics of the historical traffic sequence data, including the mean, standard deviation and coefficient of variation, set an outlier identification threshold based on the statistical characteristics, and mark data points that deviate from the mean by more than a multiple of the standard deviation as outliers.

[0051] Optionally, this embodiment innovatively designs a data processing solution based on time series indexing to meet the technical requirements of multi-source traffic data collection and preprocessing. This embodiment first builds a data collection framework and achieves comprehensive monitoring of traffic conditions through multi-level data synchronization. The system designs a data synchronization evaluation formula: Sync_Score = α Time_ Alignment +β Data_Quality - γ × Delay_Factor, where Time_Alignment represents time alignment, Data_Quality represents data quality, Delay_Factor represents the delay factor, and α, β, and γ are dynamic adjustment coefficients. In traffic monitoring scenarios, this multi-dimensional synchronous evaluation method can effectively ensure the temporal consistency of data.

[0052] This embodiment deeply optimizes the data collection strategy. In view of the multi-source characteristics of traffic data, a collection mechanism based on a unified interface is designed. Through a standardized data interface, unified acquisition of data from different sources is achieved. Special attention is paid to the real-time nature of the data. When an abnormal response from a data source is detected, the system will perform supplementary collection through a backup channel. For example, when monitoring the traffic status of a city's main roads, multi-source data collection can simultaneously obtain key information such as changes in traffic volume, weather changes, and road conditions, which is crucial for building a complete traffic status model.

[0053] This embodiment innovatively implements a time series indexing mechanism. To address the temporal consistency requirements for multi-source data, the system builds a timestamp-based indexing framework. Through precise time series alignment, unified organization of multi-source data is achieved. Special attention is paid to index uniqueness, and a conflict resolution mechanism is designed to ensure accurate association of time series data. This index-based data organization method effectively supports subsequent data processing.

[0054] This embodiment deeply optimizes the data normalization strategy. The system builds a normalization processing framework based on statistical features, effectively controlling data distribution through dynamic parameter adjustment. It pays special attention to the distribution characteristics of data and improves data comparability by designing an adaptive normalization mechanism. This comprehensive normalization approach provides a reliable data foundation for feature extraction.

[0055] This embodiment ensures data integrity by handling missing values. The system constructs a prediction framework based on exponential decay and interpolates adjacent data for supplementation. Special attention is paid to data continuity, and a weighted average mechanism is established to achieve accurate estimation of missing values. This systematic filling scheme provides continuous data support for subsequent analysis. This embodiment uses the outlier detection formula: Anomaly_Score = |x - μ| / (σ×k), where x represents the observed value, μ represents the mean, σ represents the standard deviation, and k is the adjustment coefficient.

[0056] The innovative design of this embodiment not only addresses data quality issues encountered in traditional methods but also establishes a sustainable and optimized preprocessing framework. Through multi-level data processing and quality control, the system is able to extract effective traffic features from complex raw data. This multi-source fusion-based processing mechanism ensures that the system maintains effective data quality even in complex traffic environments. This intelligent preprocessing solution significantly improves data availability in traffic status analysis.

[0057] This embodiment achieves intelligent analysis upgrades for traffic monitoring by establishing a complete data processing chain. The system dynamically adjusts processing strategies based on real-time data quality, avoiding the limitations of traditional fixed-rule solutions. Multi-dimensional data cleaning and quality control significantly improves data reliability and validity, providing reliable data support for subsequent traffic analysis. This intelligent processing mechanism demonstrates strong adaptability and optimization effects in traffic monitoring.

[0058] This embodiment not only improves data processing accuracy but also establishes a continuously evolving preprocessing system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate data support for subsequent analysis. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and analysis effectiveness, providing reliable technical support for traffic management.

[0059] In one embodiment of the multi-source data processing method of the present application, the following contents may also be specifically included: Step S301: construct a local time window for the data points marked as outliers, calculate the Euclidean distance of the data points in the window, determine a weight coefficient based on the distance value, perform a weighted sum of the weight coefficient and the data points in the window, perform smoothing correction on the outliers, perform sliding segmentation on the historical traffic sequence data based on a preset window length, construct the traffic data in each time window into a feature vector, and perform normalization on the feature vector to generate a feature matrix; Step S302: Establish a category mapping dictionary for the weather data, convert the weather type into a numerical identifier, establish a hash mapping table for the road section data, convert the road section number into a continuous numerical identifier, align the feature matrix, the weather data and the road section data in the time dimension, and divide the aligned data into a training data set and a verification data set according to a preset ratio.

[0060] Optionally, this embodiment innovatively designs a data optimization solution based on local weighting to address the technical difficulties in traffic data anomaly processing and feature construction. This embodiment first constructs an anomaly processing framework and realizes intelligent correction of traffic anomalies through multi-level data smoothing. The system designs a local weighted calculation formula: Smooth_Value = ∑(wi×xi) / ∑wi, where wi represents the weight coefficient of the i-th data point, xi represents the corresponding data value, and the weight coefficient wi is calculated by exp(-di^2 / h), di represents the distance from the target point, and h is the bandwidth parameter. In the traffic flow analysis scenario, this distance-based weighting method can effectively maintain the local characteristics of the data.

[0061] This embodiment deeply optimizes the exception handling strategy. In view of the time series characteristics of traffic flow data, a local processing mechanism based on time windows is designed. Through dynamic window division, accurate positioning and correction of abnormal data can be achieved. Special attention is paid to the continuity of the data. When an abnormal point is detected, the system will evaluate the similarity between data points by calculating the Euclidean distance of local data, and then determine the appropriate weight coefficient. For example, when dealing with sudden fluctuations in urban traffic flow, local weighted smoothing can effectively eliminate abnormal disturbances while maintaining the true trend of the data, which is crucial for building a reliable prediction model.

[0062] This embodiment innovatively implements a feature construction mechanism. In view of the temporal characteristics of traffic data, the system constructs a feature extraction framework based on a sliding window. By dividing the historical data into continuous time segments, the structured expression of historical data is achieved. Special attention is paid to the expressive power of the features, and the comparability of the features is ensured by designing an adaptive normalization strategy. This window-based feature construction method can effectively capture the temporal pattern of traffic flow. This embodiment adopts the feature normalization formula: X_norm = (X - μ) / σ, where X represents the original eigenvalue, μ represents the mean, and σ represents the standard deviation.

[0063] This embodiment deeply optimizes the data mapping strategy. The system builds a mapping and conversion framework based on categorical features, implementing encoding conversion for non-numeric features by establishing a mapping dictionary. Special attention is paid to encoding uniqueness, and a hash mapping mechanism is designed to improve the efficiency of feature conversion. This comprehensive mapping approach provides a unified data format for feature fusion.

[0064] This example achieves unified organization of multi-source data through time alignment. The system builds a timestamp-based alignment framework, integrating data with multi-dimensional features. It pays special attention to the temporal consistency of data and establishes a synchronization mechanism to achieve precise alignment of data from different sources. This systematic alignment solution provides a complete training sample for subsequent analysis.

[0065] The innovative design of this embodiment not only addresses the data anomaly issue in traditional methods but also establishes a continuously optimized feature learning framework. Through multi-level data processing and feature construction, the system is able to extract effective predictive features from complex traffic data. This local weighted processing mechanism ensures that the system maintains effective data quality even in complex traffic scenarios. In traffic flow analysis, this intelligent processing solution significantly improves prediction accuracy.

[0066] This embodiment achieves intelligent analysis upgrades for traffic forecasting by establishing a complete data processing chain. The system dynamically adjusts processing strategies based on real-time data characteristics, avoiding the limitations of traditional fixed rule-based solutions. Through multi-dimensional data optimization and feature construction, the quality and reliability of feature expression are significantly improved, providing reliable data support for subsequent traffic forecasting. This intelligent processing mechanism demonstrates strong adaptability and optimization effects in traffic flow analysis.

[0067] This embodiment not only improves data processing accuracy but also establishes an evolving feature learning system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system continuously improves as traffic patterns change, providing increasingly accurate feature representations for subsequent predictions. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction effectiveness, providing reliable technical support for traffic management.

[0068] In one embodiment of the multi-source data processing method of the present application, the following contents may also be specifically included: Step S401: constructing a multi-head attention feature extraction network based on the training data set, dividing the feature matrix into equal lengths according to the timestamps, performing linear transformation on the divided feature sequences using three independent weight matrices to generate a query vector sequence, a key vector sequence, and a value vector sequence, respectively, performing a matrix transposition operation on the query vector sequence and the key vector sequence, and performing a dot product operation on the transposed matrices to generate a feature correlation matrix; Step S402: Dimension scaling is performed on the feature correlation matrix, and the scaled matrix is ​​input into the softmax function for normalization to generate an attention weight matrix, where each element of the attention weight matrix represents the strength of association between features at different time steps. The attention weight matrix is ​​sparsified to filter out feature associations with weight values ​​lower than a threshold.

[0069] Optionally, this embodiment innovatively designs a feature learning solution based on multi-head attention to address the problems of insufficient extraction of time series features and inaccurate feature association modeling in traffic flow prediction. This embodiment first constructs a feature extraction framework and implements in-depth analysis of traffic characteristics through multi-level attention calculations. The system designs an attention calculation formula: Attention(Q,K,V) = softmax((QK^T) / √d)V, where Q represents the query vector, K represents the key vector, V represents the value vector, d represents the feature dimension, and √d is the scaling factor. In the traffic flow analysis scenario, this feature extraction method based on multi-head attention can effectively capture the dynamic correlation between time series features.

[0070] This embodiment deeply optimizes the feature segmentation strategy. In view of the time series characteristics of traffic flow data, a sequence processing mechanism based on equal-length segmentation is designed. Through precise time alignment, effective division of historical traffic characteristics is achieved. Special attention is paid to the continuity of the data. When a time series breakpoint is detected, the system will smooth it through a supplementary sampling method. For example, when analyzing the traffic flow on urban main roads, time series segmentation can accurately distinguish the characteristic patterns of different time periods such as morning and evening peaks and off-peak periods, which is crucial for building an accurate prediction model.

[0071] This embodiment innovatively implements a feature transformation mechanism. In view of the multi-dimensional attributes of traffic features, the system constructs a feature transformation framework based on a weight matrix. Through three independent linear transformations, multi-angle mapping of input features is achieved. Special attention is paid to the orthogonality of the transformation, and the diversity of feature expression is ensured by designing independent mapping spaces. This feature processing method based on multi-head mapping can effectively extract temporal features from different perspectives. This embodiment adopts the feature correlation evaluation formula: Correlation_Score = exp(QK^T / τ), where τ is the temperature parameter used to control the softness and hardness of attention distribution.

[0072] This embodiment deeply optimizes the attention calculation strategy. The system builds a feature correlation framework based on matrix operations and accurately calculates attention weights through dimensional scaling. It pays special attention to the distribution characteristics of weights and improves the stability of attention scores by designing a normalization mechanism. This comprehensive attention calculation method provides a reliable weight foundation for feature learning.

[0073] This example achieves optimized screening of feature associations through sparsification. The system constructs a threshold-based screening framework that combines weights for feature filtering. It pays special attention to feature importance and establishes an adaptive threshold mechanism to filter out invalid feature associations. This systematic screening approach provides high-quality feature representation for subsequent analysis.

[0074] The innovative design of this embodiment not only solves the feature extraction problem in traditional methods, but also establishes a sustainable and optimized feature learning framework. Through multi-level attention calculation and feature screening, the system is able to extract effective time series features from complex traffic data. This multi-head attention-based learning mechanism ensures that the system always maintains effective feature expression capabilities when faced with complex traffic scenarios. In traffic flow prediction, this intelligent feature extraction solution significantly improves prediction accuracy.

[0075] This embodiment achieves an intelligent upgrade for traffic flow prediction by establishing a complete feature processing chain. The system dynamically adjusts processing strategies based on real-time data characteristics, avoiding the limitations of traditional fixed feature solutions. Through multi-dimensional feature extraction and screening, the quality and reliability of feature expression are significantly improved, providing reliable feature support for subsequent traffic flow prediction. This intelligent processing mechanism demonstrates strong adaptability and optimization effects in traffic flow prediction.

[0076] This embodiment not only improves the accuracy of feature processing but also establishes an evolving learning system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate feature representations for subsequent predictions. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction results, providing reliable technical support for traffic management.

[0077] In one embodiment of the multi-source data processing method of the present application, the following contents may also be specifically included: Step S501: performing a matrix multiplication operation on the attention weight matrix and the value vector sequence to obtain a weighted feature representation, constructing a multilayer perceptron network for the weighted feature representation, performing a nonlinear transformation on the features through two fully connected layers, adding an activation function between the two fully connected layers to enhance the network's expressive power, and performing a residual connection between the transformed features and the original input features; Step S502: Calculate the mean and variance of the connected features, standardize the features based on the mean and the variance, introduce learnable scaling parameters and translation parameters to adaptively adjust the standardized features, generate a fused feature vector, perform dimensionality reduction mapping on the fused feature vector, construct a feature importance score vector, and screen the fused features based on the feature importance score vector.

[0078] Optionally, this embodiment innovatively designs an optimization solution based on deep feature transformation to address the technical difficulties in traffic feature fusion and optimization. This embodiment first constructs a feature fusion framework to achieve deep integration of traffic features through multi-level nonlinear transformations. The system designs a feature optimization evaluation formula: Feature_Score = α Attention_Weight +β Residual_Value + γ × Layer_Norm, where Attention_Weight represents the attention weight, Residual_Value represents the residual value, Layer_Norm represents the layer normalization value, and α, β, and γ are dynamic adjustment coefficients. In traffic flow analysis scenarios, this multi-dimensional feature optimization method can effectively improve the expressiveness of features.

[0079] This embodiment deeply optimizes the feature fusion strategy. In view of the complex correlation of traffic features, a feature fusion mechanism based on matrix multiplication is designed. Through the precise calculation of attention weights, dynamic weighting of value vector sequences is achieved. Special attention is paid to the importance distribution of features. When abnormal feature weights are detected, the system will optimize the fusion effect through an adaptive adjustment mechanism. For example, when analyzing traffic flow at urban intersections, feature fusion can effectively integrate traffic flow features from different directions and accurately capture the overall operating status of the intersection, which is crucial to improving prediction accuracy.

[0080] This embodiment innovatively implements a nonlinear transformation mechanism. In response to the expressiveness requirements of features, the system constructs a transformation framework based on a multi-layer perceptron. Through the deep transformation of two fully connected layers, a full exploration of the feature space is achieved. Special attention is paid to the expressiveness of the network, and the nonlinear characteristics of the feature transformation are ensured by designing an activation function strategy. This transformation method based on a deep network can effectively improve the expressiveness of features. This embodiment adopts the layer normalization formula: y =γ×(x-μ) / σ + β, where x represents the input feature, μ represents the mean, σ represents the standard deviation, and γ and β are learnable scaling and translation parameters, respectively.

[0081] This embodiment deeply optimizes the feature standardization strategy. The system builds a standardization processing framework based on statistical features, achieving precise control of feature distribution through dynamic parameter adjustment. Special attention is paid to the distribution characteristics of features, and a learnable parameter mechanism is designed to improve the adaptability of standardization. This comprehensive standardization approach provides a reliable processing foundation for feature optimization.

[0082] This example achieves dimensionality optimization through feature screening. The system constructs a feature selection framework based on importance scoring, combining it with dimensionality reduction mapping for feature compression. It pays special attention to the contribution of features and establishes a scoring mechanism to effectively screen feature dimensions. This systematic screening approach provides high-quality feature support for subsequent analysis.

[0083] The innovative design of this embodiment not only solves the feature fusion problem in traditional methods but also establishes a continuously optimized feature learning framework. Through multi-level feature transformation and optimization, the system is able to extract effective predictive features from complex traffic data. This deep learning-based optimization mechanism ensures that the system maintains effective feature expression capabilities even in complex traffic scenarios. In traffic flow analysis, this intelligent optimization solution significantly improves prediction accuracy.

[0084] This embodiment achieves an intelligent upgrade for traffic flow prediction by establishing a complete feature processing chain. The system dynamically adjusts optimization strategies based on real-time feature distribution, avoiding the limitations of traditional fixed-feature solutions. Through multi-dimensional feature transformation and optimization, the quality and reliability of feature expression are significantly improved, providing reliable feature support for subsequent traffic flow prediction. This intelligent optimization mechanism demonstrates strong adaptability and optimization effects in traffic flow analysis.

[0085] This embodiment not only improves the accuracy of feature processing but also establishes an evolving feature learning system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate feature representations for subsequent predictions. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction effectiveness, providing reliable technical support for traffic management.

[0086] In one embodiment of the multi-source data processing method of the present application, the following contents may also be specifically included: Step S601: Input the fused feature vector into a multi-layer fully connected neural network, which includes an input layer, a hidden layer, and an output layer. A weight matrix and a bias vector are constructed between adjacent layers. A linear transformation is performed on the input features. Batch normalization is used in the hidden layer to eliminate feature distribution offsets. Feature representations at different levels are fused through a skip connection structure. A dropout mechanism is introduced at the output of each hidden layer to prevent overfitting. Step S602: Use the modified linear unit function to perform nonlinear mapping on the transformed features, input the mapping result into the output layer to generate a predicted value, calculate the difference between the predicted value and the true value, square the difference and calculate the mean to obtain the mean square error loss value, construct an objective function based on the mean square error loss value, and calculate the gradient of the objective function with respect to the network parameters.

[0087] Optionally, this embodiment innovatively designs a prediction optimization solution based on deep neural networks to address the problems of insufficient model expression ability and poor training stability in traffic flow prediction. This embodiment first constructs a prediction model framework and implements deep modeling of traffic characteristics through multi-level feature transformation. The system designs an objective function evaluation formula: Loss = (1 / n)∑(yi - ŷi)^2 + λ×||W||^2, where yi represents the true value, ŷi represents the predicted value, n represents the number of samples, W represents the network weight matrix, and λ is the regularization coefficient. In the traffic flow prediction scenario, this loss calculation method based on mean square error can effectively guide the optimization direction of the model.

[0088] This embodiment deeply optimizes the network structure strategy. In response to the complexity of traffic characteristics, a deep architecture based on a multi-layer perceptron is designed. Through a carefully designed hierarchical structure, a full exploration of the feature space is achieved. Special attention is paid to the learning ability of the network. When a performance bottleneck is detected, the system will optimize the network structure by dynamically adjusting the layer width and depth. For example, when predicting traffic flow on urban arterial roads, a deep network can effectively model complex spatiotemporal dependencies, including various factors such as periodic changes and the impact of emergencies, which is crucial to improving prediction accuracy.

[0089] This embodiment innovatively implements a feature transformation mechanism. In response to the expression requirements of features, the system constructs a transformation framework based on a weight matrix. Through the combination of linear transformation and batch normalization, precise control of feature distribution is achieved. Special attention is paid to the statistical characteristics of the features, and the stability of the feature distribution is ensured by designing a normalization strategy. This feature processing method based on normalization can effectively improve the training efficiency of the model. This embodiment adopts the batch normalization formula: BN(x) = γ×(x-μB) / σB + β, where x represents the input feature, μB and σB represent the batch mean and standard deviation, respectively, and γ and β are learnable scaling and translation parameters.

[0090] This embodiment deeply optimizes the network training strategy. The system builds a gradient optimization framework based on the loss function, enabling precise adjustment of network parameters through backpropagation. Special attention is paid to training stability, and a dropout mechanism is designed to improve the model's generalization capabilities. This comprehensive training approach provides reliable technical support for model optimization.

[0091] This embodiment achieves multi-scale feature fusion through skip connections. The system builds a residual-based feature transfer framework, combining feature representations at different levels for comprehensive modeling. It pays special attention to feature complementarity and effectively utilizes feature information by establishing a multi-layer connection mechanism. This systematic fusion solution provides rich feature support for prediction tasks.

[0092] The innovative design of this embodiment not only solves the model optimization problem in traditional methods but also establishes a continuously optimized prediction framework. Through multi-level feature transformation and model optimization, the system is able to learn effective prediction patterns from complex traffic data. This deep learning-based prediction mechanism ensures that the system maintains effective prediction capabilities when faced with diverse traffic scenarios. In traffic flow prediction, this intelligent optimization solution significantly improves the accuracy and stability of predictions.

[0093] This embodiment achieves an intelligent forecast upgrade for traffic flow by establishing a complete forecast processing chain. The system dynamically adjusts optimization strategies based on real-time forecast results, avoiding the limitations of traditional fixed model solutions. Through multi-dimensional model optimization and training, the quality and reliability of forecasts are significantly improved, providing reliable data support for traffic management decisions. This intelligent forecasting mechanism demonstrates strong adaptability and optimization effects in traffic flow analysis.

[0094] This embodiment not only improves prediction accuracy but also establishes a continuously evolving prediction system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate prediction results for traffic management. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction effectiveness, providing reliable decision-making support for traffic management.

[0095] In one embodiment of the multi-source data processing method of the present application, the following contents may also be specifically included: Step S701: constructing a parameter update equation based on the objective function gradient, iteratively updating the network parameters using an adaptive moment estimation optimization algorithm, calculating the first-order moment estimation and second-order moment estimation of the parameters in each iteration, adaptively adjusting the learning rate based on historical gradient information, and performing gradient clipping on the updated parameters to prevent gradient explosion. Optimization is stopped when the objective function converges or a preset number of iterations is reached, and a traffic flow prediction model is generated. Step S702: Input the verification data set into the traffic flow prediction model in chronological order, calculate the mean absolute error and root mean square error of the prediction results, screen the optimal model parameters based on the verification indicators, apply the screened model to real-time traffic flow data, perform rolling predictions on traffic flows at different time scales, and output the prediction results.

[0096] Optionally, this embodiment innovatively designs a model training scheme based on adaptive optimization to address problems such as unstable model convergence and unreliable prediction results in traffic flow prediction. This embodiment first constructs a parameter optimization framework and achieves precise optimization of the prediction model through multi-level gradient adjustment. The system designs a parameter update formula: θt = θt-1- η×mt / √(vt + ε), where θt represents the current moment parameter, mt represents the first-order moment estimate, vt represents the second-order moment estimate, η represents the basic learning rate, and ε is the stability coefficient. In the traffic flow prediction scenario, this optimization method based on adaptive moment estimation can effectively improve the training efficiency of the model.

[0097] This embodiment deeply optimizes the parameter update strategy. In view of the complexity of the traffic prediction model, an adaptive optimization mechanism based on moment estimation is designed. By accumulating historical gradient information, dynamic adjustment of the learning rate is achieved. Special attention is paid to the stability of the optimization. When a gradient anomaly is detected, the system will effectively control it through the gradient clipping method. For example, when optimizing the urban traffic flow prediction model, the adaptive learning rate can effectively handle the differences in traffic changes in different time periods, including slow changes in the off-peak period and violent fluctuations in the peak period, which is crucial to improving the adaptability of the model.

[0098] This embodiment innovatively implements a convergence judgment mechanism. In view of the termination conditions of model training, the system constructs a convergence evaluation framework based on multiple indicators. By monitoring the changing trend of the loss function, precise control of the training process is achieved. Special attention is paid to the stability of convergence, and by designing an early stopping strategy, it is ensured that the model stops training in the optimal state. This multi-dimensional convergence judgment method can effectively avoid the overfitting problem. This embodiment adopts the verification evaluation formula: Score = α MAE +β RMSE, where MAE stands for mean absolute error, RMSE stands for root mean square error, and α and β are weight coefficients.

[0099] This embodiment deeply optimizes the model validation strategy. The system builds a model evaluation framework based on the validation dataset, comprehensively evaluating prediction effectiveness through multi-dimensional performance indicators. Special attention is paid to prediction stability, and a rolling forecast mechanism is designed to improve the model's reliability in long-term forecasts. This comprehensive validation approach provides a reliable evaluation basis for model selection.

[0100] This example optimizes model performance through parameter screening. The system constructs a parameter selection framework based on validation metrics and combines it with multi-scale predictions for comprehensive evaluation. Special attention is paid to the model's generalization capabilities, and a multi-timescale testing mechanism is established to fully verify the model's stability. This systematic screening approach provides high-quality prediction models for practical applications.

[0101] The innovative design of this embodiment not only solves the optimization issues inherent in traditional methods but also establishes a sustainable optimization training framework. Through multi-level parameter optimization and model validation, the system is able to learn effective prediction models from complex traffic data. This adaptive optimization-based training mechanism ensures that the system maintains effective prediction capabilities in diverse traffic scenarios. In traffic flow forecasting, this intelligent optimization solution significantly improves prediction accuracy and stability.

[0102] This embodiment achieves an intelligent forecast upgrade for traffic flow by establishing a complete optimization and verification chain. The system dynamically adjusts optimization strategies based on real-time prediction results, avoiding the limitations of traditional fixed model solutions. Through multi-dimensional model optimization and verification, the quality and reliability of forecasts are significantly improved, providing reliable data support for traffic management decisions. This intelligent optimization mechanism demonstrates strong adaptability and optimization effects in traffic flow analysis.

[0103] This embodiment not only improves prediction accuracy but also establishes a continuously evolving prediction system through continuous strategy optimization and performance analysis. This optimization mechanism, based on real-time feedback, ensures that the system can continuously improve as traffic patterns change, providing increasingly accurate prediction results for traffic management. In practical applications, this self-optimization mechanism has significantly improved the system's long-term service quality and prediction effectiveness, providing reliable decision-making support for traffic management.

[0104] In order to effectively solve the deficiencies of traditional technologies in data processing, feature extraction, and model optimization, and significantly improve the accuracy and reliability of traffic flow prediction, the present application provides an embodiment of a multi-source data processing device for implementing all or part of the multi-source data processing method, see Figure 2 , the multi-source data processing device specifically includes the following contents: A data set determination module 10 is configured to collect historical traffic sequence data from a traffic monitoring device, obtain weather data from a meteorological monitoring station, and obtain road segment data from a road network management system; fill missing values ​​in the historical traffic sequence data using an exponential smoothing method; supplement missing values ​​in the weather data and the road segment data using linear interpolation of adjacent time data; calculate the mean and standard deviation of the historical traffic sequence data; mark data that deviates from the mean by more than a multiple of the standard deviation as outliers; smooth out the outliers using a local weighted regression algorithm; construct a feature matrix from the historical traffic sequence data based on a sliding window method; perform numerical encoding conversion on the weather data and the road segment data; and divide the preprocessed data into a training data set and a validation data set according to time series; A feature fusion module 20 is configured to construct an attention feature extraction network based on the training data set, segment the feature matrix according to timestamps, generate a query vector sequence, a key vector sequence, and a value vector sequence, calculate a dot product matrix between the query vector sequence and the key vector sequence, normalize the dot product matrix to generate an attention weight matrix, multiply the attention weight matrix by the value vector sequence to obtain a weighted feature representation, perform linear transformation and layer normalization on the weighted feature representation, and generate a fused feature vector; The traffic prediction module 30 is used to input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

[0105] As can be seen from the above description, the multi-source data processing device provided in the embodiment of the present application can realize the correction of abnormal missing data from multiple sources through innovative construction of data preprocessing mechanism, exponential smoothing and local regression. Design an attention-based feature extraction network, combine query-key-value calculation and normalization processing, and establish a fusion representation strategy for time series features. Introduce a deep optimization mechanism, and realize accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction.

[0106] From a hardware perspective, in order to effectively address the deficiencies of traditional technologies in data processing, feature extraction, and model optimization, and significantly improve the accuracy and reliability of traffic flow prediction, this application provides an embodiment of an electronic device for implementing all or part of the multi-source data processing method. The electronic device specifically includes the following: A processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to transmit information between the multi-source data processing device and related devices such as core business systems, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiments of the multi-source data processing method and the multi-source data processing device in the embodiments, and their contents are incorporated herein, and any repetitions are not repeated.

[0107] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0108] In practical applications, portions of the multi-source data processing method may be executed on the electronic device side as described above, or all operations may be performed on the client device. The specific selection may be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not impose any restrictions on this. If all operations are performed on the client device, the client device may also include a processor.

[0109] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0110] Figure 3 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0111] In one embodiment, the multi-source data processing method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control: Step S101: historical traffic sequence data is collected from a traffic monitoring device, weather data is obtained from a meteorological monitoring station, and road section data is obtained from a road network management system. Missing values ​​in the historical traffic sequence data are filled using an exponential smoothing method. Missing values ​​in the weather data and the road section data are supplemented using linear interpolation of adjacent time data. The mean and standard deviation of the historical traffic sequence data are calculated. Data that deviates from the mean by more than a multiple of the standard deviation is marked as an outlier. The outliers are smoothed using a local weighted regression algorithm. A feature matrix is ​​constructed from the historical traffic sequence data based on a sliding window method. The weather data and the road section data are converted into numerical codes. The preprocessed data is divided into a training data set and a validation data set according to time series. Step S102: constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps, generating a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; Step S103: Input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

[0112] As can be seen from the above description, the electronic device provided in the embodiment of the present application realizes the correction of abnormal missing data from multiple sources through innovatively constructing a data preprocessing mechanism, through exponential smoothing and local regression. An attention-based feature extraction network is designed, and a fusion representation strategy for time series features is established by combining query-key-value calculation and normalization processing. A deep optimization mechanism is introduced to achieve accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction.

[0113] In another embodiment, the multi-source data processing device can be configured separately from the central processing unit 9100. For example, the multi-source data processing device can be configured as a chip connected to the central processing unit 9100, and the multi-source data processing method function can be implemented under the control of the central processing unit.

[0114] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 3 In addition, the electronic device 9600 may also include all components shown in Figure 3 For components not shown, reference may be made to the prior art.

[0115] like Figure 3 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0116] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0117] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0118] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0119] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0120] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0121] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless local area network modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130, providing audio output via the speaker 9131 and receiving audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0122] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the multi-source data processing method in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program implements all steps of the multi-source data processing method in the above-mentioned embodiment, where the execution subject is a server or a client. For example, when the processor executes the computer program, the following steps are implemented: Step S101: historical traffic sequence data is collected from a traffic monitoring device, weather data is obtained from a meteorological monitoring station, and road section data is obtained from a road network management system. Missing values ​​in the historical traffic sequence data are filled using an exponential smoothing method. Missing values ​​in the weather data and the road section data are supplemented using linear interpolation of adjacent time data. The mean and standard deviation of the historical traffic sequence data are calculated. Data that deviates from the mean by more than a multiple of the standard deviation is marked as an outlier. The outliers are smoothed using a local weighted regression algorithm. A feature matrix is ​​constructed from the historical traffic sequence data based on a sliding window method. The weather data and the road section data are converted into numerical codes. The preprocessed data is divided into a training data set and a validation data set according to time series. Step S102: constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps, generating a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; Step S103: Input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

[0123] As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application realizes the correction of abnormal missing data from multiple sources through innovatively constructing a data preprocessing mechanism, through exponential smoothing and local regression. An attention-based feature extraction network is designed, and a fusion representation strategy for time series features is established by combining query-key-value calculation and normalization processing. A deep optimization mechanism is introduced to achieve accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction, and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction.

[0124] The embodiments of the present application also provide a computer program product capable of implementing all steps of the multi-source data processing method in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instructions are executed by a processor, the steps of the multi-source data processing method are implemented. For example, the computer program / instructions implement the following steps: Step S101: historical traffic sequence data is collected from a traffic monitoring device, weather data is obtained from a meteorological monitoring station, and road section data is obtained from a road network management system. Missing values ​​in the historical traffic sequence data are filled using an exponential smoothing method. Missing values ​​in the weather data and the road section data are supplemented using linear interpolation of adjacent time data. The mean and standard deviation of the historical traffic sequence data are calculated. Data that deviates from the mean by more than a multiple of the standard deviation is marked as an outlier. The outliers are smoothed using a local weighted regression algorithm. A feature matrix is ​​constructed from the historical traffic sequence data based on a sliding window method. The weather data and the road section data are converted into numerical codes. The preprocessed data is divided into a training data set and a validation data set according to time series. Step S102: constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps, generating a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; Step S103: Input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

[0125] As can be seen from the above description, the computer program product provided in the embodiment of the present application realizes the correction of abnormal missing data from multiple sources through innovative construction of a data preprocessing mechanism, through exponential smoothing and local regression. An attention-based feature extraction network is designed, and a fusion representation strategy for time series features is established by combining query-key-value calculation and normalization processing. A deep optimization mechanism is introduced to achieve accurate training of the prediction model through nonlinear transformation and gradient descent. This method effectively solves the shortcomings of traditional technologies in data processing, feature extraction and model optimization, and significantly improves the accuracy and reliability of traffic flow prediction.

[0126] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0130] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A multi-source data processing method, characterized in that: The method comprises: Collect historical traffic sequence data from traffic monitoring devices, obtain weather data from meteorological monitoring stations, and obtain road section data from road network management systems; fill missing values ​​in the historical traffic sequence data using exponential smoothing; supplement missing values ​​in the weather data and the road section data using linear interpolation of adjacent time data; calculate the mean and standard deviation of the historical traffic sequence data; mark data that deviates from the mean by more than a multiple of the standard deviation as outliers; smooth the outliers using a local weighted regression algorithm; construct a feature matrix from the historical traffic sequence data based on a sliding window method; perform numerical encoding conversion on the weather data and the road section data; and divide the preprocessed data into a training data set and a validation data set according to time series; Constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps to generate a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; The fused feature vector is input into a fully connected neural network, and nonlinear transformation and dimensionality reduction processing are performed on the features. An activation function is used to perform nonlinear mapping on the transformed features. The prediction error is calculated based on the mean square error loss function. The network parameters are iteratively optimized using the gradient descent method to generate a traffic flow prediction model. The verification data set is input into the traffic flow prediction model for prediction verification, and the traffic flow prediction result is output.

2. The multi-source data processing method according to claim 1, characterized in that: The method includes collecting historical traffic sequence data from a traffic monitoring device, obtaining weather data from a meteorological monitoring station, and obtaining road section data from a road network management system, filling missing values ​​in the historical traffic sequence data using an exponential smoothing method, supplementing missing values ​​in the weather data and the road section data using linear interpolation of adjacent time data, calculating the mean and standard deviation of the historical traffic sequence data, and marking data that deviates from the mean by more than a multiple of the standard deviation as an outlier, including: Based on the data acquisition interface, historical traffic sequence data, weather data, and road section data are respectively obtained from the traffic monitoring device, the meteorological monitoring station, and the road network management system, the historical traffic sequence data, the weather data, and the road section data are marked according to timestamps, a time series index table is constructed, the format of the historical traffic sequence data is converted and the data is standardized, and the weather data and the road section data are mapped to the time series index table; Missing values ​​in the historical traffic sequence data are filled with predicted values ​​calculated based on the exponential decay coefficient, and missing values ​​in the weather data and the road section data are supplemented by the weighted average of the data at adjacent moments. The statistical characteristics of the historical traffic sequence data, including the mean, standard deviation and coefficient of variation, are calculated, and an outlier identification threshold is set based on the statistical characteristics. Data points that deviate from the mean by more than a multiple of the standard deviation are marked as outliers.

3. The multi-source data processing method according to claim 1, characterized in that: The method adopts a local weighted regression algorithm to smooth outliers, constructs a feature matrix from the historical traffic sequence data based on a sliding window method, performs numerical coding conversion on the weather data and the road section data, and divides the preprocessed data into a training data set and a validation data set according to the time series, including: Construct a local time window for data points marked as outliers, calculate the Euclidean distance of the data points in the window, determine a weight coefficient based on the distance value, perform a weighted sum of the weight coefficient and the data points in the window, perform smoothing correction on the outliers, perform sliding segmentation on the historical traffic sequence data based on a preset window length, construct the traffic data in each time window into a feature vector, and normalize the feature vector to generate a feature matrix; A category mapping dictionary is established for the weather data, and the weather type is converted into a numerical identifier. A hash mapping table is established for the road section data, and the road section number is converted into a continuous numerical identifier. The feature matrix, the weather data and the road section data are aligned in the time dimension, and the aligned data are divided into a training data set and a validation data set according to a preset ratio.

4. The multi-source data processing method according to claim 1, characterized in that: The attention feature extraction network is constructed based on the training data set, the feature matrix is ​​segmented according to the timestamp, a query vector sequence, a key vector sequence, and a value vector sequence are generated, a dot product matrix of the query vector sequence and the key vector sequence is calculated, and the dot product matrix is ​​normalized to generate an attention weight matrix, including: A multi-head attention feature extraction network is constructed based on the training data set, the feature matrix is ​​divided into equal lengths according to the timestamps, a linear transformation is performed on the divided feature sequence using three independent weight matrices to generate a query vector sequence, a key vector sequence, and a value vector sequence, respectively, a matrix transposition operation is performed on the query vector sequence and the key vector sequence, and a dot product operation is performed on the transposed matrices to generate a feature correlation matrix; The feature correlation matrix is ​​dimensionally scaled, and the scaled matrix is ​​input into the softmax function for normalization to generate an attention weight matrix, where each element of the attention weight matrix represents the correlation strength between features at different time steps. The attention weight matrix is ​​sparsified to filter out feature associations with weight values ​​below a threshold.

5. The multi-source data processing method according to claim 1, characterized in that: The multiplication of the attention weight matrix and the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization processing on the weighted feature representation to generate a fused feature vector, including: Performing a matrix multiplication operation on the attention weight matrix and the value vector sequence to obtain a weighted feature representation, constructing a multilayer perceptron network for the weighted feature representation, performing a nonlinear transformation on the features through two fully connected layers, adding an activation function between the two fully connected layers to enhance the expressive power of the network, and performing a residual connection between the transformed features and the original input features; The mean and variance of the concatenated features are calculated, the features are standardized based on the mean and the variance, learnable scaling parameters and translation parameters are introduced to adaptively adjust the standardized features, a fused feature vector is generated, a dimensionality reduction mapping is performed on the fused feature vector, a feature importance score vector is constructed, and the fused features are screened based on the feature importance score vector.

6. The multi-source data processing method according to claim 1, characterized in that: The step of inputting the fused feature vector into a fully connected neural network, performing nonlinear transformation and dimensionality reduction processing on the features, performing nonlinear mapping on the transformed features using an activation function, and calculating the prediction error based on a mean square error loss function includes: Inputting the fused feature vector into a multi-layer fully connected neural network, the fully connected neural network includes an input layer, a hidden layer, and an output layer, constructing a weight matrix and a bias vector between adjacent layers, performing a linear transformation on the input features, applying batch normalization to the hidden layer to eliminate feature distribution offsets, fusing feature representations of different levels through a skip connection structure, and introducing a dropout mechanism at the output of each hidden layer to prevent overfitting; A modified linear unit function is used to perform nonlinear mapping on the transformed features, and the mapping result is input into the output layer to generate a predicted value. The difference between the predicted value and the true value is calculated, and the difference is squared and the mean is obtained to obtain the mean square error loss value. The objective function is constructed based on the mean square error loss value, and the gradient of the objective function with respect to the network parameters is calculated.

7. The multi-source data processing method according to claim 1, characterized in that: The method of iteratively optimizing network parameters using a gradient descent method to generate a traffic flow prediction model, inputting the verification data set into the traffic flow prediction model for prediction verification, and outputting a traffic flow prediction result includes: Based on the objective function gradient, a parameter update equation is constructed, and an adaptive moment estimation optimization algorithm is used to iteratively update the network parameters. In each iteration, the first-order moment estimation and second-order moment estimation of the parameters are calculated. The learning rate is adaptively adjusted based on historical gradient information. The updated parameters are gradient-clipped to prevent gradient explosion. When the objective function converges or a preset number of iterations is reached, the optimization is stopped to generate a traffic flow prediction model. The validation data set is input into the traffic flow prediction model in chronological order, the mean absolute error and root mean square error of the prediction results are calculated, the optimal model parameters are screened based on the validation indicators, the screened model is applied to real-time traffic flow data, rolling predictions are made for traffic flows at different time scales, and the prediction results are output.

8. A multi-source data processing device, characterized in that: The device comprises: a data set determination module, configured to collect historical traffic sequence data from a traffic monitoring device, obtain weather data from a meteorological monitoring station, and obtain road section data from a road network management system; fill missing values ​​in the historical traffic sequence data using an exponential smoothing method; supplement missing values ​​in the weather data and the road section data using linear interpolation of adjacent time data; calculate the mean and standard deviation of the historical traffic sequence data; mark data that deviates from the mean by more than a multiple of the standard deviation as outliers; smooth the outliers using a local weighted regression algorithm; construct a feature matrix from the historical traffic sequence data based on a sliding window method; perform numerical encoding conversion on the weather data and the road section data; and divide the preprocessed data into a training data set and a validation data set according to time series; a feature fusion module for constructing an attention feature extraction network based on the training data set, segmenting the feature matrix according to timestamps to generate a query vector sequence, a key vector sequence, and a value vector sequence, calculating a dot product matrix of the query vector sequence and the key vector sequence, normalizing the dot product matrix to generate an attention weight matrix, multiplying the attention weight matrix by the value vector sequence to obtain a weighted feature representation, performing linear transformation and layer normalization on the weighted feature representation to generate a fused feature vector; The traffic flow prediction module is used to input the fused feature vector into a fully connected neural network, perform nonlinear transformation and dimensionality reduction on the features, use an activation function to perform nonlinear mapping on the transformed features, calculate the prediction error based on the mean square error loss function, use the gradient descent method to iteratively optimize the network parameters, generate a traffic flow prediction model, input the verification data set into the traffic flow prediction model for prediction verification, and output the traffic flow prediction result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multi-source data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-source data processing method according to any one of claims 1 to 7 are implemented.

Images