A low-latency edge collaborative data processing method and system
By parallel processing and edge box deployment of vehicle traffic prediction networks on multiple cloud platforms, the problem of poor data transmission delay and real-time performance in vehicle traffic prediction is solved, efficient and low-latency vehicle traffic prediction is achieved, and multiple cloud platforms are used to achieve high-quality and low-latency vehicle traffic prediction. Coordinated learning between them protects data privacy.
Patent Information
- Application Number
- CN202411132664.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing technology has problems of high data transmission delay and poor real-time performance in vehicle traffic prediction. It is difficult for pure edge computing or cloud computing to handle complex data analysis tasks, and there is a lack of in-depth exploration of collaborative work between multiple cloud platforms. Protecting data privacy while making full use of data and computing resources of various cloud platforms is also an urgent problem to be solved.
A low-latency edge collaborative data processing method is proposed. By acquiring high-speed monitoring data, grouping and performing data feature extraction and network debugging on multiple cloud platforms, using the computing resources of multiple cloud platforms for parallel processing, realizing vehicle traffic prediction network Debugging and deploying into edge boxes for real-time prediction with low latency.
It reduces the computing power cost of the vehicle traffic prediction network, improves the accuracy and real-timeness of the prediction results, ensures collaborative learning and data privacy protection between multiple cloud platforms, and achieves efficient and low-latency vehicle traffic prediction.
Smart Images

Figure CN118658303B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of edge computing and machine learning, and specifically, to a low-latency edge collaborative data processing method and system. Background Art
[0002] With the continuous increase in highway traffic volume, the demand for real-time and accurate vehicle flow prediction is becoming increasingly urgent. Traditional vehicle flow prediction methods often rely on centralized data processing systems. However, this centralized processing method faces problems such as high data transmission delay and poor real-time performance, and it is difficult to meet the requirements of modern traffic management for low latency and high efficiency. In recent years, with the rapid development of cloud computing and edge computing technologies, new solutions have been provided for vehicle flow prediction. Cloud computing can process large-scale data sets through powerful computing power and massive storage resources, but its latency problem limits its application in scenarios with high real-time requirements. Edge computing pushes computing tasks to the edge of the network and uses computing resources closer to the data source for data processing and analysis, thereby reducing data transmission delays and improving real-time performance.
[0003] However, pure edge computing or cloud computing solutions still have limitations in the field of vehicle flow prediction. Although edge computing can achieve low-latency processing, its computing resources are relatively limited and it is difficult to handle complex data analysis tasks; cloud computing faces the problems of data transmission delay and lack of real-time performance. Therefore, how to combine the advantages of cloud computing and edge computing to achieve efficient and low-latency vehicle flow prediction has become a hot topic in current research. At present, some studies have attempted to improve the performance of vehicle flow prediction through the collaboration of cloud platforms. However, most of these studies focus on data processing and model training within a single cloud platform, and lack in-depth exploration of the collaboration between multiple cloud platforms. In addition, how to make full use of the data and computing resources of each cloud platform while protecting data privacy is also an urgent problem to be solved. Summary of the invention
[0004] The object of the present invention is to provide a low-latency edge collaborative data processing method and system. The present invention is implemented as follows:
[0005] In a first aspect, an embodiment of the present application provides a low-latency edge collaborative data processing method, including: obtaining X high-speed monitoring data; grouping the X high-speed monitoring data to obtain P high-speed monitoring data sets, wherein P is a positive integer greater than 1 and P is less than X; performing data feature extraction operations on the P high-speed monitoring data sets through P cloud platforms respectively to obtain a vehicle flow feature representation sample set of each of the P cloud platforms; adjusting the network internal parameter variables in each of the P cloud platforms according to the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms, and performing the F-th round of repeated debugging on the adjusted network internal parameter variables in each of the cloud platforms according to the vehicle flow feature representation sample set of each of the cloud platforms to obtain the debugged P a target vehicle flow prediction network, wherein F=E+1; deploying the target vehicle flow prediction network corresponding to each cloud platform to the corresponding edge box so that the edge box performs target vehicle flow prediction; wherein each cloud platform is communicatively connected to at least one edge box, and for each edge box, when performing target vehicle flow prediction, it includes: acquiring target highway monitoring data; performing data feature extraction operation on the target highway monitoring data to obtain a target vehicle flow feature representation of the target highway monitoring data; based on the target vehicle flow feature representation of the target highway monitoring data, classifying the target highway monitoring data through the deployed corresponding target vehicle flow prediction network to obtain a target vehicle flow prediction result of the target highway monitoring data.
[0006] In a second aspect, the present application provides a low-latency edge collaborative data processing method, which is applied to an edge box, and the edge box is communicatively connected to a target cloud platform. The method includes: obtaining target highway monitoring data; performing data feature extraction operations on the target highway monitoring data to obtain a target vehicle flow feature representation of the target highway monitoring data; based on the target vehicle flow feature representation of the target highway monitoring data, classifying the target highway monitoring data through a target vehicle flow prediction network in the target cloud platform to obtain a target vehicle flow prediction result of the target highway monitoring data, wherein the target vehicle flow prediction network is debugged based on the method provided in the first aspect above.
[0007] In a third aspect, the present application provides an edge collaborative data processing system, comprising multiple edge boxes and multiple cloud platforms, wherein the multiple edge boxes and the multiple cloud platforms are communicatively connected, wherein the cloud platform comprises: one or more processors; a memory; one or more computer programs; wherein one or more of the computer programs are stored in the memory and configured to be executed by one or more of the processors, and when one or more of the computer programs are executed by the processors, the method provided in the first aspect above is implemented.
[0008] The present application at least includes the following beneficial effects: The present application provides a low-latency edge collaborative data processing method and system. During the network debugging process, data feature extraction operations are performed on P high-speed monitoring data sets through P cloud platforms respectively, and the vehicle flow feature representation sample sets of each cloud platform are repeatedly debugged through each cloud platform to debug the target vehicle flow prediction network. That is, the computing power consumption relied on for network debugging is dispersed based on each cloud platform, so that the idle computing power of each cloud platform can be effectively used, so that the computing power cost of debugging P target vehicle flow prediction networks is reduced. In addition, the computing power and storage requirements for each repeated debugging are reduced. The network internal parameters in each cloud platform are adjusted according to the P network internal parameters obtained after the E-th round of repeated debugging in the P cloud platforms, and the adjusted network internal parameters in each cloud platform are debugged for the F-th round according to the sample set representing the vehicle flow characteristics of each cloud platform to obtain P debugged target vehicle flow prediction networks. That is, in the F-th round of repeated debugging, the federated learning of the P target vehicle flow prediction networks is completed based on the P network internal parameters obtained after the E-th round of repeated debugging in the shared P cloud platforms, thereby ensuring the calibration quality of the P target vehicle flow prediction networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.
[0010] Figure 1 It is a schematic diagram of the architecture of the edge collaborative data processing system provided in an embodiment of the present application.
[0011] Figure 2 This is a flowchart of a low-latency edge collaborative data processing method provided in an embodiment of the present application.
[0012] Figure 3 It is a schematic diagram of the composition of a cloud platform provided in an embodiment of the present application.
[0013] Reference numerals: 101 - processor; 102 - bus; 103 - memory; 104 - transceiver; 105 - edge box; 106 - cloud platform. DETAILED DESCRIPTION
[0014] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0015] The low-latency edge collaborative data processing method provided in the embodiment of the present application can be applied to Figure 1In the cloud platform and edge box of the edge collaborative data processing system shown. Among them, the edge box 105 communicates with the cloud platform 106 through the network. The data storage system can store the data that the cloud platform 106 needs to process when training the neural network. The data storage system can be integrated on the cloud platform 106. Among them, in the neural network reasoning stage, the high-speed monitoring data can be stored in the local storage of the edge box 105, or in the data storage system or the cloud storage associated with the cloud platform 106. It can be understood that the edge box 105 is an intelligent device installed along the road or at key nodes as part of the intelligent transportation system. These edge boxes have data processing, communication and certain intelligent analysis capabilities, and can collect and process traffic data from various sensors in real time, such as traffic flow, vehicle speed, road occupancy, etc., and upload the processed data to the cloud or send it to other related systems. Specifically, the edge box can integrate hardware components such as multiple sensor interfaces, data processing chips, communication modules, and corresponding software and algorithms. They are usually deployed close to the data source so that they can respond and process data faster and reduce data transmission delays. The cloud platform 106 can be implemented using an independent server or a server cluster consisting of multiple servers.
[0016] The present application embodiment provides a low-latency edge collaborative data processing method, which is applied to a cloud platform, such as Figure 2 As shown, the method includes:
[0017] The network training phase includes steps S10 to S40:
[0018] Step S10: Acquire X high-speed monitoring data.
[0019] In step S10, the cloud platform obtains X high-speed monitoring data from the highway monitoring system. These high-speed monitoring data are collected in real time and are used to reflect the traffic conditions of the highway. Specifically, these data usually come from various sensors and devices installed on the highway, such as traffic flow sensors, speed detectors, cameras, etc. Taking a typical highway edge collaborative data processing scenario as an example, assume that there are multiple monitoring points on a highway, and each monitoring point is equipped with a series of sensors to collect data. These sensors include, for example, infrared sensors for measuring the number of vehicles passing, radar speed meters for recording vehicle speeds, and high-definition cameras for capturing real-time traffic images. These devices will continuously generate a large amount of raw data, which contains rich traffic information. In step S10, the cloud platform is connected to these sensors and devices, and obtains these high-speed monitoring data in real time through the data acquisition interface. These data may include the number of vehicles passing per minute, average speed, vehicle type distribution, road occupancy, etc. In addition, if the camera captures real-time traffic images, these image data will also be collected for subsequent data analysis and processing.
[0020] Assuming X is 10,000, the cloud platform obtains these 10,000 highway monitoring data samples. These data samples may be recorded in the form of time series, and each sample contains traffic data within a period of time. By obtaining this data, the cloud platform can build a huge data set for subsequent model training and vehicle flow prediction.
[0021] Step S20: Grouping the X high-speed monitoring data to obtain P high-speed monitoring data sets, where P is a positive integer greater than 1 and P is less than X.
[0022] In step S20, the cloud platform groups the acquired X high-speed monitoring data to form P high-speed monitoring data sets. The purpose of this step is to split the huge data set into multiple smaller subsets so that they can be processed in parallel on multiple cloud platforms, thereby improving data processing efficiency.
[0023] Taking highway edge collaborative data processing as an example, assuming that X is 10,000, this means that the cloud platform has collected 10,000 highway monitoring data samples. These data may come from different highway sections, different time periods, or different monitoring equipment. Due to the large amount of data, processing directly on a cloud platform may face resource bottlenecks and efficiency issues. Therefore, in step S20, the cloud platform divides these 10,000 data samples into P smaller data sets according to a certain strategy. P is a positive integer greater than 1 and less than X, and the specific value can be determined according to actual conditions. For example, if the cloud platform has 5 available cloud platforms, the data can be divided into 5 data sets (ie, P=5), each containing 2,000 data samples.
[0024] The grouping strategy can be formulated according to specific application requirements and data characteristics. A common strategy is to group by highway section, that is, to divide the data samples of the same section into the same data set. This ensures that each data set reflects the traffic conditions of the same section, which is conducive to subsequent data analysis and model training. In addition, you can also consider grouping by time period, such as dividing the data of a day by hours, and forming a data set for each hour. The advantage of this is that it can capture the changes in traffic flow in different time periods, which is helpful for analyzing the periodicity of traffic flow.
[0025] Once the grouping strategy is determined, the cloud platform will group X high-speed monitoring data according to the strategy to generate P high-speed monitoring data sets. Each data set contains a portion of the original data samples and maintains the integrity and accuracy of the original data. Next, these data sets will be sent to different cloud platforms for subsequent data processing and model training. It should be noted that during the grouping process, it is necessary to ensure that each data set is representative, that is, it can reflect certain characteristics of the overall data. At the same time, the balance of the data set must also be considered to avoid a data set that is too large or too small, resulting in uneven resource allocation in the subsequent processing process. Through a reasonable grouping strategy, the advantages of parallel processing on multiple cloud platforms can be maximized to improve the efficiency of the entire data processing process.
[0026] Step S30: performing data feature extraction operations on the P high-speed monitoring data sets respectively through the P cloud platforms to obtain a sample set representing the vehicle flow feature of each of the P cloud platforms.
[0027] In step S30, the cloud platform will use P cloud platforms to perform parallel data feature extraction operations on P highway monitoring data sets to obtain a set of vehicle flow feature representation samples corresponding to each cloud platform. This step is one of the key steps in building a vehicle flow prediction model because it involves extracting meaningful information from the original highway monitoring data, which will be used for subsequent model training.
[0028] Specifically, assuming P is 5, it means that 5 cloud platforms will participate in the data feature extraction process. Each cloud platform will receive a data set containing 2,000 highway monitoring data samples (assuming that the original data set X is evenly divided into P data sets). These highway monitoring data samples may include various types of real-time information such as traffic flow, vehicle speed, road occupancy, weather conditions, etc.
[0029] On each cloud platform, data feature extraction operations will be implemented by applying specific machine learning algorithms or models. For example, convolutional neural networks (CNN) or recurrent neural networks (RNN) can be used to extract visual features related to vehicle flow from image data, or algorithms such as decision trees, random forests, or gradient boosting machines can be used to extract statistical features from numerical data. These features may include time series changes in the number of vehicles, the distribution of average vehicle speeds, road congestion index, etc.
[0030] For example, if a cloud platform receives a dataset containing camera images, it can use a pre-trained CNN model to extract visual features such as the number of vehicles and vehicle density in the image. These features will be encoded as numerical vectors, each of which represents the vehicle flow at a point in time. Similarly, if the dataset contains numerical data on vehicle speed and traffic flow, the cloud platform can calculate the statistical features of these data, such as mean, median, standard deviation, etc., as well as the characteristics of changes in time series, such as growth rate, periodicity, etc. Through this process, each cloud platform will generate a sample set of vehicle flow feature representations. This set contains a series of feature vectors, each of which corresponds to a high-speed monitoring data sample and contains meaningful information extracted from the original data. These feature vectors will be used as input for subsequent model training to learn the mapping relationship from features to vehicle flow.
[0031] By performing data feature extraction operations on P cloud platforms in parallel, the cloud platform can make full use of the computing resources of each cloud platform and accelerate the entire data processing process. At the same time, since each cloud platform processes an independent data set, it can also improve the diversity of data and the generalization ability of the model to a certain extent.
[0032] Step S40: adjusting the network internal parameters in each of the cloud platforms according to the P network internal parameters obtained after completing the E-th round of repeated debugging in the P cloud platforms, and performing the F-th round of repeated debugging on the adjusted network internal parameters in each of the cloud platforms according to the vehicle flow characteristic representation sample set of each of the cloud platforms to obtain P debugged target vehicle flow prediction networks, where F=E+1.
[0033] In step S40, the cloud platform will perform an iterative process for debugging and optimizing the P target vehicle flow prediction networks. This process involves adjusting the network internal parameters in each cloud platform based on the network internal parameters obtained after repeated debugging in the E round of the P cloud platforms, and repeating the debugging of the adjusted network internal parameters for the F round (F=E+1) based on the sample set representing the vehicle flow characteristics of each cloud platform. The following is a detailed explanation of this step, combined with an example of the scenario of highway edge collaborative data processing.
[0034] First, assume that P is 5, indicating that there are 5 cloud platforms involved in the training and optimization of the vehicle flow prediction network. Each cloud platform maintains a vehicle flow prediction network. These networks may have the same network structure and initial parameters at the beginning, but as the training progresses, they will gradually learn the specific patterns on their respective data sets. After repeated debugging in the Eth round, each cloud platform obtains a set of network internal parameters (such as the weights and biases of the neural network). These parameters represent the knowledge of the network in its current state, that is, how to predict vehicle flow from the input vehicle flow feature representation. However, since each cloud platform may process different data sets, the network internal parameters they obtain may also be different.
[0035] In order to make full use of these differences and improve the performance of the overall model, the cloud platform will perform an aggregation operation. Specifically, it will collect all network internal parameters obtained by the cloud platform after the E-th round of debugging and calculate their aggregated parameters. The specific method of the aggregation operation can be selected according to the actual situation, for example, it can be to take the average, median or use some optimization algorithm to find a global optimal parameter set.
[0036] Next, the cloud platform will use these aggregated parameters to adjust the internal parameters of the network in each cloud platform. The purpose of this step is to integrate the knowledge of each cloud platform so that each cloud platform can benefit from the learning experience of other cloud platforms. By adjusting the internal parameters of the network, the vehicle flow prediction network on each cloud platform will update its state to better adapt to the distribution of the entire data set.
[0037] After the adjustment is completed, the cloud platform will repeat the Fth round (F=E+1) of debugging of the adjusted network internal parameters based on the sample set of vehicle flow characteristics of each cloud platform. This step is to further fine-tune the network parameters to improve its prediction performance on a specific data set. The debugging process usually involves steps such as forward propagation (calculating the predicted value), calculating the loss (the difference between the predicted value and the true value), and back propagation (calculating the gradient and updating the network parameters). Specifically, for each cloud platform, the cloud platform will use part of its vehicle flow characteristic representation sample set as training data and the other part as validation data. During the training process, the cloud platform will forward propagate the input features through the vehicle flow prediction network, calculate the predicted vehicle flow value, and compare it with the true value to calculate the loss. Then, the cloud platform will calculate the gradient through the back propagation algorithm and use an optimizer (such as the gradient descent algorithm) to update the network parameters. During the validation process, the cloud platform will use the validation data to evaluate the performance of the network and adjust the hyperparameters (such as the learning rate, batch size, etc.) as needed to optimize the training effect.
[0038] This process will be repeated for multiple rounds (i.e., F rounds) until a certain stopping criterion is met (such as reaching the preset maximum number of rounds, the verification loss no longer decreases, etc.). In each round of debugging, each cloud platform will update its vehicle flow prediction network based on the network internal parameters and vehicle flow feature representation sample set obtained in the current round. Finally, when the debugging process converges or reaches the preset stopping criterion, the cloud platform will obtain P debugged target vehicle flow prediction networks, which will be used to predict vehicle flow on highways in practical applications.
[0039] In the network application stage, steps S50 to S80 are included:
[0040] Step S50: deploying the target vehicle flow prediction network corresponding to each cloud platform to the corresponding edge box so that the edge box can perform target vehicle flow prediction.
[0041] In step S50, the target vehicle flow prediction network trained by each cloud platform is deployed to the corresponding edge box. This step is a key link in the edge collaborative data processing process. It enables the ability to predict vehicle flow to sink to the edge of the network and achieve low-latency real-time prediction. In the embodiment of the present application, the edge box is usually deployed near the data source, that is, near the monitoring point of the highway. These edge boxes have certain computing power and storage capacity, and can receive and process data from the monitoring point in real time. By deploying the vehicle flow prediction network to the edge box, data can be processed on the spot, greatly reducing the delay in data transmission and improving the real-time nature of the prediction. Next, the execution process of step S50 is explained in detail. Assume that there are 5 cloud platforms, and each cloud platform has trained a target vehicle flow prediction network. These networks have undergone multiple rounds of debugging and optimization and have good prediction performance. At this point, the cloud platform will begin to deploy these networks to the corresponding edge boxes.
[0042] Specifically, for each cloud platform, the cloud platform will first serialize its trained target vehicle flow prediction network and convert the network structure and parameters into a format that can be run on the edge box. This format is usually lightweight to adapt to the computing resource limitations of the edge box. Then, the cloud platform will transmit the serialized network model to the corresponding edge box through the network connection.
[0043] After the edge box receives the network model, it will perform a deserialization operation to restore the model to an executable state. At the same time, the edge box will also perform some configuration work as needed, such as setting the format of input data, specifying the storage location of output results, etc. After completing these configurations, the edge box has the ability to perform vehicle flow prediction.
[0044] Taking a specific example, suppose that an edge box is responsible for processing data from a highway monitoring point. When the monitoring point collects new high-speed monitoring data, these data will be transmitted to the edge box in real time. After receiving the data, the edge box will perform necessary data preprocessing operations on it, such as data cleaning, feature extraction, etc. Then, it will use the target vehicle flow prediction network that has been deployed on it to predict the processed data and obtain the prediction result. This prediction result will be stored in real time by the edge box and used in subsequent application scenarios, such as traffic scheduling, road condition release, etc. Through the execution of step S50, the cloud platform successfully sinks the vehicle flow prediction capability to the edge of the network, realizing low-latency real-time prediction. This can not only improve the accuracy and real-time performance of the prediction, but also reduce the load pressure on the central server and improve the processing power and efficiency of the entire system.
[0045] Each of the cloud platforms is in communication with at least one edge box, and for each of the edge boxes, when performing target vehicle flow prediction, it includes:
[0046] Step S60: Acquire target high-speed monitoring data.
[0047] In step S60, the edge box obtains the target highway monitoring data when performing target vehicle flow prediction for each edge box. This step is the basis for the edge box to perform real-time vehicle flow prediction, ensuring that subsequent data analysis and prediction can be based on the latest and accurate traffic data.
[0048] In the embodiment of the present application, the edge box is usually deployed near the monitoring points of the highway, which are equipped with various sensors and equipment for collecting traffic data in real time. These sensors may include traffic flow sensors, speed detectors, cameras, etc., which can monitor and record information such as vehicle flow, speed, road occupancy rate, etc. on the highway in real time. When the edge box performs vehicle flow prediction, it obtains the target high-speed monitoring data from the sensors and equipment at the monitoring point through the communication interface connected to it (such as network interface, serial port, etc.). These data may include real-time information such as the number of vehicles passing through, average speed, road congestion, etc. in a specific time period.
[0049] For example, suppose an edge box is responsible for processing monitoring data from a highway toll station. When a vehicle passes through the lane of the toll station, the traffic flow sensor records the vehicle's passing time and lane number in real time and transmits this data to the edge box. After receiving this data, the edge box will use it as the target highway monitoring data for subsequent vehicle flow prediction. It should be noted that since the traffic conditions on the highway change in real time, the edge box needs to continuously obtain the latest highway monitoring data from the monitoring point to ensure the accuracy and real-time nature of the prediction results. At the same time, since the data types and formats of different monitoring points may differ, the edge box also needs to perform necessary preprocessing and formatting operations on the acquired data so that it can be input into the vehicle flow prediction network for processing.
[0050] By executing step S60, the cloud platform can ensure that the edge box can be based on the latest and accurate traffic data when making vehicle flow predictions, thereby improving the accuracy and real-time nature of the prediction results.
[0051] Step S70: performing a data feature extraction operation on the target highway monitoring data to obtain a target vehicle flow feature representation of the target highway monitoring data.
[0052] In step S70, the cloud platform will perform data feature extraction on the acquired target highway monitoring data to generate a target vehicle flow feature representation. This step is a key part of the vehicle flow prediction process. It processes and analyzes the original data to extract feature information closely related to vehicle flow prediction and provide input for subsequent prediction models.
[0053] In the embodiment of the present application, the target high-speed monitoring data usually comes from various sensors and devices deployed along the highway, such as traffic flow sensors, vehicle speed detectors, cameras, etc. These data contain real-time traffic information, such as the number of vehicles passing, vehicle speed, road occupancy rate, etc. However, the raw data often contains a lot of redundant information and noise, and directly using it in the prediction model may lead to inaccurate prediction results. Therefore, it is necessary to extract features closely related to the target vehicle flow prediction through data feature extraction operations.
[0054] In step S70, the edge box will apply a series of data preprocessing and feature extraction techniques to process the target highway monitoring data. First, the edge box cleans and organizes the raw data to remove duplicate, abnormal or invalid data to ensure the accuracy and consistency of the data. Then, the edge box selects an appropriate feature extraction method based on the needs of vehicle flow prediction.
[0055] For example, for the vehicle passing data collected by the traffic flow sensor, the edge box can directly use it as a feature. For the speed data collected by the speed detector, the edge box can calculate its statistical features such as average, maximum, and minimum values to reflect the speed distribution in different time periods. For the image data collected by the camera, the edge box can apply image processing technology to extract visual features such as vehicle quantity, density, and speed. Specifically, it is assumed that the target highway monitoring data includes the number of vehicles passing and speed data of a certain section of road in a certain period of time. The edge box will first clean and organize these data to remove invalid or abnormal data points. Then, the edge box can extract the following features: Vehicle passing number feature: directly use the number of vehicles passing recorded by the sensor as the feature value. Average speed feature: calculate the average value of all speed data in the period to reflect the average driving speed of the section. Speed standard deviation feature: calculate the standard deviation of speed data to reflect the fluctuation of speed. These extracted features will form the target traffic flow feature representation for subsequent vehicle flow prediction models. Through the data feature extraction operation in step S70, the edge box can extract feature information closely related to the target vehicle flow prediction from the original high-speed monitoring data, providing strong support for subsequent predictions.
[0056] Step S80: Based on the target vehicle flow characteristic representation of the target highway monitoring data, the target highway monitoring data is classified through the deployed corresponding target vehicle flow prediction network to obtain the target vehicle flow prediction result of the target highway monitoring data.
[0057] In step S80, the edge box will classify and predict the target highway monitoring data according to the target vehicle flow characteristics of the target highway monitoring data, using the target vehicle flow prediction network deployed on the edge box, so as to obtain the target vehicle flow prediction result. This step is a key link in the edge collaborative data processing process, which directly determines the accuracy and practicality of the final prediction result.
[0058] Specifically, after the edge box obtains the target highway monitoring data, it performs a data feature extraction operation (step S70) to obtain the target vehicle flow feature representation. This feature representation contains key information closely related to the target vehicle flow, such as the number of vehicles passing, the average speed, the road occupancy rate, etc. Next, the edge box will pass these feature representations as input to the target vehicle flow prediction network deployed on the edge box. This prediction network is obtained through distributed training on multiple cloud platforms, and has learned the mapping relationship from feature representation to vehicle flow. Therefore, when a new feature representation is received, the prediction network can quickly calculate the corresponding vehicle flow prediction results.
[0059] During the prediction process, the prediction network will perform a series of calculations and inferences on the input feature representation. This process may involve forward propagation of multiple layers of neural networks, and each layer will calculate a new output based on the output of the previous layer and its own parameters. Ultimately, the network will output one or more predicted values, which represent the vehicle flow situation corresponding to the target highway monitoring data.
[0060] For example, suppose the edge box obtains a set of target highway monitoring data, and after data feature extraction, it obtains a feature vector containing features such as the number of vehicles passing, average speed, and road occupancy rate. This feature vector is input into the target vehicle flow prediction network, and after a series of calculations, the network outputs a prediction value, such as "medium flow". This prediction value is the target vehicle flow prediction result, which indicates that the vehicle flow of this section of road will be at a medium level in the future.
[0061] It should be noted that since the traffic conditions on the highway are changing in real time, the edge box needs to continuously obtain new target highway monitoring data and make real-time vehicle flow forecasts. In this way, the traffic management department can adjust the traffic dispatch strategy in time according to the forecast results, optimize the use efficiency of road resources, and improve the efficiency and safety of traffic operation.
[0062] As an implementation mode, in the above-mentioned network training process, the initial network internal parameter variables in each cloud platform are consistent, and each high-speed monitoring data set in the P high-speed monitoring data sets includes X / P high-speed monitoring data.
[0063] In one implementation, the cloud platform ensures that the initial network internal parameters (such as the weights and biases of the neural network) of each cloud platform remain consistent when training the vehicle flow prediction network. At the same time, the acquired X high-speed monitoring data are evenly divided into P high-speed monitoring data sets, each of which contains X / P high-speed monitoring data. This implementation aims to achieve collaborative learning between cloud platforms and ensure that each cloud platform can learn similar vehicle flow prediction patterns from different parts of the entire data set.
[0064] In machine learning, the initial parameters of the model have an important impact on the training results. When multiple cloud platforms start training with the same initial network internal parameters, they can maintain a certain degree of synchronization in the subsequent training process. This means that although the data subsets processed by each cloud platform may be different, due to the same initial parameters, the feature representations and prediction patterns they learn will have a higher similarity. This is very beneficial for subsequent model aggregation and parameter updates.
[0065] Taking highway edge collaborative data processing as an example, suppose X is 10,000 highway monitoring data samples and P is 5 cloud platforms. The cloud platform will first divide the 10,000 samples evenly into 5 data sets, each containing 2,000 samples. Then, the vehicle flow prediction network is initialized for each of the 5 cloud platforms, and their initial network internal parameters are ensured to be completely consistent.
[0066] Next, each cloud platform will use its own data set to train the network. Since the initial parameters are the same, the networks on each cloud platform will show similar performance in the early stages of training. As the training progresses, the network on each cloud platform will gradually learn the unique vehicle traffic patterns on its data set. However, due to the consistency of the initial parameters, these networks will still maintain a certain degree of synchronization in the overall structure and parameter update direction. The benefit of this implementation is that it makes full use of the computing power of multiple cloud platforms and realizes parallel processing of large-scale data sets. At the same time, by maintaining the consistency of the initial network internal parameters, it ensures that the network models trained on each cloud platform have a high degree of similarity in structure and performance. This facilitates subsequent model aggregation and parameter updates, and helps to improve the performance and stability of the entire vehicle traffic prediction system.
[0067] In practical applications, this implementation can help traffic management departments more accurately predict the vehicle flow on highways. By acquiring and processing highway monitoring data in real time, each cloud platform can quickly respond to changes in traffic conditions and provide accurate prediction results. This is of great significance for optimizing traffic scheduling, alleviating traffic congestion, and improving road use efficiency.
[0068] As an implementation mode, the step S40, adjusting the network internal parameter variables in each cloud platform according to the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms, and performing the F-th round of repeated debugging on the adjusted network internal parameter variables in each cloud platform according to the vehicle flow characteristic representation sample set of each cloud platform, to obtain the debugged P target vehicle flow prediction networks, includes:
[0069] Step S41: adjusting the network internal parameter variables in each of the cloud platforms by using the aggregated parameter variables of the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms.
[0070] In step S41, the cloud platform adjusts the network internal parameters in each cloud platform through the aggregated parameters of the network internal parameters obtained by the P cloud platforms after completing the E-th round of repeated debugging. This step is a typical application in Federated Learning, which aims to optimize the vehicle flow prediction network through the collaborative work of multiple cloud platforms while protecting data privacy.
[0071] Specifically, assume that P is 5, that is, there are 5 cloud platforms participating in the training of the vehicle flow prediction network. On each cloud platform, a vehicle flow prediction network is running, which may be built based on deep learning (such as convolutional neural network CNN or recurrent neural network RNN). During the training process, these networks will continuously update their internal parameters (i.e., the weights and biases of the neural network) according to the high-speed monitoring data sets they process. After the E-th round of repeated debugging is completed, each cloud platform obtains a set of updated network internal parameters. These parameters represent the learning results of the cloud platform on the current data set. However, since the data sets processed by different cloud platforms may be different, the parameters they obtain may also be different.
[0072] In order to integrate these differences and obtain a global update direction, the cloud platform adopts an aggregation strategy. The choice of aggregation strategy can be determined according to the specific application scenario and requirements. Common aggregation strategies include averaging, median, federated averaging, etc. In this example, it is assumed that averaging is used as the aggregation strategy. The cloud platform first collects the internal parameters of the network obtained by all cloud platforms after the E round of debugging, and averages the values at their corresponding positions. For example, assume that each cloud platform has a weight matrix W, where W1, W2, W3, W4, and W5 represent the weight matrices of the five cloud platforms after the E round of debugging. The cloud platform calculates the average value of the elements at the corresponding positions of these matrices to obtain an aggregated weight matrix W_avg.
[0073] Among them, W_avg=(W1+W2+W3+W4+W5) / 5.
[0074] This aggregate weight matrix W_avg represents the common learning results of all cloud platforms in the E-th round of debugging, and is a global update direction. Next, the cloud platform uses this aggregate weight matrix W_avg as a new network internal parameter to adjust the network internal parameters in each cloud platform. Specifically, each cloud platform will update its current weight matrix to the aggregate weight matrix W_avg, so that its network parameters are consistent with the global update direction.
[0075] Through this step, the cloud platform realizes collaborative learning among multiple cloud platforms. Each cloud platform contributes its own learning results, which are integrated into a global update direction through aggregation strategy, and their respective network parameters are adjusted accordingly. This method can not only make full use of the computing power of each cloud platform and accelerate the network training process, but also improve the generalization ability of the network model, so that it can better adapt to changes in actual highway traffic conditions. At the same time, since the data does not need to leave the cloud platform, data privacy is also protected.
[0076] Step S42: Based on the sample set representing the vehicle flow characteristics of each cloud platform, the network internal parameter variables in each cloud platform are repeatedly debugged for the Fth round until the number of repeated rounds of the network internal parameter variables in each cloud platform is equal to the maximum number of repeated rounds determined in advance or the network internal parameter variables in each cloud platform converge, thereby obtaining the P target vehicle flow prediction networks.
[0077] In step S42, the cloud platform performs the Fth round (F=E+1) of repeated debugging of the network internal parameters in each cloud platform based on the sample set of vehicle flow characteristics of each cloud platform. This step is a key part of the vehicle flow prediction network training process, which aims to improve the prediction performance of the network through iterative optimization.
[0078] 1. Initialization of the debugging process.
[0079] Before starting the Fth round of repeated debugging, the cloud platform has obtained the network internal parameters adjusted by the aggregated parameters from step S41. These parameters are used as the initial values of this round of debugging and will be used to guide the parameter update of the network.
[0080] 2. Debugging of sample sets based on traffic flow characteristics.
[0081] Each cloud platform has its own unique set of traffic flow feature representation samples, which contain key features extracted from high-speed monitoring data, such as the number of vehicles passing, average speed, road occupancy rate, etc. These features are crucial for predicting vehicle flow. In the Fth round of debugging, the cloud platform will use the traffic flow feature representation sample set of each cloud platform as training data and input it into the corresponding vehicle flow prediction network. The network performs forward propagation calculations based on the current network internal parameters to obtain the predicted vehicle flow value. Then, the cloud platform compares the predicted value with the true value and calculates the prediction error (loss function value).
[0082] 3. Back propagation and parameter update.
[0083] After obtaining the prediction error, the cloud platform will perform back propagation calculations and calculate the gradient of each parameter in the network layer by layer through the chain rule. The gradient represents the degree of influence of the parameter on the loss function and guides the direction of parameter update. Then, the cloud platform uses an optimization algorithm (such as the gradient descent algorithm) to update the internal parameters of the network according to the calculated gradient. The optimization algorithm reduces the prediction error by adjusting the size of the parameters, so that the network gradually approaches the optimal solution.
[0084] 4. Iterative debugging and convergence judgment.
[0085] The above-mentioned forward propagation, error calculation, back propagation and parameter update processes will be repeated continuously, forming an iterative optimization cycle. In each round of debugging, the network will update its internal parameters according to the new training data to improve the prediction performance. However, the iterative process is not infinite. When a certain stopping condition is met, the iterative process will terminate. The stopping condition can be reaching a predetermined maximum number of repetitions (for example, set to 100 rounds) or the convergence of the internal parameters of the network (that is, the parameter change is less than a preset threshold). When one of these conditions is met, the cloud platform believes that the network has been fully trained and a debugged target vehicle flow prediction network has been obtained.
[0086] For example, suppose that three cloud platforms participate in the training of the vehicle flow prediction network, and each cloud platform has its own unique set of samples representing vehicle flow characteristics. In the Fth round of debugging, each cloud platform inputs its sample set into the corresponding network and performs forward propagation calculation to obtain the predicted value. Then, each cloud platform calculates the prediction error separately, and performs back propagation and parameter update. After multiple rounds of iterative debugging, when the change in the internal parameters of all cloud platforms is less than the preset threshold, the iterative process terminates and three debugged target vehicle flow prediction networks are obtained. These networks will be deployed to the edge box for real-time vehicle flow prediction.
[0087] As an implementation mode, the step S42, based on the sample set representing the vehicle flow characteristics of each cloud platform, performs the Fth round of repeated debugging on the network internal parameter variables in each cloud platform until the number of repeated rounds of the network internal parameter variables in each cloud platform is equal to the maximum repeated rounds determined in advance or the network internal parameter variables in each cloud platform converge, and obtains the P target vehicle flow prediction networks, including:
[0088] Step S421: Multiply the network internal parameter variables in each cloud platform with the target vehicle flow characteristic representation in the vehicle flow characteristic representation sample set of each cloud platform, subtract the product result from the priori mark in the vehicle flow characteristic representation sample set of each cloud platform to obtain the subtraction result, multiply the subtraction result with the target vehicle flow characteristic representation in the vehicle flow characteristic representation sample set of each cloud platform to obtain the bias vector of the network internal parameter variables in each cloud platform.
[0089] In step S421, it is assumed that there are P cloud platforms, each cloud platform maintains a vehicle flow prediction network, and each network contains a set of network internal parameters, which usually include the weights and biases of the neural network. At the same time, each cloud platform has its own unique vehicle flow feature representation sample set, which contains key features extracted from high-speed monitoring data, such as the number of vehicles, vehicle speed, road occupancy, etc. The execution process of step S421 is divided into several sub-steps:
[0090] The product of feature representation and network parameters:
[0091] For each cloud platform, the cloud platform first multiplies the target traffic flow feature representation in its traffic flow feature representation sample set with the network internal parameter variables. The target traffic flow feature representation here is usually a feature vector containing multiple feature values related to vehicle flow. The network internal parameter variables represent the current state of the network, that is, how to map from input features to output prediction values. The multiplication operation is actually the forward propagation process of the network, which simulates the network's response to the input features based on the current parameters.
[0092] For example, suppose the vehicle flow prediction network on a cloud platform is a simple fully connected neural network with a weight matrix W and a bias vector b, and a target vehicle flow feature is represented by x. Then the forward propagation process is to perform matrix multiplication of x and W, and add b to get an output prediction value y.
[0093] The difference between the predicted value and the prior label:
[0094] Next, the cloud platform subtracts the predicted value obtained by forward propagation from the priori mark in the sample set representing the traffic flow characteristics. The priori mark is the real vehicle flow data, which is used to compare with the network prediction value to evaluate the prediction performance of the network. The result of the subtraction operation is an error vector, which reflects the difference between the network prediction value and the true value.
[0095] Continuing with the above example, assuming the prior is labeled y_true, then the error vector e=y_true-y.
[0096] The product of the error vector and the target traffic flow characteristic representation is:
[0097] Finally, the cloud platform multiplies the error vector by the target traffic flow feature representation, and the result is the bias vector (gradient) of the network's internal parameters. This bias vector indicates how the network should adjust its internal parameters to reduce the prediction error under the current parameters. Specifically, each element of the bias vector corresponds to a parameter in the network's internal parameters (such as weight or bias), and the size and direction of its value indicate the amount and direction in which the parameter should be adjusted.
[0098] In the above example, the bias vector g can be obtained by multiplying the error vector e with the target vehicle flow feature representation x, that is, g = e*x (where "*" represents element-wise multiplication). It should be noted that in practical applications, since the internal parameters of the network may contain a large number of weights and biases, the product operation is usually based on matrices and vectors and may involve more complex calculation processes. Through the calculation of step S421, the cloud platform obtains the bias vectors of the vehicle flow prediction network in each cloud platform. These bias vectors will be used in the subsequent parameter update process to gradually optimize the prediction performance of the network.
[0099] Step S422: Multiply the tuning parameters of the network internal parameters in each cloud platform and the bias vectors of the network internal parameters in each cloud platform to obtain the adjustment results of the network internal parameters in each cloud platform.
[0100] In step S422, the cloud platform uses the core idea of the gradient descent algorithm to guide the update of network parameters by calculating the adjustment results of the network's internal parameters. This step is a key link in optimizing the performance of the vehicle flow prediction network. It ensures that the network can gradually adjust its internal parameters in the direction of reducing the prediction error. The tuning parameters of the network's internal parameters are usually called the learning rate, which determines the step size of the network's parameter update in each iteration. A suitable learning rate can ensure that the network converges to the optimal solution stably and efficiently. The bias vector (i.e., gradient) is calculated in step S421, which indicates how the network should adjust its internal parameters to reduce the prediction error under the current parameters.
[0101] In the embodiment of the present application, it is assumed that the vehicle flow prediction network on each cloud platform has undergone forward propagation and error calculation to obtain its own bias vector. This bias vector is a vector with the same dimension as the internal parameter of the network. Each of its elements corresponds to a weight or bias term in the network parameters, and indicates the direction and magnitude of these parameters that should be adjusted. Next, the cloud platform will execute step S422, which is to multiply the tuning parameters (learning rate) of the internal parameters of the network in each cloud platform with the bias vector. This product operation is actually scaling each element of the bias vector, and the scaling factor is the learning rate. The size of the learning rate determines the amplitude of the parameter update. A larger learning rate means that the step size of the parameter update in each iteration is larger, which may speed up the convergence speed but may also cause the network to oscillate near the optimal solution; a smaller learning rate may result in slower convergence but more stable.
[0102] For example, suppose that a vehicle flow prediction network on a cloud platform has a weight matrix W, its corresponding bias vector is g, and the tuning parameter is η. Then, the adjustment result ΔW of the internal parameters of the network on the cloud platform can be calculated by the following formula: ΔW=η*g.
[0103] Here, ΔW is a matrix with the same dimension as W, and each of its elements represents the amount by which the corresponding element in W should be adjusted in the next iteration. By calculating ΔW, the cloud platform obtains guidance information for updating network parameters. Finally, the cloud platform will use the adjustment result ΔW to update the network's internal parameter W, that is, perform the operation W = W-ΔW. In this way, the network adjusts its internal parameters based on the latest training data to reduce prediction errors and improve prediction performance. Through multiple rounds of iterations and parameter updates, the vehicle flow prediction network can gradually learn the accurate mapping relationship from input features to output prediction values.
[0104] Step S423: obtain the P network internal parameter variables obtained after completing the Fth round of repeated debugging in the P cloud platforms by subtracting the adjustment result of the network internal parameter variables in each cloud platform from the network internal parameter variables in each cloud platform.
[0105] In step S423, the cloud platform obtains updated network internal parameters by subtracting the corresponding adjustment results from the network internal parameters in each cloud platform. This step is the core link in the gradient descent algorithm, which is used to guide the network model to iteratively optimize in the direction of reducing the prediction error. Specifically, in an embodiment of the present application, each cloud platform maintains a vehicle flow prediction network, which contains a set of network internal parameters, such as weights and biases of the neural network. In step S422, the cloud platform has calculated the adjustment results of the network internal parameters on each cloud platform. This result is calculated based on the bias vector and tuning parameters of the network, which indicates how the network should adjust its internal parameters under the current parameters to reduce the prediction error.
[0106] In step S423, for each vehicle flow prediction network on the cloud platform, the cloud platform will subtract the corresponding adjustment result from the current network internal parameter. This subtraction operation is actually a small step move in the direction of gradient descent in the network parameter space, so that the network parameters gradually approach the optimal solution. For example, suppose that the vehicle flow prediction network on a certain cloud platform has a weight matrix W, and its corresponding adjustment result is ΔW. In step S423, the cloud platform will perform the operation W = W-ΔW, that is, subtract the adjustment result ΔW from the original weight matrix W to obtain the updated weight matrix W'. This W' is the new network internal parameter obtained by the cloud platform after the Fth round of repeated debugging, which contains the parameter information of the network adjusted according to the latest training data.
[0107] By executing step S423, the cloud platform completes the parameter update of the vehicle flow prediction network on each cloud platform. These updated network internal parameters will be used for the next round of forward propagation and error calculation to further iterate and optimize the network model. Through multiple rounds of repeated debugging and parameter updates, the vehicle flow prediction network can gradually learn the accurate mapping relationship from input features to output prediction values, thereby improving the accuracy and robustness of the prediction.
[0108] Step S424: If F is equal to the predetermined maximum number of repetition rounds or the bias vector of the network internal parameter variables in each cloud platform is equal to 0, the P target vehicle flow prediction networks are obtained.
[0109] In step S424, the cloud platform determines whether the stop condition has been reached based on the number of iterations F and the bias vectors of the internal parameters of the network in each cloud platform, thereby determining whether the target vehicle flow prediction network has been obtained. This step is an important part of the vehicle flow prediction network training process, ensuring that the network stops iterating after sufficient learning and optimization, avoiding overfitting or other unnecessary consumption of computing resources.
[0110] Specifically, in the embodiment of the present application, each cloud platform runs a vehicle flow prediction network, which optimizes its internal parameters (such as the weights and biases of the neural network) through multiple rounds of iterative debugging. In each round of iteration, the cloud platform calculates the bias vector of the network and updates the network parameters based on the bias vector and learning rate. This process is repeated until a certain stop condition is met.
[0111] Two stop conditions are defined in step S424:
[0112] Iteration rounds reach the maximum repetition rounds: Before training begins, a maximum repetition round F_max is usually set as the upper limit of the iteration process. This value can be determined based on the specific application scenario and computing resources. When the iteration round F reaches or exceeds F_max, the cloud platform believes that the network has been fully trained, and it can stop iterating and output the current network as the target vehicle flow prediction network.
[0113] For example, suppose the maximum number of repetitions F_max is set to 100. If the system still does not meet other stopping conditions (such as the bias vector is close to 0) after the 100th iteration, the cloud platform will stop training after the 100th iteration and use the network at this time as the target vehicle flow prediction network.
[0114] The bias vector is close to 0: The bias vector reflects the direction and size of the network parameters that should be adjusted in the feature space. When the bias vector is close to 0, it means that the gradient of the network parameters on the current data is already very small, that is, the network parameters are close to or have reached the optimal solution. At this point, continuing to iterate may not significantly improve the performance of the network, but will waste computing resources. Therefore, when the bias vector is less than a preset threshold ε (for example, ε=1e-6), the cloud platform will also stop iterating and output the current network as the target vehicle flow prediction network. For example, suppose that in a certain iteration round, all elements of the bias vector calculated by the cloud platform are less than 1e-6. This means that the gradient of the network parameters on the current data is already very small, and the cloud platform can assume that the network has converged to the vicinity of the optimal solution. Therefore, the cloud platform will stop iterating and use the network at this time as the target vehicle flow prediction network.
[0115] Through the judgment in step S424, the cloud platform can ensure that in the process of training the vehicle flow prediction network, computing resources are fully utilized for optimization while avoiding unnecessary waste of resources, which helps to improve the prediction performance of the network while reducing training costs.
[0116] Step S425: If F is less than the predetermined maximum number of repetitions or the bias vector of the network internal parameter variables in each cloud platform is equal to 0, then based on the vehicle flow characteristic representation sample set of each cloud platform and the P network internal parameter variables obtained after completing the Fth round of repeated debugging in the P cloud platforms, the network internal parameter variables in each cloud platform are repeatedly debugged for the Gth time until the number of repetitions of the network internal parameter variables in each cloud platform is equal to the predetermined maximum number of repetitions or the network internal parameter variables in each cloud platform converge, and the P target vehicle flow prediction networks are obtained, wherein G=F+1.
[0117] In step S425, the cloud platform determines whether the next round of repeated debugging is needed based on the current iteration round F and the bias vector of the network internal parameter variables in each cloud platform. This is a key step to ensure that the vehicle flow prediction network can achieve optimal performance. Through continuous iterative debugging, the network system can gradually approach the optimal solution.
[0118] Specifically, in the embodiment of the present application, each cloud platform runs a vehicle flow prediction network, and these networks have undergone multiple rounds of iterative debugging. In each iteration, the cloud platform calculates the bias vector of the network based on the current network internal parameters and the sample set representing the vehicle flow characteristics, and updates the network parameters based on the bias vector and tuning parameters. This process is repeated until a certain stop condition is met.
[0119] However, in step S424, if the current iteration round F is less than the maximum repetition round F_max determined in advance, or although F has reached F_max, the bias vector of the network internal parameter variables in each cloud platform is still not equal to 0 (that is, the network has not yet converged), the cloud platform needs to continue to execute step S425 for the next round of iterative debugging. In step S425, the cloud platform first collects the network internal parameter variables and the corresponding vehicle flow feature representation sample set obtained by each cloud platform after the Fth round of iteration. Then, the cloud platform will use these parameter variables and sample sets as input to start the Gth round (G=F+1) of iterative debugging. The process of this round of debugging is similar to the previous iteration, including the steps of forward propagation to calculate the predicted value, calculate the prediction error, back propagation to calculate the bias vector, and update the network parameters.
[0120] By continuously repeating the iterative debugging process in step S425, the vehicle flow prediction network can gradually learn the accurate mapping relationship from input features to output prediction values. As the number of iterations increases, the prediction performance of the network will gradually improve. When the stopping condition is met (i.e., the iterations reach F_max or the internal parameters of the network converge), the cloud platform believes that the network has been fully trained and outputs the network at this time as the target vehicle flow prediction network.
[0121] For example, assume that the maximum number of repetition rounds F_max is set to 100 rounds, and the current iteration round F is 95 rounds. After the 95th iteration, the bias vector calculated by the cloud platform is still not equal to 0, which means that the network has not yet converged to the optimal solution. Therefore, the cloud platform will execute step S425 and start the 96th round (G=95+1) of iterative debugging. During this round of debugging, the cloud platform will update the network parameters based on the network internal parameters and vehicle flow feature representation sample set after the 95th iteration, and continue to improve the prediction performance of the network. When the iteration round reaches F_max or the internal parameters of the network converge, the cloud platform stops iterating and outputs the target vehicle flow prediction network.
[0122] As another implementation, the step S40, adjusting the network internal parameter variables in each cloud platform according to the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms, and performing the F-th round of repeated debugging on the adjusted network internal parameter variables in each cloud platform according to the vehicle flow characteristic representation sample set of each cloud platform, to obtain the debugged P target vehicle flow prediction networks, includes:
[0123] Step S4A: Sparsely extract the sample sets representing the traffic flow characteristics of each cloud platform to obtain the learning sample sets of each cloud platform and the verification sample sets of each cloud platform.
[0124] In step S4A, the cloud platform performs sparse extraction on the sample sets representing the traffic flow characteristics on each cloud platform, that is, through sampling technology, the original sample set is divided into a learning sample set (Training Set) and a verification sample set (Test Set). This step is a common data preprocessing step in machine learning. In particular, when using a supervised learning algorithm to train a model, dividing the data set into a training set and a test set is the key to ensuring the generalization performance of the model. In the embodiment of the present application, each cloud platform has a large number of traffic flow characteristic representation samples, which are extracted from high-speed monitoring data and contain key features related to vehicle flow prediction. However, directly using all samples for model training is often not the most efficient, because too many samples may lead to too long training time and may contain redundant information or noise. Therefore, by dividing the samples into training sets and test sets through sparse extraction, the sample data can be used more effectively.
[0125] Specifically, the cloud platform first determines a sampling strategy, which can be random sampling based on the number of samples, or stratified sampling based on the distribution of sample features. Then, according to this strategy, a part of the samples are randomly selected from the original sample set representing vehicle flow characteristics as a learning sample set for training the vehicle flow prediction network. The remaining samples are used as a validation sample set to evaluate the performance of the network during the training process. For example, suppose a cloud platform has 10,000 samples representing vehicle flow characteristics. The cloud platform can adopt a random sampling strategy to select 80% of the samples (i.e., 8,000) as a learning sample set for training the vehicle flow prediction network; the remaining 20% of the samples (i.e., 2,000) are used as a validation sample set to evaluate the prediction error of the network during the training process. Through the sparse extraction operation of step S4A, the cloud platform not only divides the original sample set into a training set and a test set, but also ensures the independence between the training set and the test set. This enables a more accurate evaluation of the performance of the model during the training process and helps prevent the occurrence of overfitting. At the same time, by reasonably dividing the training set and test set, the cloud platform can more effectively utilize limited computing resources and accelerate the model training and optimization process.
[0126] Step S4B: adjusting the network internal parameter variables in each of the cloud platforms by using the aggregated parameter variables of the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms.
[0127] In step S4B, the cloud platform adjusts the network internal parameters in each cloud platform through the aggregated parameters of the network internal parameters obtained by P cloud platforms after repeated debugging in the E round. This step embodies the core idea of federated learning, that is, while protecting data privacy, the global model is optimized through the collaborative work of multiple cloud platforms. In the embodiment of the present application, each cloud platform runs a vehicle flow prediction network, and these networks learn the mapping relationship from input features to vehicle flow through training. However, since the data sets processed by each cloud platform may be different, the internal network parameters (such as the weights and biases of the neural network) obtained by their training may also be different. In order to make full use of these differences and obtain a globally optimal model, the cloud platform adopts the strategy of aggregating parameters.
[0128] Specifically, after the E-th round of repeated debugging, each cloud platform obtained a set of updated network internal parameter variables. These parameter variables represent the learning results of the cloud platform on the current data set. In order to integrate these results, the cloud platform adopts an aggregation strategy (such as taking the minimum parameter, average value, median, etc.) to aggregate the network internal parameter variables on P cloud platforms to obtain an aggregate parameter variable. This aggregate parameter variable represents the common learning results of all cloud platforms in the E-th round of debugging and is a global update direction. Next, the cloud platform uses this aggregate parameter variable to adjust the network internal parameter variables in each cloud platform. This process actually updates the network parameters on each cloud platform to the value of the aggregate parameter variable. Through this step, the vehicle flow prediction network on each cloud platform incorporates the learning results of other cloud platforms, enabling the entire system to optimize model performance globally.
[0129] For example, suppose that there are three cloud platforms participating in the training of the vehicle flow prediction network. After the E-round debugging, each cloud platform obtains updated network internal parameters, which are W1, W2 and W3 (W1, W2 and W3 are simplified representations of the weight matrix of the neural network). The cloud platform adopts the aggregation strategy of taking the average value to calculate the aggregated parameter W_avg=(W1+W2+W3) / 3. Then, the cloud platform uses this aggregated parameter W_avg as the new network internal parameter and updates it to the vehicle flow prediction network on the three cloud platforms, that is, W1'=W_avg, W2'=W_avg, and W3'=W_avg. In this way, the network parameters on the three cloud platforms are uniformly updated, reflecting the common learning results of all cloud platforms in the E-round debugging. Through the execution of step S4B, the cloud platform realizes collaborative learning among multiple cloud platforms, makes full use of the data and computing resources of each cloud platform, and improves the performance and generalization ability of the vehicle flow prediction network. At the same time, since the data does not need to leave the cloud platform, data privacy and security are also protected.
[0130] Step S4C: performing an F-th round of repeated debugging on the network internal parameter variables in each cloud platform according to the learning sample set of each cloud platform, and obtaining P network internal parameter variables obtained after completing the F-th round of repeated debugging in the P cloud platforms.
[0131] In step S4C, the cloud platform performs the Fth round of repeated debugging on the vehicle flow prediction network in each cloud platform based on the learning sample set of each cloud platform. This step is the core link in the machine learning model training process. By iteratively optimizing the internal parameters of the network (such as the weight and bias of the neural network), the network can more accurately predict the vehicle flow on the highway.
[0132] In the embodiment of the present application, each cloud platform maintains a vehicle flow prediction network and holds a learning sample set, which is obtained from the original vehicle flow feature representation sample set through sparse extraction. The learning sample set contains key feature data used to train the network model, such as the number of vehicles passing, average speed, road occupancy, etc., as well as the corresponding real vehicle flow labels.
[0133] In step S4C, the cloud platform first inputs the learning sample set on each cloud platform into the corresponding vehicle flow prediction network. The network performs forward propagation calculations on the input samples based on the current internal parameters (i.e., the weights and biases of the neural network) to generate predicted vehicle flow values. Then, the cloud platform compares the difference between the predicted value and the true label and calculates the loss function value (such as mean square error MSE) to quantify the prediction performance of the network under the current parameters. Next, the cloud platform uses the backpropagation algorithm to calculate the gradient of the loss function with respect to the internal parameters of the network (i.e., the bias vector). This gradient indicates how the network should adjust its internal parameters to reduce the prediction error under the current parameters. Based on this gradient information, the cloud platform uses an optimization algorithm (such as a gradient descent algorithm) to update the internal parameters of the network so that the prediction performance of the network is improved in the next round of iterations.
[0134] Specifically, the cloud platform will multiply the learning rate by the gradient to obtain the update amount (i.e., the adjustment result) of each internal parameter. Then, the cloud platform will subtract this update amount from the current internal parameter to obtain the updated internal parameter. This process will be performed for all parameters in the network, thereby achieving parameter updates for the entire network. After completing a round of parameter updates, the cloud platform has completed an iterative debugging. In step S4C, this process will be repeated for F rounds to ensure that the network can fully learn the patterns in the data and achieve better prediction performance. In each round of iteration, the cloud platform will use the latest set of learning samples to evaluate the performance of the network, and use the evaluation results to guide the parameter update in the next round of iteration.
[0135] Through the iterative debugging process of step S4C, the cloud platform can gradually optimize the vehicle flow prediction network on each cloud platform, so that it can more accurately predict the vehicle flow on the highway. This helps the traffic management department make more scientific decisions, improve the utilization efficiency of road resources, and alleviate traffic congestion and other problems.
[0136] Step S4D: Based on the verification sample sets of the respective cloud platforms, the errors of the network internal parameter variables in the respective cloud platforms are evaluated to obtain target error values indicating the errors of the network internal parameter variables in the P cloud platforms.
[0137] In step S4D, the cloud platform evaluates the error of the internal parameters of the vehicle flow prediction network in each cloud platform after the Fth round of repeated debugging based on the verification sample set of each cloud platform. This step is an important part of the machine learning model training process, which is used to verify the performance of the network model on unseen data and judge whether the model has reached the preset training standard.
[0138] In an embodiment of the present application, each cloud platform holds a verification sample set, which is obtained from the original vehicle flow feature representation sample set through sparse extraction (i.e., sampling) and is independent of the learning sample set used for training. The verification sample set contains real highway traffic data, which is used to simulate the performance of the model in actual applications. In step S4D, the cloud platform first inputs the verification sample set of each cloud platform into the corresponding vehicle flow prediction network. The network performs forward propagation calculations on the input samples based on the current internal parameters (i.e., the weights and biases of the neural network) to generate predicted vehicle flow values. Then, the cloud platform compares the difference between the predicted value and the actual vehicle flow value in the verification sample, and calculates the loss function value (such as mean square error MSE) to quantify the prediction error of the network under the current parameters.
[0139] The loss function value (target error value) is a key indicator for evaluating network performance. It reflects the prediction accuracy of the network model on the validation sample set, that is, the model's generalization ability for unseen data. A smaller loss function value means that the network model has a smaller prediction error on the validation sample and better performance.
[0140] Specifically, suppose a cloud platform's validation sample set contains 1,000 pieces of data, each of which has a corresponding real vehicle flow value. After the validation sample is input into the vehicle flow prediction network, the network will generate 1,000 predicted vehicle flow values. Then, the cloud platform will calculate the mean square error (MSE) between these 1,000 predicted values and the real values to obtain the target error value on the cloud platform.
[0141] Through the error evaluation process in step S4D, the cloud platform can understand the performance of the vehicle flow prediction network on each cloud platform on the verification sample. This provides an important basis for subsequent model selection and tuning. If the target error value is small, it means that the network has achieved good prediction performance; if the target error value is large, it may be necessary to continue iteratively debugging network parameters or adjusting the network structure to optimize performance.
[0142] Step S4E: Determine whether the target error value meets the evaluation conditions. If the target error value meets the evaluation conditions, obtain the P target vehicle flow prediction networks; if the target error value does not meet the evaluation conditions, adjust the network internal parameters in each cloud platform through the aggregated parameters of the P network internal parameters obtained after completing the Fth round of repeated debugging in the P cloud platforms, and perform the Gth repeated debugging on the network internal parameters in each cloud platform based on the learning sample set of each cloud platform, until the target error value meets the evaluation conditions, obtain the P target vehicle flow prediction networks, where G=F+1.
[0143] In step S4E, the cloud platform judges the target error value calculated in step S4D to determine whether the vehicle flow prediction network on each cloud platform has reached the preset training standard. This step is a key decision point in the model training process, which determines whether it is necessary to continue iteratively debugging the network parameters to optimize performance. Specifically, in step S4D, the cloud platform has calculated the target error value based on the verification sample set of each cloud platform, which quantifies the prediction error of the network model on the verification sample. In step S4E, the cloud platform compares this target error value with the preset evaluation condition. The evaluation condition is usually a specific numerical threshold used to determine whether the performance of the network model meets the requirements.
[0144] For example, suppose the preset evaluation condition is that the target error value is no greater than 0.01. If the target error value on a cloud platform is less than or equal to 0.01, it means that the vehicle flow prediction network on the cloud platform has reached the preset training standard, and the iterative debugging can be stopped, and the current network model is output as the target vehicle flow prediction network. However, if the target error value is greater than 0.01, it means that the performance of the network model has not yet met the requirements and iterative debugging needs to continue. In this case, the cloud platform will perform the following operations:
[0145] Aggregate parameter adjustment: First, the cloud platform collects the network internal parameters obtained by each cloud platform after repeated debugging in the Fth round, and calculates the aggregated parameters of these parameters. The aggregated parameters represent the common learning results of all cloud platforms in the current iteration round and are a global update direction. Then, the cloud platform uses this aggregated parameter to adjust the network internal parameters of each cloud platform, so that the network on each cloud platform can integrate the learning results of other cloud platforms.
[0146] Repeated debugging: After adjusting the internal parameters of the network, the cloud platform will repeat the debugging for the Gth round (G=F+1) based on the learning sample sets of each cloud platform. The process of this round of debugging is similar to the previous iteration, including forward propagation to calculate the predicted value, calculate the loss function value, back propagation to calculate the gradient, and update the network parameters. Through this round of debugging, the network model will try to further reduce the prediction error and improve performance.
[0147] This process will be repeated until the target error value meets the evaluation conditions. In each iteration, the cloud platform will calculate the target error value based on the latest set of validation samples, and guide the parameter update direction in the next iteration based on the size of the error value. In this way, the cloud platform can gradually optimize the vehicle flow prediction network on each cloud platform so that its performance gradually approaches the optimal solution. Finally, when the target error values on all cloud platforms meet the evaluation conditions, the cloud platform will stop iterative debugging and output the network model at this time as the target vehicle flow prediction network. These target vehicle flow prediction networks will have higher prediction accuracy and generalization ability, and can better adapt to changes in actual highway traffic conditions.
[0148] As an implementation mode, the step S4A, sparsely extracting the sample sets representing the traffic flow characteristics of the respective cloud platforms to obtain the learning sample sets of the respective cloud platforms and the verification sample sets of the respective cloud platforms, includes:
[0149] Step S4A1: According to the category of the sample set representing the traffic flow characteristics of each cloud platform, sparse extraction is performed on the sample set representing the traffic flow characteristics of each cloud platform based on a set percentage to obtain a learning sample set of each cloud platform and a verification sample set of each cloud platform.
[0150] In the implementation of step S4A, the cloud platform divides the learning sample set and the verification sample set by sparsely extracting the sample sets representing the traffic flow characteristics on each cloud platform. The key to this step is how to reasonably perform sparse extraction according to the characteristics and requirements of the sample set.
[0151] In step S4A1, the cloud platform first considers the category of the sample set representing the traffic flow characteristics on each cloud platform. In the embodiment of the present application, the sample set may contain a variety of different types of data, such as traffic flow data in different time periods, different road sections, and different weather conditions. These data may have different statistical characteristics and distribution patterns, so these differences need to be considered when performing sparse extraction. Next, based on the set percentage, the cloud platform performs sparse extraction on the sample set representing the traffic flow characteristics on each cloud platform. This percentage is usually set based on actual needs and experience to determine the size ratio of the learning sample set and the verification sample set. For example, 80% of the samples can be set as the learning sample set for training the vehicle flow prediction network; the remaining 20% of the samples are used as the verification sample set to evaluate the performance of the network during the training process. In specific operations, the cloud platform randomly selects a corresponding number of samples as the learning sample set and the verification sample set according to the category of the sample set and the set percentage. This process is usually random to ensure the independence of the learning sample set and the verification sample set and to avoid possible deviations.
[0152] For example, suppose that the traffic flow feature representation sample set on a cloud platform contains 10,000 data, which covers traffic flow data in different time periods and different road sections. The cloud platform sets 80% of the samples as the learning sample set, that is, 8,000 data are selected as learning samples; the remaining 20% of the samples, that is, 2,000 data are used as the verification sample set. During the selection process, the cloud platform will ensure that the distribution ratio of samples of different categories (such as different time periods and different road sections) in the learning sample set and the verification sample set is roughly the same to maintain the diversity and representativeness of the samples.
[0153] Through the sparse extraction operation in step S4A1, the cloud platform can effectively divide the learning sample set and the verification sample set, providing strong support for the subsequent vehicle flow prediction network training and performance evaluation. At the same time, since the category and distribution characteristics of the sample set are taken into consideration, this division method can also ensure the diversity and representativeness of the samples, and improve the generalization ability and prediction accuracy of the model.
[0154] As an implementation mode, the step S4A1, according to the category of the vehicle flow characteristic representation sample set of each cloud platform, performs sparse extraction on the vehicle flow characteristic representation sample set of each cloud platform based on a set percentage to obtain the learning sample set of each cloud platform and the verification sample set of each cloud platform, including:
[0155] Step S4A11: If the category of the vehicle flow feature representation sample set of each cloud platform is a distributed data category that conforms to the consistent feature space but different sample space, then based on the set percentage, sparse extraction is performed on the target vehicle flow feature representation and prior labels in the vehicle flow feature representation sample set of each cloud platform to obtain the learning sample set of each cloud platform and the verification sample set of each cloud platform.
[0156] In step S4A11, the cloud platform targets a specific type of distributed data, that is, the case where the feature space is consistent but the sample space is different, which usually occurs in the scenario of horizontal federated learning. In the context of collaborative data processing at the edge of highways, this situation means that although different cloud platforms have collected feature data about traffic flow (such as speed, traffic flow, road occupancy, etc.), these data come from different highway sections or different time periods, so the sample spaces are different.
[0157] In order to effectively use these data to train the vehicle flow prediction network and ensure that the model has good generalization ability, the cloud platform uses a sparse extraction method to divide the learning sample set and the verification sample set. Specifically, in step S4A11, the cloud platform first determines a set percentage for dividing the ratio of learning samples and verification samples. This percentage is usually determined based on experience or experiments, aiming to balance the training efficiency and generalization ability of the model. For example, a common division ratio is 80% of the data as learning samples and 20% of the data as verification samples. Next, for each set of vehicle flow feature representation samples on the cloud platform, the cloud platform randomly samples according to the set percentage. During the sampling process, the cloud platform will simultaneously select the target vehicle flow feature representation and the corresponding prior label (that is, the real vehicle flow data). The target vehicle flow feature representation is a multi-dimensional vector that contains various feature values describing the vehicle flow; and the prior label is the real vehicle flow value corresponding to these feature values, which is used to supervise the error calculation in the learning process.
[0158] For example, suppose a cloud platform has 10,000 traffic flow feature representation samples, each of which contains feature values such as vehicle speed, traffic flow, road occupancy rate, and the corresponding real vehicle flow value. If the percentage is set to 80%, the cloud platform will randomly select 8,000 of the 10,000 samples as the learning sample set, and the remaining 2,000 as the verification sample set.
[0159] In this way, the cloud platform not only ensures the independence of the learning sample set and the validation sample set, but also ensures the diversity and representativeness of the samples. The learning sample set will be used to train the vehicle flow prediction network, while the validation sample set is used to evaluate the performance of the model during the training process to ensure that the model has good generalization ability.
[0160] Step S4A12: If the category of the vehicle flow feature representation sample set of each cloud platform conforms to the distributed data category with different feature spaces but the same sample space, then based on the set percentage, sparse extraction is performed on the target vehicle flow feature representation and priori mark in the vehicle flow feature representation sample set of the first cloud platform among the P cloud platforms, and based on the set percentage, sparse extraction is performed on the target vehicle flow feature representation in the vehicle flow feature representation sample set of cloud platforms other than the first cloud platform among the P cloud platforms, so as to obtain the learning sample set of each cloud platform and the verification sample set of each cloud platform.
[0161] In step S4A12, the cloud platform processes another specific distributed data category, that is, a distributed data category with different feature spaces but the same sample space, which usually occurs in the scenario of longitudinal federated learning. In the context of collaborative data processing at the edge of highways, this means that the traffic flow data collected by different cloud platforms correspond to the same samples (for example, the same road section, the same time period), but each cloud platform only has part of the feature information of these samples. In order to effectively use data to train the vehicle flow prediction network in this scenario, the cloud platform adopts a special sparse extraction strategy to divide the learning sample set and the verification sample set. Specifically, the cloud platform first selects a cloud platform as a benchmark cloud platform (assuming it is the first cloud platform), which has a complete feature space, that is, contains all feature information related to vehicle flow prediction. Next, the cloud platform performs sparse extraction on the vehicle flow feature representation sample set on the benchmark cloud platform according to a set percentage. This percentage is usually determined based on experience or experiments and is used to divide the ratio of learning samples and verification samples. During the extraction process, the cloud platform will simultaneously select the target vehicle flow feature representation and the corresponding prior label (that is, the real vehicle flow data). These extracted samples will constitute the learning sample set and the verification sample set of the benchmark cloud platform.
[0162] For other cloud platforms (i.e., non-benchmark cloud platforms), since they only have partial feature information, the sparse extraction strategy is slightly different. The cloud platform extracts the target traffic flow feature representation only from the traffic flow feature representation sample set of these cloud platforms according to the same percentage, without extracting the prior label. This is because these cloud platforms lack a complete feature space and cannot provide a complete feature vector corresponding to the prior label. These extracted samples will be used as part of the learning sample set of the non-benchmark cloud platform. Finally, the cloud platform integrates the learning sample set and the validation sample set of the benchmark cloud platform and other cloud platforms. The integrated learning sample set will contain sample data from different cloud platforms, which are the same in sample space (corresponding to the same road section and the same time period), but may differ in feature space. The integrated validation sample set only contains sample data from the benchmark cloud platform because these sample data have a complete feature space and corresponding prior labels.
[0163] For a specific example, suppose there are three cloud platforms involved in the training of the vehicle flow prediction network. Among them, cloud platform A, as the benchmark cloud platform, has a complete feature space, including features such as vehicle speed, traffic flow, and road occupancy rate; cloud platform B only has features of vehicle speed and traffic flow; cloud platform C only has features of road occupancy rate. If the set percentage is 80%, then the cloud platform will randomly select 80% of the samples from cloud platform A as the learning sample set and the verification sample set; and randomly select 80% of the samples from cloud platform B and cloud platform C as their respective learning sample sets, but these samples do not contain prior labels. Finally, these samples will be integrated together to train the vehicle flow prediction network.
[0164] In this way, the cloud platform can effectively utilize data from different cloud platforms in the scenario of vertical federated learning, while ensuring the accuracy and effectiveness of the model training process. This division method not only takes into account the differences in the feature space of the data, but also makes full use of the data complementarity between different cloud platforms, which helps to improve the performance of the vehicle flow prediction network.
[0165] As an implementation manner, the step S4D, based on the verification sample set of each cloud platform, evaluates the error of the network internal parameter variable in each cloud platform to obtain a target error value indicating the error of the network internal parameter variable in the P cloud platforms, including:
[0166] Step S4D1: Obtain a phased estimated value of each cloud platform by multiplying the network internal parameter variables in each cloud platform with the target vehicle flow characteristic representation in the verification sample set of each cloud platform.
[0167] In step S4D1, the cloud platform performs a key operation, that is, by multiplying the internal parameters of the network in each cloud platform (usually the weights and biases of the neural network) with the target traffic flow feature representation in the verification sample set of the corresponding cloud platform, to obtain the staged estimated value of each cloud platform. This step is an important part of the model evaluation process, which is used to verify the performance of the network model under the current parameter settings. Specifically, in the embodiment of the present application, each cloud platform maintains a vehicle flow prediction network and holds a verification sample set. The verification sample set contains real highway traffic data to simulate the performance of the model in actual applications. Each sample contains multiple features related to vehicle flow prediction (such as vehicle speed, vehicle flow, road occupancy, etc.), which are encoded as target traffic flow feature representations. In step S4D1, the cloud platform takes a sample from the verification sample set and extracts its target traffic flow feature representation. Then, it multiplies this feature representation with the internal parameters of the vehicle flow prediction network on the current cloud platform. This multiplication operation actually simulates the forward propagation process of the network, that is, the output prediction value is calculated according to the current parameter settings and input features.
[0168] Assume that the vehicle flow prediction network is a simple fully connected neural network with a weight matrix W and a bias vector b, and the target vehicle flow feature of a certain verification sample is represented by x. Then, the interim estimated value y can be calculated by the following formula: y=W*x+b. The "*" here represents matrix multiplication. The calculated y is a interim estimated value, which represents the predicted output of the network for the input feature x under the current parameter settings. This predicted output is a numerical value used to represent the predicted vehicle flow size or level.
[0169] By repeatedly executing step S4D1, the cloud platform can calculate the corresponding interim estimated value for each sample in the validation sample set. These estimated values will be used in subsequent steps to compare with the actual vehicle flow values to evaluate the performance of the network model. In this way, the cloud platform can understand the performance of the network model on unseen data, providing an important basis for subsequent model selection and tuning.
[0170] Step S4D2: Classify the periodic estimated values of each cloud platform through a preset classifier to obtain the target vehicle flow prediction results obtained by each cloud platform.
[0171] In step S4D2, the cloud platform classifies the phased preliminary estimates obtained by each cloud platform in the previous step S4D1 using a preset classifier, thereby obtaining the target vehicle flow prediction result. This step converts the continuous numerical prediction values into more practically meaningful vehicle flow level labels, making the prediction results easier to understand and apply. In the embodiments of the present application, each cloud platform calculates the phased preliminary estimates through forward propagation based on its specific vehicle flow prediction network model and validation sample set. These preliminary estimates are usually a series of continuous numerical values representing the predicted vehicle flow magnitude. However, for traffic management departments, they are more concerned about the levels or grades of vehicle flow, such as "low flow", "medium flow", or "high flow", because these level labels can more intuitively reflect the traffic conditions and provide guidance for traffic scheduling and planning.
[0172] To meet this requirement, the cloud platform introduces a preset classifier in step S4D2. This classifier can be implemented based on various machine learning algorithms, such as logistic regression, decision tree, support vector machine, etc. In the context of highway vehicle flow prediction, a simple threshold classifier may be sufficient. For example, a threshold range can be set. Samples with preliminary estimates less than a certain threshold are classified as "low flow", samples with preliminary estimates between two thresholds are classified as "medium flow", and samples with preliminary estimates greater than the larger threshold are classified as "high flow". Specifically, assume that the preset classifier uses three thresholds: T1, T2, and T3 (T1 < T2 < T3). For the phased preliminary estimate y of a certain cloud platform, if y <= T1, it is classified as "low flow"; if T1 < y <= T2, it is classified as "medium flow"; if y > T2, it is classified as "high flow". In this way, the cloud platform can convert the phased preliminary estimates of each cloud platform into the target vehicle flow prediction result, that is, the vehicle flow level label.
[0173] For example, assume that the phased preliminary estimate of a certain cloud platform is 200, and the preset thresholds are T1 = 100 and T2 = 300 respectively. Since 200 is between T1 and T2, this preliminary estimate will be classified as "medium flow". This process will be repeated for each preliminary estimate in the validation sample set, thereby obtaining the target vehicle flow prediction results of each cloud platform on the validation samples. These results will be used in subsequent error calculation and model evaluation steps to provide guidance for optimizing the vehicle flow prediction network.
[0174] Step S4D3: Determine the target error value based on the target vehicle flow prediction results obtained by each cloud platform and the prior labels in the validation sample sets of each cloud platform.
[0175] In step S4D3, the cloud platform determines the target error value by comparing the target vehicle flow prediction results obtained by each cloud platform on the verification sample set with the prior mark (i.e., the real vehicle flow level label). This step is a key link in evaluating the performance of the vehicle flow prediction network. It quantifies the difference between the network prediction results and the true value, and provides an important basis for subsequent model tuning. In the embodiment of the present application, the verification sample set contains real highway traffic data, and each piece of data has a corresponding prior mark, i.e., the real vehicle flow level label (such as "low flow", "medium flow", "high flow", etc.). In step S4D2, the cloud platform has classified the staged estimated values of each cloud platform through a preset classifier and obtained the target vehicle flow prediction result.
[0176] In step S4D3, the cloud platform first compares the target vehicle flow prediction results with the corresponding prior labels one by one. For each verification sample, if the prediction result is consistent with the prior label, it means that the network's prediction on the sample is accurate; if it is inconsistent, it means that there is an error in the prediction. In order to quantify this error, the cloud platform adopts an error measurement method, such as accuracy, recall, F1 score, etc. Taking accuracy as an example, it calculates the proportion of correct predictions in all verification samples. Specifically, assuming that there are N samples in the verification sample set, and the prediction results of M samples are consistent with the prior labels, the accuracy Accuracy can be calculated by the following formula: Accuracy=M / N.
[0177] This accuracy is a form of expression of the target error value, which intuitively reflects the overall performance of the network model on the validation sample set. The higher the accuracy, the stronger the prediction ability of the network model and the smaller the error. For example, suppose there are 100 samples in the validation sample set of a cloud platform, and the prediction results of 85 samples are consistent with the prior labels. Then, the target error value (expressed in accuracy) of the cloud platform is 85%. This means that the network model made accurate predictions on 85% of the samples, but errors occurred on the remaining 15% of the samples. Based on this error value, the cloud platform can determine whether the network model has reached the preset performance standard, and then decide whether further model tuning or iterative training is needed.
[0178] As an implementation mode, the step S4D3 determines the target error value according to the target vehicle flow prediction results obtained by each cloud platform and the priori marks in the verification sample set of each cloud platform, including:
[0179] Step S4D31: Obtain the error target vehicle flow characteristic representation of the network internal parameter variables in each cloud platform by subtracting the target vehicle flow prediction result obtained by each cloud platform from the priori mark in the verification sample set of each cloud platform.
[0180] In step S4D31, the cloud platform calculates the error target vehicle flow feature representation of the network internal parameter variables by comparing the prior labels (real vehicle flow level labels) of each cloud platform on the verification sample set and the target vehicle flow prediction results. This step is an important part of evaluating the performance of the vehicle flow prediction network, which helps the system quantify the difference between the prediction results and the actual labels.
[0181] In the embodiment of the present application, each cloud platform holds a verification sample set, which contains real traffic data and its corresponding priori labels. The priori labels are usually manually labeled according to the actual situation, representing the real vehicle flow level (such as "low flow", "medium flow", "high flow", etc.). The target vehicle flow prediction result is the predicted value calculated by the vehicle flow prediction network based on the input features, which is also expressed as the vehicle flow level.
[0182] In step S4D31, the cloud platform first takes a sample from the validation sample set and extracts its prior label and the target vehicle flow prediction result. Since both are classification labels, they cannot be directly subtracted numerically, so an indirect method is needed to quantify the difference between them.
[0183] A common approach is to assign a numerical label to each traffic level. For example, you can assign a label of 1 to "low traffic", 2 to "medium traffic", and 3 to "high traffic". The cloud platform then compares the numerical labels of the prior mark and the target prediction result and calculates the difference between them. This difference represents the prediction error of the network on this sample.
[0184] For example, suppose a validation sample is labeled as “medium traffic” (numerical label 2), and the target prediction result given by the vehicle flow prediction network is “low traffic” (numerical label 1). Then, the cloud platform will calculate the difference between the two numerical labels, that is, 2-1=1. This difference of 1 is the error target vehicle flow characteristic representation of the network internal parameter variable on the sample.
[0185] By repeating the above process, the cloud platform can calculate the corresponding error target traffic flow feature representation for each sample in the validation sample set. These error feature representations will be used in subsequent steps to further quantify the size of the network prediction error and evaluate the overall performance of the network. In this way, the cloud platform can accurately understand the performance of the network on the validation samples, providing strong support for subsequent model tuning and iterative training.
[0186] Step S4D32: Flip the error target vehicle flow characteristic representation of the network internal parameter variables in each cloud platform, and multiply the flipped characteristic representation with the error target vehicle flow characteristic representation of the network internal parameter variables in each cloud platform to obtain the error value of the network internal parameter variables in each cloud platform. In step S4D32, the cloud platform processes the error target vehicle flow characteristic representation of the network internal parameter variables in each cloud platform to further quantify the error size of the network when predicting vehicle flow. This step is a key step in the process of evaluating the performance of the vehicle flow prediction network. It operates on the error characteristic representation to obtain a value that can reflect the size of the network prediction error.
[0187] First, it should be clear that in step S4D31, the cloud platform has calculated the error target vehicle flow feature representation of each cloud platform on the verification sample set. These feature representations are actually a set of values, each of which corresponds to the prediction error of a verification sample. In the embodiment of the present application, these error values reflect the prediction accuracy of the vehicle flow prediction network on a specific sample.
[0188] In step S4D32, the cloud platform first flips (i.e., transposes) these error target traffic flow feature representations. However, in this particular scenario, since the error value is one-dimensional (each sample corresponds to an error value), the transposition operation does not actually change the structure of the data. The transposition here is more to keep consistent with subsequent operations and ensure the correctness of the data format. Next, the cloud platform multiplies the flipped error feature representation (which is actually the same as the original error feature representation) with itself. This multiplication operation actually calculates the square of each error value. Why calculate the square? This is because the square operation can amplify the impact of the error, making the larger error occupy a larger proportion of the total error. This is very useful for subsequent error analysis and model tuning. For example, suppose there are three validation samples with error target traffic flow feature representations of 1, 2, and 3 (these values represent the size of the prediction error). In step S4D32, the computer will square these error values and obtain 1²=1, 2²=4, and 3²=9. These squared error values are the error values of the network internal parameters in each cloud platform.
[0189] Through this step, the cloud platform obtains an array or vector containing the square of all sample errors. This array or vector intuitively reflects the distribution of the network's prediction error on the validation sample set. A larger square error value indicates that the network has a larger prediction error on this sample and needs to be focused on and tuned. Finally, these error values will be used for calculations in subsequent steps, such as the mean calculation in step S4D33, to further quantify the overall prediction error of the network and evaluate the performance of the vehicle flow prediction network.
[0190] Step S4D33: Calculate the mean of the error values of the network internal parameter variables in each cloud platform to obtain a mean error, and use the mean error as the target error value.
[0191] In step S4D33, the cloud platform calculates the mean error of the network internal parameter variables in each cloud platform to obtain an indicator that can comprehensively reflect the size of the network prediction error, namely the mean error. This mean error will be used as the target error value to evaluate the overall performance of the vehicle flow prediction network on the verification sample set.
[0192] In the embodiment of the present application, each cloud platform holds a set of validation samples, and has calculated the error value of each sample through the previous steps (these error values reflect the size of the network's prediction error on the sample). These error values form an error value array or vector, which contains all the prediction error information of the network on the validation sample set. In step S4D33, the cloud platform performs mean calculation on this error value array or vector. Mean calculation is a statistical method used to calculate the average level of a set of values. Here, it helps the cloud platform integrate multiple discrete error values into a single numerical indicator, namely the mean error.
[0193] Specifically, the cloud platform adds all the elements in the error value array or vector and divides it by the number of elements (that is, the number of validation samples) to obtain the mean error. This mean error represents the average prediction error of the network on the validation sample set and is an indicator for comprehensively evaluating network performance.
[0194] For example, suppose the validation sample set contains 100 samples, each with a corresponding error value. These error values form an error value array containing 100 elements. The cloud platform adds these 100 error values together to get the sum, and then divides it by 100 (the number of samples) to get the mean error. Assuming the sum is 1000, the mean error is 10 (1000 / 100=10).
[0195] This mean error is the target error value, which intuitively reflects the overall prediction error size of the vehicle flow prediction network on the verification sample set. By comparing the target error values under different networks or different iteration rounds, the cloud platform can determine whether the network has reached the preset performance standard, and then decide whether further model tuning or iterative training is needed. In the embodiment of the present application, this helps the traffic management department understand the prediction accuracy of the network model and take appropriate measures to optimize the traffic management strategy as needed.
[0196] In another aspect of an embodiment of the present application, a low-latency edge collaborative data processing method is provided. In this embodiment, it is introduced from the perspective of an edge box, that is, the low-latency edge collaborative data processing method is applied to an edge box, and the edge box is communicatively connected to a target cloud platform. The method includes:
[0197] Step S100: Acquire target high-speed monitoring data;
[0198] Step S200: performing a data feature extraction operation on the target highway monitoring data to obtain a target vehicle flow feature representation of the target highway monitoring data;
[0199] Step S300: According to the target vehicle flow characteristic representation of the target highway monitoring data, the target highway monitoring data is classified through the target vehicle flow prediction network in the target cloud platform to obtain the target vehicle flow prediction result of the target highway monitoring data, wherein the target vehicle flow prediction network is debugged based on the method of the above embodiment. The specific principle will not be repeated here.
[0200] The present application embodiment provides a cloud platform, such as Figure 3 As shown, the cloud platform 106 includes: a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, such as through a bus 102. Optionally, the cloud platform 106 may also include a transceiver 104. It should be noted that in actual applications, the transceiver 104 is not limited to one, and the structure of the cloud platform 106 does not constitute a limitation on the embodiments of the present application.
[0201] An embodiment of the present application provides a cloud platform, and the cloud platform in the embodiment of the present application includes: one or more processors; a memory; one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more programs are executed by the processor, the method of the first aspect mentioned above is implemented.
[0202] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A low-latency edge collaborative data processing method, characterized in that: include: Get X high-speed monitoring data; Grouping the X high-speed monitoring data to obtain P high-speed monitoring data sets, where P is a positive integer greater than 1 and P is less than X; Performing data feature extraction operations on the P high-speed monitoring data sets through the P cloud platforms respectively, to obtain a sample set representing vehicle flow characteristics of each of the P cloud platforms; Adjust the network internal parameter variables in each cloud platform according to the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms, and perform the F-th round of repeated debugging on the adjusted network internal parameter variables in each cloud platform according to the vehicle flow characteristic representation sample set of each cloud platform, to obtain P debugged target vehicle flow prediction networks, where F=E+1; Deploy the target vehicle flow prediction network corresponding to each cloud platform to the corresponding edge box so that the edge box can perform target vehicle flow prediction; Wherein, each of the cloud platforms is in communication connection with at least one edge box, and for each of the edge boxes, when performing target vehicle flow prediction, it includes: Obtain target high-speed monitoring data; Performing a data feature extraction operation on the target highway monitoring data to obtain a target vehicle flow feature representation of the target highway monitoring data; According to the target vehicle flow characteristic representation of the target highway monitoring data, the target highway monitoring data is classified through the deployed corresponding target vehicle flow prediction network to obtain the target vehicle flow prediction result of the target highway monitoring data; The method comprises: adjusting the network internal parameter variables in each cloud platform according to the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms, and performing the F-th round of repeated debugging on the adjusted network internal parameter variables in each cloud platform according to the vehicle flow characteristic representation sample set of each cloud platform to obtain the debugged P target vehicle flow prediction networks, including: Adjusting the network internal parameter variables in each of the cloud platforms by using the aggregated parameter variables of the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms; According to the sample set representing the vehicle flow characteristics of each cloud platform, the network internal parameter variables in each cloud platform are repeatedly debugged for the Fth round until the number of repeated rounds of the network internal parameter variables in each cloud platform is equal to the maximum number of repeated rounds determined in advance or the network internal parameter variables in each cloud platform converge, thereby obtaining the P target vehicle flow prediction networks; The initial network internal parameter variables in each cloud platform are consistent, and each of the P high-speed monitoring data sets includes X / P high-speed monitoring data; Alternatively, the method of adjusting the network internal parameter variables in each cloud platform according to the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms, and performing the F-th round of repeated debugging on the adjusted network internal parameter variables in each cloud platform according to the vehicle flow characteristic representation sample set of each cloud platform to obtain the debugged P target vehicle flow prediction networks, includes: Sparsely extracting the sample sets representing the traffic flow characteristics of each cloud platform to obtain the learning sample sets of each cloud platform and the verification sample sets of each cloud platform; Adjusting the network internal parameter variables in each of the cloud platforms by using the aggregated parameter variables of the P network internal parameter variables obtained after completing the E-th round of repeated debugging in the P cloud platforms; According to the learning sample sets of each cloud platform, perform an F-th round of repeated debugging on the network internal parameter variables in each cloud platform to obtain P network internal parameter variables obtained after completing the F-th round of repeated debugging in the P cloud platforms; According to the verification sample sets of the respective cloud platforms, the errors of the network internal parameter variables in the respective cloud platforms are evaluated to obtain target error values indicating the errors of the network internal parameter variables in the P cloud platforms; Determine whether the target error value meets the evaluation conditions. If the target error value meets the evaluation conditions, obtain the P target vehicle flow prediction networks; if the target error value does not meet the evaluation conditions, adjust the network internal parameters in each cloud platform through the aggregated parameters of the P network internal parameters obtained after completing the Fth round of repeated debugging in the P cloud platforms, and perform the Gth repeated debugging on the network internal parameters in each cloud platform according to the learning sample set of each cloud platform, until the target error value meets the evaluation conditions, obtain the P target vehicle flow prediction networks, wherein G=F+1.
2. The method according to claim 1, characterized in that The method of performing an F-th round of repeated debugging on the network internal parameter variables in each cloud platform according to the vehicle flow characteristic representation sample set of each cloud platform until the number of repeated rounds of the network internal parameter variables in each cloud platform is equal to a predetermined maximum number of repeated rounds or the network internal parameter variables in each cloud platform converge, and obtaining the P target vehicle flow prediction networks includes: Multiply the network internal parameter variables in each cloud platform with the target vehicle flow characteristic representation in the vehicle flow characteristic representation sample set of each cloud platform, subtract the multiplication result from the priori mark in the vehicle flow characteristic representation sample set of each cloud platform to obtain the subtraction result, multiply the subtraction result with the target vehicle flow characteristic representation in the vehicle flow characteristic representation sample set of each cloud platform to obtain the bias vector of the network internal parameter variables in each cloud platform; Multiplying the tuning parameters of the network internal parameters in each cloud platform with the bias vectors of the network internal parameters in each cloud platform to obtain adjustment results of the network internal parameters in each cloud platform; Obtain P network internal parameter variables obtained after completing the Fth round of repeated debugging in the P cloud platforms by subtracting the adjustment result of the network internal parameter variables in each cloud platform from the network internal parameter variables in each cloud platform; If F is equal to the predetermined maximum number of repetition rounds or the bias vector of the network internal parameter variables in each cloud platform is equal to 0, the P target vehicle flow prediction networks are obtained; If F is less than the predetermined maximum number of repetitions or the bias vector of the network internal parameters in each cloud platform is equal to 0, then based on the vehicle flow characteristic representation sample set of each cloud platform and the P network internal parameters obtained after completing the Fth round of repeated debugging in the P cloud platforms, the network internal parameters in each cloud platform are repeatedly debugged for the Gth time until the number of repetitions of the network internal parameters in each cloud platform is equal to the predetermined maximum number of repetitions or the network internal parameters in each cloud platform converge, and the P target vehicle flow prediction networks are obtained, wherein G=F+1.
3. The method according to claim 1, characterized in that The sparse extraction of the vehicle flow feature representation sample sets of each cloud platform to obtain the learning sample sets of each cloud platform and the verification sample sets of each cloud platform includes: According to the category of the sample set representing the characteristics of the traffic flow of each cloud platform, the sample set representing the characteristics of the traffic flow of each cloud platform is sparsely extracted based on the set percentage to obtain the learning sample set of each cloud platform and the verification sample set of each cloud platform.
4. The method according to claim 3, characterized in that The method of obtaining the learning sample set of each cloud platform and the verification sample set of each cloud platform by sparsely extracting the sample set of each cloud platform based on the set percentage according to the category of the sample set of vehicle flow characteristics of each cloud platform comprises: If the category of the sample set of vehicle flow feature representations of each cloud platform is a distributed data category that conforms to the consistent feature space but different sample spaces, then based on the set percentage, sparse extraction is performed on the target vehicle flow feature representations and priori labels in the sample set of vehicle flow feature representations of each cloud platform to obtain the learning sample set of each cloud platform and the verification sample set of each cloud platform; If the category of the vehicle flow feature representation sample set of each cloud platform conforms to the distributed data category with different feature spaces but the same sample space, then based on the set percentage, sparse extraction is performed on the target vehicle flow feature representation and priori mark in the vehicle flow feature representation sample set of the first cloud platform among the P cloud platforms, and based on the set percentage, sparse extraction is performed on the target vehicle flow feature representation in the vehicle flow feature representation sample set of cloud platforms other than the first cloud platform among the P cloud platforms, so as to obtain the learning sample set of each cloud platform and the verification sample set of each cloud platform.
5. The method according to claim 1, characterized in that The step of evaluating the errors of the network internal parameter variables in each cloud platform according to the verification sample sets of each cloud platform to obtain target error values indicating the errors of the network internal parameter variables in the P cloud platforms includes: By multiplying the network internal parameter variables in each cloud platform with the target vehicle flow characteristic representation in the verification sample set of each cloud platform, a phased estimated value of each cloud platform is obtained; Classifying the periodic estimated values of each cloud platform by a preset classifier to obtain the target vehicle flow prediction results obtained by each cloud platform; The target error value is determined according to the target vehicle flow prediction results obtained by each cloud platform and the priori marks in the verification sample set of each cloud platform.
6. The method according to claim 5, characterized in that The determining the target error value according to the target vehicle flow prediction results obtained by each cloud platform and the priori marks in the verification sample set of each cloud platform includes: Obtain the error target vehicle flow characteristic representation of the network internal parameter variables in each cloud platform by subtracting the target vehicle flow prediction result obtained by each cloud platform from the priori mark in the verification sample set of each cloud platform; Flipping the error target vehicle flow characteristic representation of the network internal parameter variable in each of the cloud platforms, and multiplying the flipped characteristic representation with the error target vehicle flow characteristic representation of the network internal parameter variable in each of the cloud platforms to obtain the error value of the network internal parameter variable in each of the cloud platforms; The error values of the network internal parameter variables in each cloud platform are averaged to obtain a mean error, and the mean error is used as the target error value.
7. An edge collaborative data processing system, characterized in that: It includes multiple edge boxes and multiple cloud platforms, and the multiple edge boxes and the multiple cloud platforms are communicatively connected, wherein the cloud platform includes: one or more processors; memory; one or more computer programs; wherein one or more of the computer programs are stored in the memory and configured to be executed by one or more of the processors, and when the one or more computer programs are executed by the processor, the method described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Side cloud collaborative digital twin intelligent production scheduling application operation position adaptation method
CN113112088A
Cloud edge collaborative machine learning deployment application method and device for millimeter-wave radar intersection traffic flow monitoring system
CN113837222A
Cloud-edge-end cooperative highway cloud control system and control method
US20210295684A1
Cloud-edge collaborative processing system and method for ICU data based on third-party business
US20230395253A1
