Method and system for offline modeling of quality of service prediction for connected vehicles

Abstract

A route prediction system implements a method for utilizing real-time mobile communication network data, the method comprising: receiving the real-time mobile communication network data. The method further comprises: aggregating the real-time mobile communication network data with historical data to form a training set; filtering the training set to be relevant to service route predictions for connected vehicles; training multiple offline models using the filtered training set; and selecting a best-performing propagation model from the multiple offline models.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 012042, filed April 17, 2020, which is hereby incorporated by reference. Technical Field

[0003] Embodiments of the present invention relate to the field of connected vehicle navigation using mobile networks; and more particularly to methods and apparatus for providing improved routing of connected vehicles using real-time mobile communication network information. Background Art

[0004] An autonomous vehicle navigates independently of local human control. As used herein, an 'autonomous' vehicle refers to a vehicle that is self-governing. Any type of vehicle may serve as an autonomous vehicle, including cars, trucks, drones, airplanes, ships, and similar vehicles of any size or type. Autonomous operation generally indicates that the autonomous vehicle is capable of navigating in an uncontrolled navigation environment where significant uncertainties may exist, such that the autonomous vehicle is able to compensate for these uncertainties and system problems or failures without human intervention. The level of automation may be a category such as levels 0-5 as defined by the Society of Automotive Engineers (SAE).

[0005] Autonomous vehicles utilize a large amount of data to make navigation decisions. Some of this data is collected locally on the vehicle via sensors, while other data is received via mobile communication systems. Local sensors can include lidar, stereo vision, global positioning systems (GPS), inertial measurement units (IMUs), and similar sensors. The locally collected data, along with locally stored terrain, maps, geospatial, or similar data, can be processed as input by the autonomous vehicle's navigation algorithms. The navigation algorithms can control various systems of the autonomous vehicle, such as the propulsion system, steering system, and similar systems. The navigation algorithms can also utilize computing resources and data that are not local to the autonomous vehicle.

[0006] An autonomous vehicle connected to external computing and data resources via a communications network is a 'connected vehicle,' which can use a mobile communications system to receive and transmit large amounts of data between the autonomous vehicle's onboard computing system and a remote computing system. The received data may include data about local conditions, such as weather, traffic, road conditions, and the like. The transmitted data may include data collected by the autonomous vehicle's sensors. However, in situations where the mobile communications network has poor or no coverage, making reliance on external computing resources and data problematic, the exchange of this data with the external computing resources may be interrupted or degraded to a lower-than-required quality of service (QoS) level. Summary of the Invention

[0007] In a first embodiment, a method for a route prediction system utilizing real-time mobile communication network data includes: receiving real-time mobile communication network data. The method further includes: aggregating the real-time mobile communication network data with historical data to form a training set; filtering the training set to be relevant to service route predictions for connected vehicles; training multiple offline models using the filtered training set; and selecting a best-performing propagation model from the multiple offline models.

[0008] In another embodiment, a network device is configured to implement a method for a route prediction system utilizing real-time mobile communication network data. The network device includes: a non-transitory machine-readable storage medium having a route prediction block stored therein; and a processor coupled to the non-transitory machine-readable storage medium. The processor executes the route prediction block, which is configured to: receive real-time mobile communication network data; aggregate the real-time mobile communication network data with historical data to form a training set; filter the training set to be relevant to service route predictions for connected vehicles; use the filtered training set to train multiple offline models; and select a best-performing propagation model from the multiple offline models.

[0009] In a further embodiment, a networked device is configured to execute multiple virtual machines. The multiple virtual machines implement network function virtualization (NFV). The network device includes: a non-transitory machine-readable storage medium having a route prediction block stored therein; and a processor coupled to the non-transitory machine-readable storage medium. The processor executes at least one of the multiple virtual machines. The at least one of the multiple virtual machines executes the route prediction block. The route prediction block receives real-time mobile communication network data; aggregates the real-time mobile communication network data with historical data to form a training set; filters the training set to correlate with service route predictions for connected vehicles; uses the filtered training set to train multiple offline models; and selects a best-performing propagation model from the multiple offline models.

[0010] In one embodiment, an electronic device is located in a software-defined networking (SDN) network that includes multiple data plane devices. The electronic device includes: a non-transitory machine-readable storage medium having a prediction service stored therein; and a processor coupled to the non-transitory machine-readable storage medium. The processor executes the prediction service. The prediction service receives real-time mobile communication network data; aggregates the real-time mobile communication network data with historical data to form a training set; filters the training set to be relevant to service route predictions for connected vehicles; trains multiple offline models using the filtered training set; and selects a best-performing propagation model from the multiple offline models. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention may best be understood by referring to the following description and accompanying drawings which illustrate embodiments of the invention. In the drawings:

[0012] Figure 1 is a simplified diagram of one embodiment of an overview of a prediction system.

[0013] Figure 2A is a simplified diagram of one embodiment of the online process of the prediction system.

[0014] Figure 2B is a flow chart of one embodiment of an online process implemented by a route prediction block (RPB).

[0015] Figure 3A is a simplified diagram of the offline process used to predict the system.

[0016] Figure 3B is a flow chart of one embodiment of the operation of an RPB implementing an offline process.

[0017] Figure 4 is a simplified diagram of an example of a propagation model.

[0018] Figure 5 is a flow diagram of one example embodiment of a process for filtering network data to be used by an offline process.

[0019] Figure 6 is a flow chart of one embodiment of a process of a mapping function that correlates mobile communication network data with routes.

[0020] Figure 7 is a flow chart of one embodiment of an offline SLI modeling implementation.

[0021] Figure 8 is a flow diagram of one example embodiment of a feedback function for an online SLI scoring implementation.

[0022] Figure 9 is a flow chart of an example embodiment of an offline retraining process.

[0023] Figure 10 is a simplified diagram of one embodiment of a system for proxying information between multiple mobile network operator (MNO) sources.

[0024] Figure 11 is a flow chart of one embodiment of a process for integrating an autonomous driving fleet (ADF) with a predictive system.

[0025] Figure 12 is a flow chart of one embodiment of an online process implemented on a cloud platform (CP) to support information brokers.

[0026] Figure 13 is a flow chart of one example embodiment of a process for handling route information at a CP to support information brokers for prediction systems and services.

[0027] Figure 14 is a simplified diagram of one embodiment of a cloud implementation of prediction systems and services.

[0028] Figure 15A Connectivity between network devices (NDs) within an exemplary network and three exemplary implementations of NDs are shown according to some embodiments of the present invention.

[0029] Figure 15B An exemplary manner of implementing a dedicated network device according to some embodiments of the present invention is shown.

[0030] Figure 15C Various exemplary ways in which virtual network elements (VNEs) may be coupled according to some embodiments of the present invention are shown.

[0031] Figure 15D A network with a single network element (NE) on each of the NDs is shown according to some embodiments of the present invention, and within this straightforward approach, a traditional distributed approach (common with traditional routers) is contrasted with a centralized approach (also known as network control) for maintaining reachability and forwarding information.

[0032] Figure 15E A simple case according to some embodiments of the present invention is shown, where each of the NDs implements a single NE, but a centralized control plane has abstracted (represented) multiple NEs in different NDs into (represented) a single NE in one of (one or more) virtual networks.

[0033] Figure 15F A scenario according to some embodiments of the present invention is shown, wherein multiple VNEs are implemented on different NDs and coupled to each other, and wherein a centralized control plane has abstracted the multiple VNEs so that they appear as a single VNE within one of the virtual networks.

[0034] Figure 16 A generic control plane apparatus is shown with centralized control plane (CCP) software 1650 according to some embodiments of the present invention. DETAILED DESCRIPTION

[0035] The following description describes methods and apparatus for improving the field of wireless mobile communication systems and connected vehicles. A connected vehicle may include any type of vehicle that has the capability to communicate with an external computing resource. Connected vehicles may include autonomous vehicles such as drones and driverless cars and trucks, autonomous sidewalk robots, vehicles or applications that consume media or other data sources in real time while on the move. Embodiments provide methods and apparatus for accelerating the arrival of connected vehicles by collaborating with mobile communication network operators to use real-time data from the mobile communication networks to provide route and trip-specific data for connected vehicles, including road-focused predictions of the quality of service for connected vehicles along different potential routes, trips, or route segments, enabling greater certainty about the level of mobile service that a connected vehicle will experience on different routes of a potential trip.

[0036] In the following description, numerous specific details are set forth, such as logic implementation, operation codes, means for specifying operands, resource partitioning / sharing / duplication implementations, types and interrelationships of system components, and logic partitioning / integration selections, in order to provide a more thorough understanding of the present invention. However, those skilled in the art will recognize that the present invention can be practiced without such specific details. In other cases, control structures, gate-level circuits, and complete software instruction sequences are not shown in detail to avoid obscuring the present invention. Using the included descriptions, one of ordinary skill in the art will be able to implement appropriate functionality without undue experimentation.

[0037] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0038] Text within brackets and blocks with dashed borders (e.g., long dash, short dash, dot-dash, and dotted lines) may be used herein to indicate optional operations that add additional features in embodiments of the present invention. However, such markings should not be taken to mean that these are the only options or optional operations, and / or that blocks with solid borders are not optional in certain embodiments of the present invention.

[0039] In the following description and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended as synonyms for each other. "Coupled" is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, cooperate or interact with each other. "Connected" is used to indicate that communication is established between two or more elements that are coupled to each other.

[0040] Electronic devices use machine-readable media (also known as computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid-state drives, read-only memory (ROM), flash memory devices, phase-change memory) and machine-readable transmission media (also known as carriers) (e.g., electrical, optical, radio, acoustic, or other forms of propagated signals—such as carrier waves, infrared signals) to store and transmit (internally and / or between other electronic devices over a network) code (consisting of software instructions and sometimes referred to as computer program code or computer programs) and / or data. Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., where the processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, other electronic circuit, or a combination of one or more of the foregoing) coupled to one or more machine-readable storage media that stores code for execution on the set of processors and / or stores data. For example, an electronic device may include non-volatile memory containing code, as non-volatile memory can retain code / data even when the electronic device is turned off (when power is removed), and when the electronic device is turned on, the portion of code to be executed by the electronic device's processor(s) is typically copied from the slower non-volatile memory to the electronic device's volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)). A typical electronic device also includes a set of one or more physical network interfaces (NIs) to establish a network connection with other electronic devices (to transmit and / or receive code and / or data using propagated signals). For example, the set of physical NIs (or the combination of a set of (one or more) physical NIs and a set of processors that execute code) may perform any formatting, encoding, or translation to allow the electronic device to send and receive data, whether via a wired and / or wireless connection. In some embodiments, the physical NI may include radio circuitry capable of receiving data from other electronic devices via a wireless connection and / or sending data to other devices via a wireless connection. This radio circuitry may include (one or more) transmitters, (one or more) receivers and / or (one or more) transceivers suitable for radio frequency communication. The radio circuitry may convert digital data into a radio signal with appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signal may then be transmitted to (one or more) appropriate recipients via an antenna. In some embodiments, the set of (one or more) physical NIs may include (one or more) network interface controllers (NICs), also known as network interface cards, network adapters, or local area network (LAN) adapters. (One or more) NICs may facilitate connecting electronic devices to other electronic devices to allow them to communicate via wires by plugging cables into physical ports connected to the NICs.One or more parts of the embodiments of the present invention may be implemented using different combinations of software, firmware and / or hardware.

[0041] A network device (ND) is an electronic device that communicatively interconnects other electronic devices on a network (e.g., other network devices, end-user devices). Some network devices are "multi-service network devices" that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, quality of service, and / or subscriber management) and / or provide support for multiple application services (e.g., data, voice, and video).

[0042] Overview

[0043] Autonomous vehicles and other types of vehicles equipped with communication devices rely on mobile communication systems (e.g., 4G or 5G mobile communication networks) to handle a wide range of data transfers between the vehicle and computing resources accessible via the mobile communication network with a certain quality of service (QoS). Such vehicles, referred to herein as 'connected vehicles,' rely on external data inputs to make navigation and other decisions. Each connected vehicle operator or manufacturer may have different requirements, depending on the design of their connected vehicles and / or the operational and regulatory environment in which the connected vehicles operate.

[0044] The QoS of mobile communication networks varies with time and location. The variations can be large or small and can occur over short periods of time or over short distances. In some cases, these variations are driven by known patterns such as road traffic and are therefore seasonal and easily predictable. However, in some cases, these variations are not easily predictable and can pose significant risks to the ability of connected vehicles and their operators to meet key operational and regulatory requirements. Some situations that may give rise to these unexpected variations include gatherings of users of mobile communication network services (e.g., concerts, traffic accidents, and similar events) or unexpected events within the mobile communication network such as tower failures. Connected vehicles and their operators mitigate this risk by entering into mobile communication network service contracts with multiple Mobile Network Operators (MNOs) with the expectation that at any given time and place, one of the MNOs will be able to meet the connected vehicle's QoS requirements.

[0045] An autonomous vehicle is one example of a connected vehicle. Autonomous vehicles are discussed herein by way of example and not limitation. Those skilled in the art will appreciate that the principles, functions, processes, and structures discussed herein with respect to autonomous vehicles are applicable to other types of connected vehicles, and for the sake of clarity and brevity, reference is made to autonomous vehicles. The processes of the prediction system may be applicable to other types of connected vehicles that are not necessarily autonomous, such as vehicles that provide assistance to the driver or pilot via any type of feedback or information system, or vehicles or applications that consume or provide media or other data sources in real time while on the move (e.g., trucks for television networks, ambulances with connected mobile devices, public transportation with localized services such as WiFi).

[0046] The QoS provided by a mobile communication network may vary across different areas of the mobile communication network that a connected vehicle may traverse. Variations in QoS across a mobile communication network may be due to different network capabilities in different parts of the network (e.g., 5G or 4G service may not be available in all areas of the network), due to temporary changes in network conditions (e.g., outages or heavy network traffic), weather conditions that affect signal quality, or similar issues.

[0047] Existing connected vehicle navigation systems focus on minimizing travel time when determining a route from point A to point B, without taking into account the mobile network QoS along that route. Some systems consider historical mobile network QoS along a route, where historical mobile network QoS is collected through user reports of past mobile network QoS experienced in different areas on the corresponding mobile network. These systems that use historical mobile network QoS data use crowdsourced data and are inherently not real-time. As used herein, 'real-time' data is data that represents conditions within seconds, minutes, or hours of data collection, rather than days, weeks, or months of data collection. 'Historical' data is data collected days, weeks, or months prior to the current time. While these crowdsourced historical data tools and / or techniques are helpful, they become inapplicable or inaccurate if insufficient data is available for a specific route (a common problem with crowdsourced data) or if conditions change, which can occur frequently (e.g., every hour or even every millisecond) due to the dynamic mobility patterns of the mobile network environment.

[0048] Connected vehicles and their operators face avoidable uncertainty when selecting a route from point A to point B. Existing route selection algorithms may recommend a route that has met the connected vehicle's QoS connectivity requirements in the past, but during that particular moment or in the near future, while the connected vehicle is traveling along the recommended route, QoS may degrade and jeopardize the outcome of the trip and the achievement of its purpose. Severe QoS degradation can endanger the public and lead to incidents that negatively impact the image and acceptance of connected vehicles.

[0049] The non-determinism of existing route selection algorithms is one of the main reasons why connected transport operators contract mobile communication services with two or more different MNOs in an attempt to build redundancy and reliability into their operations. However, because existing route selection algorithms do not take into account real-time mobile communication network service quality, the increased cost of contracting services with multiple MNOs does not always correlate with a reduced risk of insufficient QoS, and is therefore often an unnecessary cost.

[0050] Embodiments overcome these problems and drawbacks of the prior art. Embodiments provide an improved optimization method by which a system can optimize route selection options for a connected vehicle with a certain degree of confidence for different levels of QoS of the mobile communication network supported by the connected vehicle and for different mobile network operators (MNOs) accessible to the connected vehicle.

[0051] Embodiments provide a route prediction method and system based on real-time mobile communication network data. This route prediction method and system can be used with any connected vehicle or related application that would benefit from knowing the expected mobile communication network QoS for a planned trip. Based on the expected start time of the trip, the prediction system can determine a recommendation for which MNO to select for each segment of the route and provide this recommendation to the requesting connected vehicle. Consequently, selecting the best available MNO ensures uninterrupted, required QoS for the connected vehicle.

[0052] Example applications include autonomous vehicles capable of conditional driving automation (Level 3) or above (as defined by the Society of Automotive Engineers in its publication SAE J3016, Revision 2018-06 and subsequent revisions), unmanned aerial vehicles (e.g., large unmanned aerial vehicles (UAVs) or small drones), vehicles or applications that consume media or other data sources in real time while on the move (e.g., video games, streaming video applications, and similar software). For the sake of clarity and brevity, the example of an autonomous vehicle system referred to as an autonomous vehicle is used herein as an example and not a limitation. Those skilled in the art will recognize that the principles, structures, and processes described with respect to autonomous vehicles are applicable to these and other similar connected vehicle situations.

[0053] Embodiments of a prediction system and method provide a process for predicting the mobile communication system QoS that a connected vehicle will experience along multiple segments that comprise a route from point A to point B across multiple MNOs using real-time mobile communication network data as the primary data source. The prediction system and method replaces the need to install software on connected vehicles to collect training data by identifying connected vehicles in motion from real-time cellular network data. Individual connected vehicles and their experience information can be identified, and information can be collected from mobile communication network data rather than requiring connected vehicles to report experience information directly. The prediction system and method increase the robustness of the route prediction process relative to using sparse crowdsourced data. The prediction system and method uses radio coverage patterns and telecom domain knowledge expertise to identify and highlight intermittent service degradations such as coverage holes or abnormal QoS degradation.

[0054] A connected vehicle designed to utilize these embodiments can be configured with a profile that identifies information that the connected vehicle will utilize in communicating with a mobile communication network as part of the route selection process. The profile can include identification information of the mobile communication network operator (MNO) with which the connected vehicle has a contract and a preferred order for utilizing the MNO. The configuration information can further include supported mobile communication network service QoS levels (e.g., uplink or downlink throughput of at least 10 Mbps or at least 5 Mbps), which are mapped to service level indicators (SLIs) (e.g., SLI3 for greater than 10 Mbps, SLI2 for greater than 5 Mbps but less than 10 Mbps, and SLI1 for less than 5 Mbps).

[0055] Embodiments include prediction systems and processes provided by one or more electronic devices of the operator of the prediction system (and in some embodiments, one or more of these MNOs). The prediction service receives a request for a route for a particular trip or a prediction for a given route for a particular trip from a connected vehicle. Profile information is also shared with the prediction system. Based on the MNO stored in the profile, and using various mobile communication network parameters and real-time mobile communication network data such as coverage area, required QoS, current subscriber-level and cell-level cellular wireless QoS, antennas, alarms, past performance, handover settings, and reliable propagation models using digital models (e.g., digital terrain models (DTMs), digital elevation models (DEMs), or digital surface models (DSMs)) and / or clutter maps for optimizing and developing realistic prediction algorithms, the prediction system calculates the route or the prediction for the given route. The prediction system's calculations also take into account past predictions and the correlation of these past predictions with the actual mobile communication network QoS experienced by the connected vehicle during the current trip or after completing the trip. The prediction service can also use input from external sources such as mapping companies, weather services, and similar sources.

[0056] The prediction system can receive a destination and determine a route to that destination, or it can receive a set of routes from connected vehicles for analysis. As used herein, a 'set' can refer to any integer term including one. The route is split into multiple segments, and a prediction is created for each segment and for each applicable MNO based on the profile for the connected vehicle. The prediction system determines the most likely QoS level to be experienced in each segment along the route for each MNO, based on the supported levels defined in the profile. Based on the profile or policies set by the prediction system, the prediction process will consider the preferred MNO, previous segment MNO recommendations, and the most desirable key performance indicator (KPI) levels for the prediction.

[0057] In some embodiments, the connected vehicle may include software dedicated to interacting with the prediction system to report the actual QoS of the utilized mobile communication network experienced along the journey along with some other relevant parameters such as location, speed, driving conditions and similar information.

[0058] Embodiments provide advantages over existing technologies by utilizing real-time mobile communication network data for route prediction of connected vehicles. Because resource allocation in mobile communication networks is crucial for supporting mobility and providing QoS, accurately capturing the movement patterns of connected vehicles is very important. Embodiments further provide propagation models for predicting QoS issues in mobile communication networks. Because the environment constantly changes over time, the prediction of path loss or degradation is an important element in the operation of the embodiments' processes and systems. In some embodiments, a combination of machine learning models that consider weather and time aspects is utilized.

[0059] A further aspect of an embodiment is a process and system for purchasing data from available MNOs to provide a connected vehicle with a single point of purchase for navigation and related QoS, regardless of the number of MNOs that the connected vehicle has access to or that provide contracted services to the connected vehicle.

[0060] In some embodiments, the forecasting process may be deployed (e.g., in a distributed manner) within the devices and components of the forecasting system operator. In other embodiments, some components may be deployed within the devices of each MNO to ensure that the respective MNO's proprietary data used for the forecasting process does not leave the components, devices, and facilities of that MNO (i.e., only anonymized, transformed, or similarly obfuscated data is provided outside the associated MNO's mobile communication network). However, the components deployed within the MNO's devices may be managed by the forecasting system operator.

[0061] These aspects of the embodiments can be combined in any combination or separately to enable finding the best route based on a predictive model built on collected data, including real-time data from available mobile communication networks. The predictive model and process collects, analyzes, and acts on mobile communication network data in real time, and utilizes machine learning tools to generate real-time connected transportation route recommendations.

[0062] Figure 1 FIG1 is a simplified diagram of one embodiment of an overview of a forecasting system. The diagram illustrates a high-level view of the end-to-end data flow across an embodiment of the forecasting system. The diagram shows two related data flows. One data flow is an online process flow, while the other data flow is an offline process flow. The diagram illustrates the interrelationship of components that are part of the online and offline processes, with arrows indicating the movement of data between components for both the online and offline processes.

[0063] Whenever the prediction system receives data from a collection of autonomous driving fleets (ADFs) 101 via a cloud platform (CP) 102 or a similar computing environment that takes over the prediction system, an online data flow process is initiated. An ADF is a collection of autonomous vehicles with any number or type of autonomous vehicles. Each ADF can represent a separate service, navigation system, autonomous vehicle manufacturer, or similar vehicle grouping. The autonomous vehicles of each ADF can communicate directly with the prediction system, or the ADF can have a supporting network infrastructure that relays information to the prediction system, such as a mobile communication network provided by an MNO. In some embodiments, ADF 101 is a grouping of autonomous vehicles according to their primary connection MNO. As an example and not a limitation, an example case of ADF 101 utilizing a prediction system is provided. Those skilled in the art will understand that groups of connected vehicles can utilize the prediction system in a similar manner.

[0064] The data received from the ADF 101 may include a route prediction request. In some embodiments, the received data may also include any amount or type of collected sensor, environmental, and driving-related data. The data received from each ADF 101 may be sent by the cloud platform 102 to a route prediction block (RPB) 104. The RPB 104 may be executed on the hardware of the prediction system service provider. In other embodiments, the RPB 104 may be executed at each MNO to process the data collected by that MNO and obtain the prediction. However, even when the RPB 104 is executed on the MNO's hardware, the prediction system service provider may manage the RPB 104. The RPB 104 may provide a route prediction to the requesting ADF 101 or similar entity. Each RPB 104 responds with a route QoS prediction, which is then sent (or sent via the cloud platform 120) to the ADF 101 or specific autonomous vehicle that requested the route prediction. These RPBs 104 may be executed on the same cloud platform 102, on separate computing facilities, or distributed in some other manner. Each RPB 104 may include its own propagation model. The RPB 104 and associated cluster (CC) 105 may be hosted on the MNO network hardware (i.e., on a per-MNO basis). In other embodiments, either the RPB 104 or the CC 105 may be hosted on the hardware of the predictive system service provider. In either case, the predictive system service provider may manage the RPB 104.

[0065] RPB 104 includes a machine learning prediction model, a radio propagation model (referred to herein as a 'propagation model'), and a focused mobility model. The machine learning prediction model estimates and predicts key performance indicators (KPIs) for each vehicle and route segment by taking engineering telecommunication features as input. Mobile communication network features include network KPIs and subscriber KPIs filtered by the focused mobility model. Figure 5 and Figure 6 A focused mobility model is described.

[0066] Based on the data received from ADF 101, the online process can be further divided into an online scoring process and an online experience correction process. The online scoring process is initiated whenever a request is received from ADF 101 with a list of locations, segments, or similar identifiers for which QoS scores or ratings are requested. The prediction system implemented via the corresponding RPB 104 responds to ADF 101 by sending a predicted QoS score for the location, segment, or similar element identified by the request from ADF 101. Multiple online processes can be running simultaneously, and each online process will be tracked by a process or request ID to enable correlation between received requests and responses from the associated RPB 104.

[0067] Whenever the RPB 104 receives ADF 101 experience data, an online experience correction process can be initiated. This experience data is correlated at the RPB 104 with past predictions from the online process. Thus, the predicted QoS provided by the RPB 104 to the ADF 101 is correlated with the experience data returned by the ADF 101, allowing the prediction to be compared with the actual experienced QoS. This correlation of experience data enables measurement of the model's current accuracy while providing a correction mechanism for future predictions. Online experience correction captures information that the ADF 101 can share with the CP 102, for example, at the time of making a route prediction request or at a later time, and uses this information to identify specific deviations and anomalies for a given ADF 101 and account for them in the offline model.

[0068] The offline process is a continuous process that runs all the time or at regular intervals. The offline process collects data from the mobile communication network 106, collects experience data from the ADF 101, and correlates these data with each other. The collected data is enhanced using other sources 103 and data collected in the past. The associated RPB 104 stores this correlated data for historical purposes. The data collected from the mobile communication network 106 may include QoS information such as throughput metrics, operating status, latency, and similar information. In some embodiments, each MNO can pass mobile communication network data to an associated cluster (CC) 105. The CC can be managed by the MNO, or by a separate provider of the operation forecast system. The CC can aggregate information for the mobile communication network associated with the MNO, or similarly organize or aggregate data to be provided to the RPB 104 associated with the MNO.

[0069] The offline process can be further divided into an offline service level indicator (SLI) modeling process and an offline retraining process. Multiple offline models can be created from a combination of these sources. Offline models can include SLI models, mobility models, and / or propagation models, which are generally referred to herein as offline models. The SLI model matches network information, such as cell tower KPI values, with SLI values ​​in a geographic area associated with a route or segment. SLI modeling involves feeding network data to all offline models and then scoring them based on performance. Each RPB 104 can utilize any number and type of offline models. Each offline model can be a machine learning (ML) model trained on a different input dataset to generate SLI predictions for a region, segment, route, location, or similar division of the mobile communication network.

[0070] The mobility model identifies mobile / driving devices from the mobile communication network data in order to align more relevant data for offline training. The propagation model manages how the mobile communication network (e.g., specific cell towers) covers a geographic area and determines KPI values ​​for more precise geographic locations. In some embodiments, the route selection model is used to split the route into segments, and the route can be optimized and matched with the predicted SLI level from the SLI model. The route selection model is considered part of the online process. Different offline models can use different inputs and training sets. For example, the offline model can contain input information based on weather and time of day, which can be used to contribute to the overall route prediction. The offline model of weather effects takes into account atmospheric effects that cause an average of -3 to -20 dB loss to the signal in the mobile communication network. The offline model that performs best in the offline process is encouraged to make predictions in the online process. Similarly, the current offline model can be downgraded and no longer used by the online process.

[0071] The offline retraining process involves storing the experience data received from the ADF 101. This experience data is then compared with the actual forecast data from the offline model. The deviation between the actual forecast data and the experience data is used to calibrate the offline model. This deviation is then fed back to the offline model as training data or as adjustments to hyperparameters. This enables the offline model to continuously learn from the feedback experience data. The offline model retraining process extracts the ADF 101's past experience from mobile communication network data and uses this information to calibrate the offline model.

[0072] exist Figure 1 In the simplified diagram of FIG, the offline process shows that the ADF 101 provides vehicle experience feedback about the autonomous vehicles that make up the ADF and their requests for predictions. The vehicle experience feedback may include location information, actual QoS information, and similar information. In some embodiments, the vehicle experience information is forwarded from the CP 102 to the RPB 104 or other components.

[0073] In response to the prediction request, as part of the online process, ADF 101 receives prediction information from CP 102. In some embodiments, ADF 101 further provides vehicle experience feedback information to CP 102, either together with or separately from the prediction request. CP 102 aggregates the location information into a latitude-longitude pair (LLP) list, which lists the requested locations for which predictions are requested. The LLP list is sent to RPB 104. RPB 104 generates a set of SLI predictions based on the current propagation model and returns these predictions to CP 102. CP 102 then forwards the experience data to RPB 104. RPB 104 also receives continuous updates of mobile communication network metrics collected from mobile communication network 106 from associated cluster 105.

[0074] The operation of the components in the online and offline processes is further described in conjunction with Figures 2 and 3, which further break down the operation of each of these components in the prediction system.

[0075] These components are independent of each other and can be considered autonomous and replaceable modules. This characteristic of the entire process allows the prediction system to isolate the computationally intensive data association and modeling steps from lightweight application programming interface (API) calls from potential end users (e.g., from autonomous vehicles). In addition, the prediction system offers the advantage of connecting the model of the online process with different offline modules through configuration, depending on the manager's preferences.

[0076] The operations in the flowcharts will be described with reference to the exemplary embodiments of the other figures. However, it should be understood that the operations of the flowcharts may be performed by embodiments of the present invention other than those discussed with reference to the other figures, and that the embodiments of the present invention discussed with reference to these other figures may perform operations different from those discussed with reference to the flowcharts.

[0077] Figure 2A 201 is a simplified diagram of one embodiment of the online process of a prediction system. The illustrated example shows the API call flow from a prediction request from an autonomous vehicle (ADV) 201 to the final response returned to the ADV 201 by the CP 202. The online process uses the best current offline model available to the RPB 203, which is used to score the ADV input data. The process begins with a prediction request 205 from the ADV system 201 to the API deployed on the CP 202 (e.g., using a POST request) with detailed route information, including latitude and longitude pairs (LLPs), and other supporting information. The route and experience information can be published 205 on a per-ADV 201 basis, using individual data points, or aggregated (in batches). The route and experience data is pre-processed and cleaned for further use by the CP 202 206, with the most relevant information being sent to the RPB 207 (e.g., via push route information). If the request from the ADV 201 does not include a pre-defined route, pre-processing can include creating several possible routes. Any process or mechanism may be used at CP 202 to clean or process data, such as discarding corrupted data, removing duplicate data, and the like.

[0078] The RPB 203 can store or access mobile communication network topology and operational data (e.g., cell tower network data and similar operational metrics) and associate this information with the route LLP. For example, the mobile communication network topology and operational data can be associated with the route and experience data request based on the distance of the mobile communication device (e.g., cell tower) from the route and cell tower characteristics. The RPB 203 can further associate the detailed location information (e.g., relevant cell tower or grid) provided for forecast key performance indicator (KPI) value selection with the optimal offline model for each grid. In some embodiments, the offline model is based on a cell coverage area (e.g., centered around a cell tower) or a geographic grid (e.g., centered around a latitude and longitude rectangle). Using a grid approach, when a forecast is available, the relevant route area is broken down into small grids (e.g., 10×10 or 30×30 meters) and experience forecasts are generated for those grids, regardless of cell location, which can be handled in detail through propagation modeling. In other embodiments, a forecast is generated for each cell relative to its coverage area. An approximate probability distribution is generated for the area served by each cell tower. The empirical forecast combines these probability distributions as a weighted average.

[0079] This correlation process is followed by an online scoring process using the best offline model, adjusted with current ADV information, aggregated into an SLI rating prediction based on a dynamic threshold rating, and then correlated with the location of the route segment 208. Finally, the predicted SLI rating for the route segment is sent back to the cloud platform 209, and the CP 202 then sends the SLI rating to the original call sender ADV 210. While the prediction of the SLI rating is ongoing in a short-term batch mode, the KPI value forecast is linked to the route not only by location from a given LLP, but also by time, taking into account expected vehicle speed and distance (e.g., in default mode) or third-party routing API predictions (e.g., in advanced mode).

[0080] Figure 2B is a flow chart of one embodiment of an online process implemented by an RPB. The diagram is described as operating for a single RPB associated with one MNO, however, the process is applicable to embodiments in which multiple RPBs operate in association with multiple MNOs. The RPB process is initiated by receiving a route request originating from an ADV (block 251). The route request may specify a route using an LLP list or similar description of a route. In some embodiments, experience data indicating the actual QoS for a given location and MNO may be provided. The RPB may organize the route information into segments (block 253). The segments of the route may correspond to areas covered by the mobile communications network, or be similarly defined; thus, mobile communications network resources are associated with the route, as well as explicitly associated with specific segments of the route (block 255).

[0081] As determined by the current offline model utilized by the online process, a set of KPI ratings is identified for the segment (block 257). Each offline model may utilize different inputs. For example, some offline models utilize specific weather condition information (e.g., cloud cover or precipitation), while others may not. The current offline model is the offline model selected for use by the online process and outputs a set of KPIs that rate mobile communication network coverage for a segment (e.g., for a specific cell tower covering the segment). If current experience data has been received, this data may be processed and utilized to adjust KPIs for the mobile communication network or specific components of the mobile communication network (block 259). For example, if the ADV reports low QoS for the current segment and the selected mobile communication network, the rating (i.e., KPI) may be downgraded or similarly adjusted to reflect the feedback. The set of output KPIs from the current offline model may be aggregated and correlated with the SLI rating (block 261). The KPIs output by the current offline model may be correlated with the SLI rating using a threshold level or similar mechanism for mapping KPIs to SLI ratings.

[0082] Next, the process may aggregate the SLI levels of the mobile communication network components and resources for each segment of the received route (block 263). This collection of SLI levels or similar QoS ratings for each segment may then be prepared and sent to the requester (or via the cloud platform 202) using any communication method, protocol, or format (block 265).

[0083] Figure 3A is a simplified diagram of one embodiment of an offline process for a prediction system. Figure 3A The example diagram illustrates an offline process for generating per-network resource (e.g., cell tower) KPI level forecasts. The offline process can be initiated with raw network data association 305 in a scalable association cluster (CC) 304. Processed data (e.g., KPI values ​​for relevant cells) is then continuously pushed 306 to a specific RPB 303, where it is accumulated for predictive modeling. Once sufficient data is obtained, a set 307 of offline models (e.g., one or more models for per-cell tower KPI value prediction) is constructed. The construction of the offline models utilizes machine learning techniques. Received network data is accumulated with historical data. The data can be filtered to focus on data relevant to serving mobile devices (i.e., autonomous vehicles, including autonomous vehicles). Each of the offline models can be trained using the accumulated data or a varying subset thereof.

[0084] In one example embodiment, each cell tower time series (TS) data is classified based on predictability and statistical properties. The updated offline models are validated to confirm that they perform appropriately for test inputs or similar validation mechanisms. The updated offline models can be model objects, and these model objects are stored along with their validations for use by the online process, where the online process can select the offline model with the best accuracy, experience data feedback, or similar selection mechanism. Periodically, a push job 308 or similar update mechanism from the RPB is used in the CP to update the table or similar data structure that tracks the offline models and validation results to keep the CP with the latest information for use by the ADV.

[0085] Figure 3B 3 is a flow chart of one embodiment of the operation of an RPB implementing an offline process. The RPB may initiate an iteration of the offline process to update an offline model in response to receiving network data from a CC or similar mobile communication network data source (block 351). The network data may include any number of different metrics for any number of mobile communication network devices, including operational status, throughput, latency, maintenance, resource utilization, and similar information. The received mobile communication network data may be combined with previously received mobile communication network data to form a training data set to train the offline model (block 353). This training data set may be filtered or similarly refined to focus on mobile communication network data relevant to predicting QoS for autonomous vehicles as they traverse a route (block 355).

[0086] After the training data set is prepared, the offline models can be trained against the updated information (box 357). After the offline models are trained and further modified, they can be validated through a testing mechanism to determine the accuracy of each offline model and identify errors (box 361). The offline model with the best performance can be selected for use in the online process and can be referred to as the 'current' offline model during its use by the online process (box 363).

[0087] Figure 4401 is a simplified diagram of an example propagation model. By providing a digital terrain model (DTM), cluster classification, and antenna properties 401, example propagation models are generated for different combinations of time of day and weather. In other embodiments, a digital elevation model (DEM) or digital surface model (DSM) may be utilized instead of or in addition to the DTM. In the example shown, the propagation model consists of 16 different models generated for different time-weather combinations 402. The propagation model can output information such as KPIs including received signal received power (RSRP) and / or received signal received quality (RSRQ) 403. Multiple propagation models can be maintained, and the properties of each model can vary depending on the MNO and similar considerations. The predictions of the propagation model are fed to the RPB (e.g., autonomous vehicle route controller (AVR)) 404.

[0088] Figure 5 5 is a flow chart of an example embodiment of a process for filtering network data to be used by an offline process. This filtering process can be implemented using a focused mobility model in the offline process. This example filtering process is provided by way of example and not limitation, and those skilled in the art will appreciate that other similar filtering processes can be utilized in conjunction with online and offline processes. This process provides classification of the data to filter out data that is not relevant for training the offline model. The filtering process filters the network data to identify network data associated with high mobility. Input data is obtained from a CC (block 501) and can be filtered sequentially to obtain desirable data for the offline process. In this example, the first filter verifies whether any handovers have occurred within a short-duration flow (block 502). If the data is not related to a handover, the data can be discarded or ignored. For data related to a handover, the function checks whether the distance between mobile communication network cells is not walkable (i.e., long distance) for a given period (block 503). If the data is related to walkable or short distances, the data can be discarded or ignored. For data related to longer distances (i.e., non-walkable distances), the data is analyzed to determine if it is geospatially correlated with nearby roads (block 504) to confirm that the data is relevant to movement along public roads that the autonomous vehicle may navigate. If the data is not correlated with navigable roads, it may be discarded or ignored. Data related to navigable roads may be stored for use in an offline process (block 505). This filtering functionality is provided by way of example and not limitation. The filtering of the data may utilize any number or type of classifications, thereby increasing the relevance of the data to the route prediction process of the prediction system.

[0089] Figure 66 is a flowchart of one embodiment of a process for a mapping function that correlates mobile communication network data with a route. This example mapping function is provided by way of example and not limitation, and those skilled in the art will appreciate that other similar mapping functions may be utilized in conjunction with online and offline processes. In the illustrated example, the mapping function identifies relevant mobile communication network components (e.g., cell towers) for an input route. The process may be initiated via an API call from a CP that provides a list of location-based positions (LLPs) for the route (block 601). The mapping function also utilizes mobile communication network component LLP information (e.g., cell tower location information) from a reference dataset as input (block 602). These datasets are merged for further processing (block 603).

[0090] Two sets of associations are involved in the mapping process, namely, whether the mobile communication network components (e.g., cells) are geographically located close enough to the input route (block 604), and whether the mobile communication network components (e.g., cells) have physically strong signal coverage on the input route (block 605), taking into account cell characteristics such as frequency band, tilt, probability of being selected in that area, and similar considerations. If the mobile communication network components are not geographically relevant or do not provide route coverage, the data associated with these components can be ignored or discarded. Based on this mapping, only relevant mobile communication network components (e.g., tower cells) are output for further data filtering or processing in an offline process (block 606).

[0091] Figure 7 7 is a flow chart of one embodiment of an offline SLI modeling implementation. The illustrated process illustrates an offline process application, wherein the best model (i.e., the winning forecast model) is selected for application in an online process. This example forecast model training is provided as an example and not a limitation, and one skilled in the art will appreciate that other similar modeling functions may be utilized in conjunction with online and offline processes. The process is initiated using input mobile communication network data obtained from a CC or similar source (block 701). The input data is filtered so as to be relevant to the route selection of an autonomous vehicle (e.g., as described with respect to the filtering functions described herein) and classified into different time series (TS) clusters based on sequence statistics (block 702). Offline models corresponding to the most likely categories are trained based on the TS data (block 703), which can then be analyzed using cross-validation techniques (block 704). All offline models, validation metrics, and best model objects are stored (block 705) for further use in the online process.

[0092] Figure 8801 is a flowchart of an example embodiment of a feedback function for an online SLI scoring implementation. This example feedback function is provided as an example and not a limitation, and those skilled in the art will appreciate that other similar feedback functions can be utilized in conjunction with online and offline processes. As described herein, the online scoring process utilizes an offline model identified by the offline process to generate route prediction information. The feedback function utilizes experience data reported from autonomous vehicles to determine the accuracy of the predictions made by the prediction service. In this example embodiment, the feedback function of the online scoring process can be initiated in response to receiving data obtained from the ADF (block 801) or a similar mechanism (e.g., via an API POST call). The feedback function accesses the currently selected offline model as identified by the offline process (block 802). The API POST call is checked to see if it contains experience data (block 803). If the API POST call does not contain experience data or similar information about the current network performance and mobility experience, the minimum required data will be scored using the current offline model and the process is completed. In some embodiments, the experience data received from the API POST call is used along with additional indicators of current performance (block 804) by correlating the current experience data (i.e., indirect KPI) with the predicted primary KPI value (block 805). If the predicted value is misaligned with the current performance, the forecast is adjusted by the misalignment factor (block 806).

[0093] Figure 9is a flow chart of an example embodiment of an offline retraining process. This example retraining function is provided by way of example and not limitation, and those skilled in the art will appreciate that other similar retraining functions may be utilized in conjunction with online and offline processes. The retraining function enriches the offline modeling with relevant validation responses. The retraining process may be initiated in response to receiving current driving experience data from the ADF (e.g., via an API call from the CP) (block 901). The experience data is stored and accessible by the RPB for backtesting and historical comparison with the dynamics of the master KPI (block 902). The received experience data indicates the actual QoS experience, which is compared with the predicted QoS from the offline model. The QoS experience data and the predicted QoS data may be represented as SLI levels, KPIs, and similar metrics on a per-segment, per-route, per-location, per-mobile communication network component, or similar basis. Based on the comparison, the deviation between the actual value and the predicted indirect indicator (e.g., SLI level and KPI value) is calculated (block 903). The bias is checked to determine whether a large bias exists (block 904), where if the bias is large, the observation receives an increased weight based on the bias value (block 905), and all historical data with adjusted observation weights can be rebuilt at regular intervals (e.g., daily) by a corresponding offline process (block 906). If no large bias is detected, the weighting may not be adjusted, and the offline process can rebuild the offline model at regular intervals (e.g., daily).

[0094] Figure 10 is a simplified diagram of one embodiment of a system for brokering information between multiple MNO sources. The brokering system shown illustrates a high-level view of the data sources and the brokering process. Figure 11-13 Further describe the functionality of the agent process. In particular, Figure 11-13 The described functionality shows how the forecasting system and services can be distributed in order to protect each MNO's proprietary information and thereby enable brokering of required information that the MNOs want to share with the forecasting system and services.

[0095] exist Figure 10In the example, in response to a query to the prediction system for a specified route, the ADF 1001 receives prediction information (e.g., predicted QoS information for a segment) 1005. As a result of the prediction system process, a monetary billing flow exists between the requesting ADF 1001 and the prediction system and prediction service. Prediction information can be provided via the CP 1002 using multiple data sources. The primary data source (e.g., mobile network metric information) can be the MNO, which provides real-time data and helps maintain the ecosystem for the RPBs 1004 associated with or associated with the respective MNO. The RPBs 1004 can be services provided by the prediction system and managed by the operator of the prediction service. This creates a monetary billing flow between the CP's prediction system and services and the RPBs of the respective MNO, based on the number of requests for mobile network data sent to the respective MNO. Other third-party data sources 1003 may also exist, providing weather, traffic, optimal routes, DTM, and similar information to the prediction system and services. This relationship creates a monetary billing flow between the prediction system and services and each third-party source, depending on the number of requests for the respective third-party services. In one example embodiment, the forecasting system and services are implemented on an open cloud platform 1002 or similar platform, which creates a monetary billing flow between the forecasting system and services and the CP provider. This set of monetary billing relationships is handled by a set of proxy functions that handle the billing of expenses between the interrelated components and systems.

[0096] Figure 11 1105 . The ADF database 1105 may contain any number of records 1107 that provide parameters related to each ADF, such as country of operation, KPI to SLI level mapping, KPI thresholds, subscription information, and similar parameters.

[0097] Figure 121203. FIG1204 is a flowchart of an embodiment of an online process implemented at a CP to support an information agent. The process processes prediction requests at the CP for prediction systems and services. This example prediction request handling functionality is provided as an example and not a limitation, and those skilled in the art will appreciate that other similar prediction request handling functionality may be utilized in conjunction with the online scoring process. The prediction request handling process may be initiated in response to receiving a 'push' prediction request or similar prediction request from an autonomous vehicle (e.g., ADV) associated with an ADF (box 1201). A check is performed to determine whether the source of the route is to be determined by a third-party data source or provided by the autonomous vehicle (box 1203). In the event that the route is to be derived from a third-party or other data source (e.g., the autonomous vehicle only provides a destination and starting location that require routing at the CP or an external service), the route is determined using the CP service or external service based on the available data from the request (box 1205).

[0098] In the event that the autonomous vehicle provides a route or after the route is determined, the route is broken into segments (block 1207). The segments can be identified using any process or function, including correlation with mobile communication network component locations, coverage maps, or similar techniques. Parameters stored in the ADF database can be used to determine the available MNOs (e.g., local mobile communication networks) for the country of origin of the autonomous vehicle, and in some cases, to determine the preferred provider (block 1209). For example, the ADF database can be queried for the MNO utilized by a given ADF or autonomous vehicle in a given country. The route information, including the segment identification, can then be forwarded ('pushed') to each of the RPBs of the corresponding MNO identified based on the ordered list of MNOs (block 1211).

[0099] Figure 131301 is a flowchart of an example embodiment of a process for processing route information for prediction systems and services at a CP to support information brokers. This example push route functionality is provided by way of example and not limitation, and those skilled in the art will appreciate that other similar push route functionality can be utilized in conjunction with the online scoring process. Upon receiving a prediction request, the push route process can be initiated by determining whether MNO binding has occurred (i.e., whether the ADV uses MNO binding) (block 1301). An ADV using MNO binding will access multiple MNOs simultaneously, while an ADV not using MNO binding will use a single MNO at a time, though it may switch between MNOs depending on the prediction provided by the prediction system. If MNO binding does not exist, the push route process sends the route information to the RPB of the first MNO in the ordered list (block 1303). All segments are checked to see if they have a first KPI value above a specified threshold (e.g., as defined in the ADF parameters) (block 1305). If none of the segments have a first KPI value above the threshold, the route information is sent to the next MNO in the ordered list (block 1307). Similar checks are performed on additional KPI values ​​(blocks 1309 - 1311 ) until all KPI values ​​are found to be above the respective thresholds or the routing information is sent to the RPB of the next MNO in the ordered list.

[0100] If all segments of the route have KPI values ​​above the corresponding threshold, the process combines the predicted SLI for the route from the RPBs that received the route information (block 1317). The combined predicted SLI is then sent ('pushed') to the requesting ADV (block 1319). Similarly, if an MNO binding exists (block 1301), the route information is pushed to the RPBs of each MNO in the ordered list (block 1315). The process then combines the predicted SLI for the route from the RPBs that received the route information (block 1317). The combined predicted SLI is then sent ('pushed') to the requesting ADV (block 1319).

[0101] Figure 14 is a simplified diagram of one embodiment of a cloud implementation of prediction systems and services. Figure 14The flow of online scoring processing in a cloud implementation is described. These embodiments utilize a cloud platform to receive and send data from ADVs 1401, 1405. These embodiments are containerized and partially located on the cloud platform, making it easy to scale and resulting in a distributed system. In addition, other components of the prediction system and services can run in a distributed manner on RPBs 1402, 1403 (including all offline modeling, online and offline processes, and similar components). The entire end-to-end flow can use the rest API, which results in a distributed system that is scalable not only vertically but also horizontally. Data streams from ADVs 1401 are sent to RPBs 1402 using the API, and then these data streams are processed, predictions are added, and sent back from the RPBs 1403. Because feedback is handled in the loop, predictions are received at the ADVs, which in turn provide experience data. The experience data is used to grade the predictions 1404, which are sent to the corresponding ADVs 1405, closing the data loop.

[0102] Figure 15A Connectivity between network devices (NDs) within an exemplary network and three exemplary implementations of NDs are shown according to some embodiments of the present invention. Figure 15A NDs 1500A-H are shown, and their connectivity is shown by lines between 1500A-1500B, 1500B-1500C, 1500C-1500D, 1500D-1500E, 1500E-1500F, 1500F-1500G, and 1500A-1500G, as well as between 1500H and each of 1500A, 1500C, 1500D, and 1500G. These NDs are physical devices, and the connectivity between these NDs can be wireless or wired (commonly referred to as links). Additional lines extending from NDs 1500A, 1500E, and 1500F illustrate that these NDs serve as entry and exit points for the network (and therefore, these NDs are sometimes referred to as edge NDs; while other NDs may be referred to as core NDs).

[0103] Figure 15A Two of the exemplary ND implementations in are: 1) a dedicated network device 1502, which uses a custom application-specific integrated circuit (ASIC) and a dedicated operating system (OS); and 2) a general-purpose network device 1504, which uses a common off-the-shelf (COTS) processor and a standard OS.

[0104] The dedicated network device 1502 includes networking hardware 1510, which includes a set of one or more processors 1512, forwarding resource(s) 1514 (typically including one or more ASICs and / or network processors), a plurality of physical network interfaces (NIs) 1516 (through which network connections are made, such as those shown by the connectivity between NDs 1500A-H), and a non-transitory machine-readable storage medium 1518 having stored therein networking software 1520. During operation, the networking software 1520 can be executed by the networking hardware 1510 to instantiate a set of one or more networking software instances 1522. Each of the one or more networking software instances 1522 and the portion of the networking hardware 1510 that executes the networking software instance (whether it is hardware dedicated to the networking software instance and / or a time slice of hardware temporarily shared by the networking software instance and other instances of the one or more networking software instances 1522) form a separate virtual network element 1530A-R. Each of the (one or more) virtual network elements (VNEs) 1530A-R includes a control communication and configuration module 1532A-R (sometimes referred to as a local control module or control communication module) and (one or more) forwarding tables 1534A-R, such that a given virtual network element (e.g., 1530A) includes a control communication and configuration module (e.g., 1532A), a collection of one or more forwarding tables (e.g., 1534A), and the portion of the networking hardware 1510 that executes the virtual network element (e.g., 1530A).

[0105] The non-transitory machine-readable storage medium 1518 may also store therein a prediction service 1565. The prediction service 1565 may include any number or combination of functions related to the prediction system described herein. The prediction service 1565 may be distributed across multiple dedicated network devices 1502 and other devices.

[0106] The dedicated network device 1502 is typically considered to be physically and / or logically comprised of: 1) an ND control plane 1524 (sometimes referred to as a control plane), comprising (one or more) processors 1512 that execute (one or more) control communications and configuration modules 1532A-R; and 2) an ND forwarding plane 1526 (sometimes referred to as a forwarding plane, data plane, or media plane), comprising (one or more) forwarding resources 1514 and physical NI 1516 that utilize (one or more) forwarding tables 1534A-R. For example, where the ND is a router (or implements routing functionality), the ND control plane 1524 (processor(s) 1512 executing control communications and configuration modules(s) 1532A-R) is typically responsible for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for the data) and storing that routing information in forwarding table(s) 1534A-R, while the ND forwarding plane 1526 is responsible for receiving the data on the physical NI 1516 and forwarding the data outward to the appropriate NI in the physical NI 1516 based on the forwarding table(s) 1534A-R.

[0107] Figure 15B An exemplary manner of implementing dedicated network device 1502 according to some embodiments of the present invention is shown. Figure 15B A dedicated network device is shown including a (typically hot-swappable) card 1538. While in some embodiments the cards 1538 are of two types (one or more cards operating as the ND forwarding plane 1526 (sometimes referred to as line cards), and one or more cards operating to implement the ND control plane 1524 (sometimes referred to as control cards)), alternative embodiments may combine functionality onto a single card and / or include additional card types (e.g., an additional type of card may be referred to as a service card, resource card, or multi-application card). The service cards may provide specialized processing (e.g., Layer 4 to Layer 7 services (e.g., firewall, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection System (IDS), Peer-to-Peer (P2P), Voice over IP (VoIP) Session Border Controller, Mobile Wireless Gateway (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) Gateway)). For example, a service card may be used to terminate an IPsec tunnel and perform the accompanying authentication and encryption algorithms. The cards are coupled together by one or more interconnect mechanisms, shown as backplane 1536 (e.g., a first fully meshed interconnect couples the line cards, while a second fully meshed interconnect couples all the cards).

[0108] Return to Figure 15A, the general network device 1504 includes hardware 1540, which includes a set 1542 of one or more processors (typically COTS processors) and a physical NI 1546, and a non-transitory machine-readable storage medium 1548 having stored therein software 1550. During operation, the processor(s) 1542 execute the software 1550 to instantiate one or more sets 1564A-R of one or more applications. Although one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment, the virtualization layer 1554 represents a kernel of an operating system (or a shim executing on top of a base operating system) that allows for the creation of multiple instances 1562A-R of software containers, each of which can be used to execute one (or more) of a set of applications 1564A-R; wherein the multiple software containers (also known as virtualization engines, virtual private servers, or jails) are user spaces (typically virtual memory spaces) that are separate from each other and from the kernel space in which the operating system runs; and wherein the set of applications running in a given user space cannot access the memory of other processes unless explicitly allowed to do so. In another such alternative embodiment, the virtualization layer 1554 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a management program that executes on top of a host operating system, and each of the sets of applications 1564A-R runs on top of a guest operating system within an instance 1562A-R called a virtual machine (which in some cases can be considered a strictly isolated form of software container), which runs on top of the hypervisor - as opposed to running on a "bare metal" host electronic device, the guest operating system and applications may not be aware that they are running on a virtual machine, or by overriding the virtualization, the operating system and / or applications may be aware of the presence of virtualization for optimization purposes. In still other alternative embodiments, one, some, or all of the applications are implemented as a monolithic kernel(s), which can be generated by directly employing the application to compile only a limited set of libraries that provide the specific OS services required by the application (e.g., from a library operating system (LibOS) that includes drivers / libraries of OS services). Because the single kernel can be implemented to run directly on the hardware 1540, directly on the hypervisor (in which case the single kernel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented entirely as a single kernel running directly on the hypervisor represented by the virtualization layer 1554, as a single kernel running within a software container represented by instances 1562A-R, or as a combination of a single kernel and the above techniques (e.g., both the single kernel and the virtual machine running directly on the hypervisor, a collection of applications and the single kernel running in different software containers).

[0109] The non-transitory machine-readable storage medium 1548 may also have stored therein a prediction service 1565. The prediction service 1565 may include any number or combination of functions related to the prediction system described herein. The prediction service 1565 may be distributed across multiple general-purpose network devices 1504 and other devices. Similarly, the prediction service 1565 may be implemented in any number of general-purpose electronic devices, such as in a cloud computing environment that communicates with a mobile communication network.

[0110] The instantiation of one or more sets 1564A-R of one or more applications, and the virtualization layer (if implemented), are collectively referred to as software instance(s) 1552. Each set 1564A-R of applications, the corresponding virtualization constructs (e.g., instances 1562A-R) (if implemented), and the portion of the hardware 1540 on which they execute (whether dedicated to that execution and / or a time slice of temporarily shared hardware) form separate virtual network element(s) 1560A-R.

[0111] Virtual network element(s) 1560A-R perform similar functionality as virtual network element(s) 1530A-R—for example, similar to control communication and configuration module(s) 1532A and forwarding table(s) 1534A (this virtualization of hardware 1540 is sometimes referred to as network function virtualization (NFV)). Thus, NFV can be used to consolidate many network device types onto industry-standard, high-volume server hardware, physical switches, and physical storage devices (which can be located in data centers, NDs, and customer premises equipment (CPE)). While embodiments of the present invention are illustrated with each instance 1562A-R corresponding to one VNE 1560A-R, alternative embodiments may implement this correspondence at a finer level of granularity (e.g., line card virtual machines virtualizing line cards, control card virtual machines virtualizing control cards, etc.); it should be understood that the techniques described herein with reference to the correspondence of instances 1562A-R to VNEs also apply to embodiments using such finer levels of granularity and / or a single core.

[0112] In some embodiments, the virtualization layer 1554 includes a virtual switch that provides forwarding services similar to a physical Ethernet switch. Specifically, this virtual switch forwards traffic between instances 1562A-R and (one or more) physical NIs 1546, and optionally between instances 1562A-R. In addition, this virtual switch can enforce network isolation between VNEs 1560A-R that are not allowed to communicate with each other by policy (e.g., by implementing virtual local area networks (VLANs)).

[0113] Figure 15AA third exemplary ND implementation in

[00155] is hybrid network device 1506, which includes both a custom ASIC / dedicated OS and a COTS processor / standard OS in a single ND or a single card within an ND. In certain embodiments of such a hybrid network device, a platform VM (i.e., a VM that implements the functionality of dedicated network device 1502) can provide over-the-air virtualization of the networking hardware present in hybrid network device 1506.

[0114] Regardless of the above exemplary implementations of the ND, when considering a single VNE among multiple VNEs implemented by the ND (e.g., only one of the VNEs is part of a given virtual network), or in the case where the ND currently implements only a single VNE, the short term network element (NE) is sometimes used to refer to the VNE. Moreover, in all implementations of the above exemplary implementations, each of the VNEs (e.g., (one or more) VNEs 1530A-R, VNEs 1560A-R, and those in hybrid network device 1506) receives data on a physical NI (e.g., 1516, 1546) and forwards the data outward to the appropriate NI (e.g., 1516, 1546) among the physical NIs. For example, a VNE that implements IP router functionality forwards IP packets based on some IP header information in the IP packets; wherein the IP header information includes source IP address, destination IP address, source port, destination port (wherein, "source port" and "destination port" in this document refer to protocol ports, not physical ports of ND), transport protocol (e.g., User Datagram Protocol (UDP), Transmission Control Protocol (TCP)), and Differentiated Services Code Point (DSCP) value.

[0115] Figure 15C Various exemplary ways in which a VNE may be coupled according to some embodiments of the present invention are shown. Figure 15C Shown are VNEs 1570A.1-1570A.P (and optional VNEs 1570A.Q-1570A.R) implemented in ND 1500A and VNE 1570H.1 in ND 1500H. Figure 15C, VNEs 1570A.1-P are separate from one another in the sense that they can receive packets from outside ND 1500A and forward packets outside ND 1500A; VNE 1570A.1 is coupled to VNE 1570H.1, and thus they pass packets between their respective NDs; VNEs 1570A.2-1570A.3 can optionally forward packets between themselves without forwarding them outside ND 1500A; and VNE 1570A.P can optionally be the first in a chain of VNEs containing VNE 1570A.Q followed by 1570A.R (this is sometimes called dynamic service chaining, where each VNE in the series of VNEs provides a different service—e.g., one or more Layer 4-7 network services). Although Figure 15C Various exemplary relationships between VNEs are shown, but alternative embodiments may support other relationships (eg, more / fewer VNEs, more / fewer dynamic service chains, multiple different dynamic service chains with some common VNEs and some different VNEs).

[0116] For example, Figure 15A The ND may form part of the Internet or a private network; and other electronic devices (not shown; such as end-user devices, including workstations, laptops, netbooks, tablets, handhelds, mobile phones, smartphones, tablet phones, multimedia phones, voice over Internet Protocol (VOIP) phones, terminals, portable media players, GPS units, wearable devices, gaming systems, set-top boxes, Internet-enabled home appliances) may be coupled to the network (directly or through other networks such as access networks) to communicate with each other (directly or through servers) and / or access content and / or services over the network (e.g., the Internet or a virtual private network (VPN) laid over the Internet (e.g., tunneled through the Internet)). Such content and / or services are typically provided by one or more servers (not shown) belonging to a service / content provider or one or more end-user devices (not shown) participating in a peer-to-peer (P2P) service, and may include, for example, public web pages (e.g., free content, storefronts, search services), private web pages (e.g., web pages providing username / password access to email services), and / or corporate networks on the VPN. For example, end-user devices may be coupled (e.g., via client devices coupled (wired or wirelessly) to an access network) to edge NDs, which are coupled (e.g., via one or more core NDs) to other edge NDs, which are coupled to electronic devices acting as servers. However, through compute and storage virtualization, Figure 15AOne or more of the electronic devices operating as NDs may also host one or more such servers (for example, in the case of a general-purpose network device 1504, one or more of the software instances 1562A-R may operate as servers; this would also be true for a hybrid network device 1506; in the case of a dedicated network device 1502, one or more such servers may also run on a virtualization layer executed by (one or more) processors 1512); in this case, the server is said to be co-located with the VNE of the ND.

[0117] A virtual network is a physical network (such as a Figure 15A A virtual network can be implemented as an overlay network (sometimes called a network virtualization overlay) that provides network services (e.g., Layer 2 (L2, data link layer) and / or Layer 3 (L3, network layer) services) on an underlay network (e.g., an L3 network such as an Internet Protocol (IP) network that uses tunnels (e.g., Generic Routing Encapsulation (GRE), Layer 2 Tunneling Protocol (L2TP), IPsec) to create an overlay network).

[0118] A network virtualization edge (NVE) sits at the edge of the underlying network and participates in implementing network virtualization. The network-facing side of an NVE uses the underlying network to tunnel frames to and from other NVEs. The external-facing side of an NVE sends data to and receives data from systems outside the network. A virtual network instance (VNI) is a specific instance of a virtual network on an NVE (e.g., an NE / VNE on a ND, or a portion of an NE / VNE on an ND that is partitioned into multiple VNEs through simulation). One or more VNIs can be instantiated on an NVE (e.g., as different VNEs on the ND). A virtual access point (VAP) is a logical connection point on an NVE used to connect external systems to a virtual network. A VAP can be a physical or virtual port identified by a logical interface identifier (e.g., a VLAN ID).

[0119] Examples of network services include: 1) Ethernet LAN emulation service (an Ethernet-based multipoint service similar to Internet Engineering Task Force (IETF) Multiprotocol Label Switching (MPLS) or Ethernet VPN (EVPN) service), where external systems are interconnected across the underlying network via a LAN environment (e.g., the NVE provides separate L2 VNIs (virtual switching instances) for different such virtual networks and provides L3 (e.g., IP / MPLS) tunneling encapsulation across the underlying network); and 2) virtualized IP forwarding service (similar to IETF IP VPN (e.g., Border Gateway Protocol (BGP) / MPLS IP VPN) from a service definition perspective), where external systems are interconnected across the network via an L3 environment (e.g., the NVE provides separate L3 VNIs (forwarding and routing instances) for different such virtual networks and provides L3 (e.g., IP / MPLS) tunneling encapsulation across the underlying network). Network services may also include quality of service capabilities (e.g., traffic classification marking, traffic conditioning, and scheduling), security capabilities (e.g., filters to protect customer premises from network-originated attacks to prevent malformed route advertisements), and management capabilities (e.g., full detection and handling).

[0120] Figure 15D According to some embodiments of the present invention, Figure 15A A network with a single network element on each of the NDs is constructed, and within this straightforward approach, the traditional distributed approach (commonly used with traditional routers) is contrasted with the centralized approach (also known as network control) for maintaining reachability and forwarding information. Specifically, Figure 15D Shown with Figure 15A The ND 1500A-H has the same connectivity as the Network Element (NE) 1570A-H.

[0121] Figure 15D The distributed approach 1572 is shown distributing the responsibility for generating reachability and forwarding information across the NEs 1570A-H; in other words, the process of neighbor discovery and topology discovery is distributed.

[0122] For example, where a dedicated network device 1502 is used, the control communication and configuration module(s) 1532A-R of the ND control plane 1524 typically include reachability and forwarding information modules for implementing one or more routing protocols (e.g., exterior gateway protocols such as Border Gateway Protocol (BGP), interior gateway protocol(s) (e.g., Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), Routing Information Protocol (RIP), Label Distribution Protocol (LDP), Resource Reservation Protocol (RSVP) (including RSVP-Traffic Engineering (TE): an extension of RSVP for LSP tunneling and Generalized Multiprotocol Label Switching (GMPLS) that signals RSVP-TE)) that communicate with other NEs to exchange routes and then select those routes based on one or more routing metrics. Thus, the NEs 1570A-H (e.g., processor(s) 1512 executing control communication and configuration module(s) 1532A-R) performs its responsibilities of participating in controlling how data (e.g., packets) will be routed (e.g., the next hop for the data and the outgoing physical NI for the data) by distributively determining reachability within the network and computing their corresponding forwarding information. Routes and adjacencies are stored in one or more routing structures (e.g., Routing Information Base (RIB), Label Information Base (LIB), one or more adjacency structures) on the ND control plane 1524. The ND control plane 1524 performs its responsibilities of participating in controlling how data (e.g., packets) will be routed (e.g., the next hop for the data and the outgoing physical NI for the data) by distributing the ... The ND control plane 1524 compiles information (e.g., adjacency and routing information) into one or more forwarding tables 1534A-R (e.g., a forwarding information base (FIB), a label forwarding information base (LFIB), and one or more adjacency structures) on the ND forwarding plane 1526. For Layer 2 forwarding, the ND may store one or more bridging tables for forwarding data based on the Layer 2 information in the data. While the above example uses a dedicated network device 1502, the same distributed method 1572 can be implemented on a general-purpose network device 1504 and a hybrid network device 1506.

[0123] Figure 15DA centralized approach 1574 (also known as software-defined networking (SDN)) is shown that decouples the system that makes decisions about where to send traffic from the underlying system that forwards the traffic to the selected destination. The centralized approach 1574 shown is responsible for generating reachability and forwarding information in a centralized control plane 1576 (sometimes referred to as an SDN control module, controller, network controller, OpenFlow controller, SDN controller, control plane node, network virtualization authority, or management control entity), so that the neighbor discovery and topology discovery processes are centralized. The centralized control plane 1576 has a southbound interface 1582 with the data plane 1580 (sometimes referred to as the infrastructure layer, network forwarding plane, or forwarding plane (not to be confused with the ND forwarding plane)), which includes NEs 1570A-H (sometimes referred to as switches, forwarding elements, data plane elements, or nodes). The centralized control plane 1576 includes a network controller 1578, which includes a centralized reachability and forwarding information module 1579 that determines reachability within the network and distributes forwarding information to the NEs 1570A-H of the data plane 1580 via southbound interfaces 1582 (which may use the OpenFlow protocol). Thus, network intelligence is centralized in the centralized control plane 1576, which executes on electronic devices that are typically separate from the ND.

[0124] For example, where a dedicated network device 1502 is used in the data plane 1580 , each of the control communication and configuration module(s) 1532A-R of the ND control plane 1524 typically includes a VNE-side control agent that provides a southbound interface 1582 . In this case, the ND control plane 1524 (processor(s) 1512 executing control communication and configuration module(s) 1532A-R) performs its responsibilities for participating in controlling how data (e.g., packets) will be routed (e.g., the next hop for the data and the outgoing physical NI for the data) through a control agent that communicates with the centralized control plane 1576 to receive forwarding information (and in some cases, reachability information) from the centralized reachability and forwarding information module 1579 (it should be understood that in some embodiments of the present invention, control communication and configuration module(s) 1532A-R, in addition to communicating with the centralized control plane 1576, may also play a role in determining reachability and / or calculating forwarding information - although less so than in the case of a distributed approach; such embodiments are generally considered to belong to the centralized approach 1574, but may also be considered a hybrid approach).

[0125] Although the above example uses a dedicated network device 1502, the same centralized approach 1574 can be implemented utilizing a general-purpose network device 1504 (e.g., each VNE in VNE 1560A-R performs its responsibility for controlling how data (e.g., packets) will be routed (e.g., the next hop for the data and the outgoing physical NI for the data) by communicating with a centralized control plane 1576 to receive forwarding information (and in some cases, reachability information) from a centralized reachability and forwarding information module 1579; it should be understood that in some embodiments of the present invention, VNE 1560A-R, in addition to communicating with the centralized control plane 1576, may also play a role in determining reachability and / or calculating forwarding information - although less than in the case of a distributed approach) and a hybrid network device 1506. In fact, the use of SDN technology can enhance the NFV technology typically used in general-purpose network device 1504 or hybrid network device 1506 implementations because NFV can support SDN by providing an infrastructure on which SDN software can run, and both NFV and SDN are designed to utilize commodity server hardware and physical switches.

[0126] Figure 15D The centralized control plane 1576 is also shown as having a northbound interface 1584 to the application layer 1586, where application(s) 1588 reside. The centralized control plane 1576 is capable of forming a virtual network 1592 (sometimes referred to as a logical forwarding plane, network services, or overlay network (where the NEs 1570A-H of the data plane 1580 are the underlying network)) for the application(s) 1588. Thus, the centralized control plane 1576 maintains a global view of all NDs and configured NEs / VNEs, and it efficiently maps virtual networks to the underlying NDs (including maintaining these mappings as the physical network changes due to failures, additions, or removals of hardware (NDs, links, or ND components)).

[0127] Prediction service 1581 may include any number or combination of functions related to the prediction system described herein, implemented at application layer 1586 .

[0128] Although Figure 15DThe distributed approach 1572 is shown separate from the centralized approach 1574, but in certain embodiments of the present invention, the work of network control may be distributed in different ways, or a combination of the two. For example: 1) an embodiment may generally use a centralized approach (SDN) 1574, but delegate certain functions to the NE (e.g., a distributed approach may be used to implement one or more of fault monitoring, performance monitoring, protection switching, and primitives for neighbor and / or topology discovery); or 2) an embodiment of the present invention may perform neighbor discovery and topology discovery via both a centralized control plane and a distributed protocol, and compare the results to raise an exception if they disagree. Such embodiments are generally considered to belong to the centralized approach 1574, but may also be considered a hybrid approach.

[0129] Although Figure 15D The simplified case is shown where each of the NDs 1500A-H implements a single NE 1570A-H, but it should be understood that with reference to Figure 15D The described network control methods are also applicable to networks in which one or more of the NDs 1500A-H implement multiple VNEs (e.g., VNEs 1530A-R, VNEs 1560A-R, those in hybrid network device 1506). Alternatively or additionally, the network controller 1578 may also emulate the implementation of multiple VNEs in a single ND. Specifically, instead of (or in addition to) implementing multiple VNEs in a single ND, the network controller 1578 may present the implementation of the VNE / NE in a single ND as multiple VNEs in virtual network 1592 (all in the same virtual network in virtual network(s) 1592, each in a different virtual network in virtual network(s) 1592, or some combination). For example, the network controller 1578 may cause the ND to implement a single VNE (NE) in the underlying network and then logically partition the resources of that NE within the centralized control plane 1576 to present different VNEs in (one or more) virtual networks 1592 (wherein these different VNEs in the overlay network share the resources of the single VNE / NE implementation on the ND in the underlying network).

[0130] on the other hand, Figure 15E and Figure 15F Exemplary abstractions of NEs and VNEs are shown, respectively, which the network controller 1578 can present as part of different virtual networks in the virtual network 1592. Figure 15E Shown where each ND 1500A-H implements a single NE 1570A-H (see Figure 15D ), but according to some embodiments of the present invention, the centralized control plane 1576 has abstracted multiple NEs (NEs 1570A-C and GH) in different NDs into (represented by) Figure 15D A single NE 1570I in one of the (one or more) virtual networks 1592. Figure 15E It is shown that in this virtual network, NE 1570I is coupled to NE 1570D and 1570F, while NE 1570D and 1570F are both still coupled to NE 1570E.

[0131] Figure 15F The following scenario is shown according to some embodiments of the present invention, wherein multiple VNEs (VNE 1570A.1 and VNE 1570H.1) are implemented on different NDs (ND 1500A and ND 1500H) and are coupled to each other, and wherein a centralized control plane 1576 has abstracted these multiple VNEs so that they are Figure 15D 1570T within one of the virtual networks 1592. Thus, the abstraction of a NE or VNE can span multiple NDs.

[0132] While some embodiments of the present invention implement the centralized control plane 1576 as a single entity (e.g., a single instance of software running on a single electronic device), alternative embodiments may distribute this functionality across multiple entities for redundancy and / or scalability purposes (e.g., multiple instances of the software running on different electronic devices).

[0133] Similar to network device implementations, the electronic device(s) running the centralized control plane 1576 and, therefore, the network controller 1578 including the centralized reachability and forwarding information module 1579 may be implemented in a variety of ways (e.g., dedicated devices, general-purpose (e.g., COTS) devices, or hybrid devices). These electronic device(s) will similarly include a processor(s), a set of one or more physical NIs, and a non-transitory machine-readable storage medium having the centralized control plane software stored thereon. For example, Figure 16 A general control plane apparatus 1604 is shown comprising hardware 1640 including a set 1642 of one or more processors (typically COTS processors) and physical NI 1646 , and a non-transitory machine-readable storage medium 1648 having centralized control plane (CCP) software 1650 stored therein.

[0134] The non-transitory machine-readable storage medium 1648 may also store therein a prediction service 1681. The prediction service 1681 may include any number or combination of functions related to the prediction system described herein. The prediction service 1681 may be distributed across multiple general control plane devices 1604 and other devices.

[0135] In embodiments using computing virtualization, the processor(s) 1642 typically execute software to instantiate a virtualization layer 1654 (e.g., in one embodiment, the virtualization layer 1654 represents an operating system kernel (or shim executing on a base operating system) that allows for the creation of multiple instances 1662A-R called software containers (representing separate user spaces and also referred to as virtualization engines, virtual private servers, or jails), each of which can be used to execute a collection of one or more applications; in another embodiment, the virtualization layer 1654 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a management program that executes on top of a host operating system, and the applications are executed on the servers run by the hypervisor). In another embodiment, the application is implemented as a monolithic kernel, which can be generated by directly compiling the application with only a limited set of libraries that provide the specific OS services required by the application (e.g., from a library operating system (LibOS) that includes drivers / libraries for OS services), and the monolithic kernel can run directly on the hardware 1640, directly on the hypervisor represented by the virtualization layer 1654 (in which case the monolithic kernel is sometimes described as running within a LibOS virtual machine), or in a software container represented by one of the instances 1662A-R. Similarly, in embodiments using compute virtualization, during operation, an instance of CCP software 1650 (shown as CCP instance 1676A) is executed on the virtualization layer 1654 (e.g., within instance 1662A). In embodiments not using compute virtualization, CCP instance 1676A executes as a monolithic kernel on the "bare metal" universal control plane device 1604 or on top of a host operating system. The instantiation of CCP instance 1676A, along with virtualization layer 1654 and instances 1662A-R (if implemented), are collectively referred to as software instance(s) 1652 .

[0136] In some embodiments, the CCP instance 1676A includes a network controller instance 1678. The network controller instance 1678 includes a centralized reachability and forwarding information module instance 1679 (a middleware layer that provides the context of the network controller 1578 to the operating system and communicates with various NEs), and a CCP application layer 1680 (sometimes referred to as the application layer) above the middleware layer (providing the intelligence required for various network operations such as protocols, network situation awareness, and user interfaces). At a more abstract level, this CCP application layer 1680 within the centralized control plane 1576 operates using (one or more) virtual network views (one or more) logical views of the network), and the middleware layer provides translation from the virtual network to the physical view.

[0137] The centralized control plane 1576 delivers relevant messages to the data plane 1580 based on the CCP application layer 1680 calculations and middleware layer mappings for each flow. A flow can be defined as a collection of packets whose headers match a given bit pattern; in this sense, traditional IP forwarding is also flow-based, where a flow is defined by, for example, the destination IP address; however, in other implementations, a given bit pattern for flow definition may include more fields in the packet header (e.g., 10 or more fields). Different NDs / NEs / VNEs in the data plane 1580 may receive different messages and, therefore, different forwarding information. The data plane 1580 processes these messages and programs the appropriate flow information and corresponding actions in the appropriate NE / VNE's forwarding table (sometimes also called a flow table). The NE / VNE then maps the incoming packet to the flow represented in the forwarding table and forwards the packet based on the match in the forwarding table.

[0138] Standards such as OpenFlow define protocols for messages and a model for processing packets. The model for processing packets includes header parsing, packet classification, and making forwarding decisions. Header parsing describes how to interpret packets based on a well-known set of protocols. Some protocol fields are used to build matching structures (or keys) that will be used in packet classification (for example, the first key field can be the source media access control (MAC) address, and the second key field can be the destination MAC address).

[0139] Packet classification involves performing a lookup in memory to determine which entry in the forwarding table (also known as a forwarding table entry or flow entry) best matches the packet based on the forwarding table entry's matching structure or key. It's possible that many flows represented in the forwarding table entries may correspond to / match a particular packet; in such cases, the system is typically configured to determine a forwarding table entry from among the many forwarding table entries according to a defined scheme (e.g., selecting the first forwarding table entry that matches). A forwarding table entry includes both a specific set of matching criteria (a set of values ​​or wildcards, or an indication of which part of the packet should be compared to one or more specific values / wildcards, as defined by a matching capability—either against a specific field in the packet header or against some other packet content) and a set of one or more actions to be taken by the data plane upon receiving a matching packet. For example, an action could be to push a header onto the packet, overflow the packet, or simply drop the packet for packets using a specific port. Thus, a forwarding table entry for IPv4 / IPv6 packets with a specific Transmission Control Protocol (TCP) destination port could contain an action specifying that these packets should be dropped.

[0140] Based on the forwarding table entries identified during packet classification, a forwarding decision is made and an action is performed by performing on the packet a set of actions identified in the matching forwarding table entry.

[0141] However, when an unknown packet (e.g., a "missed packet" or "match-miss" as used in OpenFlow parlance) arrives at the data plane 1580, the packet (or a subset of the packet header and content) is typically forwarded to the centralized control plane 1576. The centralized control plane 1576 will then program a forwarding table entry into the data plane 1580 to accommodate packets belonging to the flow of the unknown packet. Once a specific forwarding table entry is programmed into the data plane 1580 by the centralized control plane 1576, the next packet with matching credentials will be matched against that forwarding table entry and the set of actions associated with that matching entry will be taken.

[0142] Although the present invention has been described in terms of several embodiments, those skilled in the art will recognize that the present invention is not limited to the embodiments described and can be practiced with modification and alteration within the spirit and scope of the appended claims. Therefore, this description is to be regarded as illustrative rather than restrictive.

Claims

1. A method for a route prediction system utilizing real-time mobile communication network data, the method comprising: receiving (351) real-time mobile communication network data; aggregating the real-time mobile communication network data with historical data (353) to form a training data set; filtering (355) the training dataset to be relevant to service route predictions connecting transportation vehicles; Use the filtered training dataset to train (357) multiple offline models; Selecting (363) the best performing offline model from the plurality of offline models; and generating route prediction information for the connected vehicle using the selected offline model, wherein the route prediction information includes a recommendation of which mobile network operator (MNO) to select on each segment of a route associated with the trip based on an expected start time of the trip of the connected vehicle, The training data set also includes mobile connected vehicles identified from the real-time mobile communication network data and their experience information.

2. The method according to claim 1, wherein Filtering removes data that is not informative for determining the quality of service of connecting vehicles along a route.

3. The method of claim 1, further comprising: The performance of at least one trained offline model from the plurality of offline models is verified (361).

4. The method according to claim 3, wherein: Validation determines the accuracy of each of the plurality of offline models.

5. The method according to any one of claims 1 to 4, wherein The real-time mobile communication network data includes key performance indicators of cells of the mobile communication network, and the key performance indicators are organized into time series data categorized by predictability and statistical characteristics of each cell.

6. The method of any one of claims 1 to 4, further comprising: Continuous tracking accuracy of each of the plurality of offline models is maintained.

7. The method according to any one of claims 1 to 4, wherein: Each of the multiple offline models can output a key performance indicator for measuring service quality.

8. The method of claim 7, wherein: The output key performance indicators include received signal power (RSRP) or received signal quality (RSRQ).

9. The method according to any one of claims 1 to 4, wherein: A separate offline model can be maintained for each mobile network operator.

10. The method of claim 5, wherein: A combination of offline models can be utilized to generate a quality of service measure for location prediction in the mobile communication network.

11. The method of claim 9, wherein: Inputs to the offline model can include a digital terrain model of cells, cluster categories, and antenna properties for the mobile network operator.

12. The method according to any one of claims 1 to 4, wherein: Filtering includes: In response to identifying that the data is from a cell tower handoff, correlated with an unwalkable distance, and along a route, the data is stored (505).

13. The method of any one of claims 1 to 4, further comprising: Mapping the real-time mobile communication network data to a route by, Merging (603) the route information with the mobile communication network location information, determining (604) whether a mobile communication network component is geographically located proximate to the route, and It is determined (605) whether the mobile communication network component signal coverage covers the route.

14. A network device, comprising: a non-transitory machine-readable storage medium (1518, 1548, 1648) having a route prediction block stored therein; as well as A processor (1512, 1542, 1642) coupled to the non-transitory machine-readable storage medium, the processor being configured to execute the route prediction block, the route prediction block performing the method of any one of claims 1-13.

15. The network device according to claim 14, wherein: The network device is configured to execute a plurality of virtual machines that implement network function virtualization (NFV).

16. An electronic device in a software-defined networking (SDN) network comprising a plurality of data plane devices, the electronic device comprising: a non-transitory machine-readable storage medium (1518, 1548, 1648) having a prediction service stored therein; as well as A processor (1512, 1542, 1642) coupled to the non-transitory machine-readable storage medium, the processor configured to execute the prediction service, the prediction service performing the method of any one of claims 1-13.

Images