Device and method for edge-based notifications through crowdsourced, live-streamed fleet communication

Abstract

Device comprising: a processor (315) in an edge server (310) that communicates with one or more vehicles (10, 90) in a fleet, to: Communicating and maintaining a continuous communication link between one or more vehicles (10, 90) of the fleet to receive a multitude of messages, wherein the multitude of messages comprises a continuous stream of message data containing state transition data of electronic control units, ECUs, contained in each vehicle (10, 90) in a fleet of vehicles (10, 90) while performing vehicle operations in an environment; Monitoring the streamed message data from each vehicle (10, 90) in real time to detect updates of data about state transitions of ECUs used in vehicle operation, with the data of each state transition of the ECUs being reported to the edge server (310) by an advanced driver assistance system, ADAS, included in each vehicle (10, 90); Crowdsource, in real time, a set of data of the state transitions of ECUs or of different operating levels of ECUs during the vehicle operation of each vehicle (10, 90) in communication with the edge server (310), to aggregate state transition data with a certain degree of commonality as crowdsourced transition data; Processing, by applying a machine learning model, the crowdsource transition data to classify crowdsource transition data that indicates an anomaly event of an assisted driving system; and Sending, in advance, a message to an identified vehicle where the assisted driving system anomaly event is likely to occur, so that a driver or the ADAS can take action.

Description

[0001] The present description generally refers to an edge-based carrier network connected to a fleet of connected vehicles, and in particular, aspects of the present description relate to systems, methods and devices for collecting real-time streaming raw signal data from a fleet of connected vehicles for analysis in order to detect operational anomalies of nearby vehicles in real time, to distribute early warnings of predicted anomalies to other vehicles in a road segment for corrective action, to complement sensor capabilities or to eliminate the need for the use of sensors of each individual connected vehicle.

[0002] The operation of modern vehicles is becoming increasingly automated, meaning they are able to control driving with less and less driver intervention. Vehicle automation has been classified into numerical levels, ranging from zero, which corresponds to no automation with full human control, to five, which corresponds to complete automation without any human control. Various advanced driver assistance systems (ADAS), such as cruise control, adaptive cruise control, and parking assistance systems, correspond to lower levels of automation, while truly "driverless" vehicles correspond to higher levels.

[0003] DE 10 2015 221 292 A1 describes a method and a device for collecting vehicle data. The method for collecting vehicle data in a vehicle data collection server, which communicates with a vehicle via a wireless network, comprises receiving vehicle data corresponding to the first to nth data elements from a first vehicle. If alternative data collection vehicles are required, the method searches for the alternative data collection vehicles in a group of vehicles for each of the first to nth data elements, receives vehicle data corresponding to at least one of the first to nth data elements from the sought-after alternative data collection vehicles, and stores the vehicle data received from the first vehicle and the vehicle data received from the alternative data collection vehicles.

[0004] German Patent Application DE 10 2019 104 454 A1 describes a system in which at least one transceiver is configured to wirelessly receive a unique identifier from a source node located outside a vehicle. A distributed ledger contains a list of unique IDs associated with trusted source nodes. An ID comparison module is configured to compare the source node's unique ID against the list. Upon determining that the source node's unique ID is not in the list, the at least one transceiver is configured to wirelessly transmit the source node's unique ID to the trusted source nodes of a connected network for the execution of a consensus algorithm. A ledger management module is configured to add the source node's unique ID to the list upon determining that the source node is trusted by executing the consensus algorithm.

[0005] German patent application DE 10 2018 122 588 A1 describes systems and methods for cooperation between autonomous vehicles. A processor-implemented method for coordinating journeys between multiple autonomous vehicles is provided. The method includes sending a collaboration request to one or more vehicles in an area to form a group for carrying out a mission, receiving an acceptance of the collaboration request to join the group (which may comprise a multitude of vehicles), cooperating in assigning leading functions for the group to one or more of the multitude of vehicles in the group, cooperating in mission negotiations for the group, cooperating in determining a formation for the group, and cooperating in generating a trajectory for the group. The vehicles in the group are operated according to the determined formation and the generated trajectory.

[0006] It is desirable to implement edge carrier network systems that can collect real-time streaming raw signal data of operating states of various vehicle systems from a fleet of connected vehicles within a defined control zone, analyze the data for anomalies relating to vehicle systems, such as hard braking, wheel slippage, or stalling, and, if detected, send instructions to relevant vehicles in the area for driving warnings or, for example, instruct an automatic emergency braking response via a vehicle control signal.

[0007] It is desirable to implement systems to generate warnings and ADAS instructions for a specific vehicle regarding incidents, operational anomalies, and the like, without relying on vehicle sensors to perceive the environment around the vehicle. It is desirable to provide predictive anomalies along a vehicle's path, detectable through vehicle-to-vehicle communication, which, unlike sensor-based detection, is free from everyday sensor operational failures, such as when vehicle detection conditions are outside the field of view of the vehicle's own sensors, or when the sensor line of sight is obstructed or the sensors are covered.

[0008] It can be considered a task to specify an improved device and an improved method for edge-based notifications in automotive fleet communication.

[0009] The problem is solved by a device according to claim 1 and a method according to claim 10. Furthermore, a driver assistance system is described, which is an application of the device according to the invention.

[0010] This document describes autonomous vehicle control systems, training systems and associated control logic for autonomous vehicle control, methods for operating such systems, and motor vehicles equipped with onboard control systems. As an example, and without limitation, a motor vehicle with in-vehicle machine learning and control systems is presented, which utilize crowdsourced data from a vehicle fleet to predict anomalies in vehicle operation.

[0011] A device according to the invention is described, wherein the device includes a processor on an edge server in communication with one or more vehicles in a fleet. The device is capable of establishing and maintaining a continuous communication link between one or more vehicles of the fleet in order to receive a plurality of messages, wherein the plurality of messages contains a continuous stream of message data, which includes data of state transitions of electronic control units (ECUs) contained in each vehicle in a fleet of vehicles while they perform vehicle operations in an environment.Monitoring the streamed message data from each vehicle in real time to detect updates to data about state transitions of ECUs used in vehicle operations, with the data of each ECU state transition being reported to the edge server by an advanced driver assistance system (ADAS) included in each vehicle; crowdsourcing in real time a set of state transition data or data from different levels of operations of ECUs during the vehicle operations of each vehicle in communication with the edge server to group state transition data with a degree of commonality as crowdsourced transition data; processing the crowdsourced transition data by applying a machine learning model to classify crowdsourced transition data that indicate an anomaly event of the assisted driving system;and transmitting a message in advance to an identified vehicle that is likely to experience the assisted driving system anomaly event, so that a driver or ADAS can take a response.

[0012] In various embodiments, the device further comprises the processor on the edge server, which communicates with the one or more vehicles in a fleet and is furthermore capable of: simulating an algorithm for an assisted driving system over a predicted route segment for an identified vehicle in order to generate a simulation result; predicting a predicted anomaly event for an assisted driving system within the predicted route segment based on an analysis of the crowdsourced data and the simulation result.The device further comprises the processor in the edge server, which communicates with the one or more vehicles in a fleet and is also capable of sending a user warning about the predicted anomaly event of the supported driving system before the identified vehicle triggers the predicted anomaly event of the supported driving system while performing vehicle operations. The device further comprises the processor in the edge server, which communicates with the one or more vehicles in a fleet and is also capable of sending a warning message to vehicles in the vicinity in response to the predicted anomaly event of the supported driving system.The device further comprises the processor on the edge server, which communicates with one or more vehicles in a fleet and is also capable of monitoring one or more key parameters relating to the operation of each vehicle in the fleet. The identified vehicle, which is likely to experience the assisted driving system anomaly, is driving behind a vehicle in which the assisted driving system anomaly has already occurred.The device further comprises the processor on the edge server, which communicates with one or more vehicles in a fleet and is further capable of receiving transition state data indicating a previous anomaly event of the assisted driving system of a vehicle in the fleet traveling on a nearby route segment, and wherein the assisted driving system anomaly event is predicted based on or in response to the previous assisted driving system anomaly event. The message data is transmitted over a cellular network via an MQTT telemetry transport and data distribution service (DDS) radio protocol.The device further comprises the processor in the edge server, which communicates with the one or more vehicles in a fleet and also serves to send the message about the anomaly event of the assisted driving system via a reverse path using the cellular protocol to a messaging client in the vehicle, in order to transmit it via a Controller Area Network (CAN) bus to an ADAS controller in the vehicle to perform a control action on an ECU located in the vehicle. The degree of commonality includes at least one vehicle location and a vehicle speed in the direction of an anomaly event location of the assistance system.

[0013] A method according to the invention is described, which is executed by a processor. The method comprises communicating a plurality of messages between a processor at an edge server and one or more vehicles in a fleet; maintaining a continuous cellular connection between the processor at the edge server and the one or more vehicles of a fleet for receiving the plurality of messages, wherein the plurality of messages contains a continuous stream of message data, which includes data of state transitions of control units contained in each vehicle in a fleet of vehicles during vehicle operation;Monitoring the streamed message data from each vehicle in real time to capture updates of data on state transitions of electronic control units (ECUs) used in vehicle operations, with the data of each ECU state transition being reported to the edge server by an advanced driver assistance system (ADAS) included in each vehicle; crowdsourcing in real time a set of data on state transitions or varying levels of operation from ECUs during the vehicle operations of each vehicle in communication with the edge server to aggregate state transition data with a degree of commonality as crowdsourced transition data; processing the crowdsourced transition data by applying a machine learning model to classify crowdsourced transition data indicating an anomaly event of the assisted driving system;and transmitting a message in advance to an identified vehicle where the assisted driving system anomaly event is likely to occur, so that a driver or ADAS can take a vehicle control action.

[0014] In various exemplary embodiments, the method further comprises the simulation of an assisted driving system solution over a predicted route segment for an identified vehicle by the processor to generate a simulation result; and the prediction of a predicted anomalous event within the predicted route segment in response to the crowdsourced data and the simulation result. The method further comprises sending a driver alert about the predicted anomaly event before the identified vehicle experiences the predicted anomaly event, the driver alert indicating a probability of the predicted anomaly event. The method further comprises the processor sending a warning message via a cellular network to vehicles in the vicinity in response to the predicted anomaly event.

[0015] In one embodiment, the method comprises monitoring one or more key parameters relating to the vehicle operation of each vehicle in the fleet. The identified vehicle with the probability of experiencing the anomalous event travels behind a vehicle where the anomalous event has already occurred. The method further comprises receiving transition state data indicating a previous anomalous event of a vehicle in the fleet traveling on a nearby track segment, and predicting the anomalous event in response to the previous anomalous event. The message data is transmitted over a cellular network using an MQTT and a DDS radio protocol.The procedure further includes sending the message about the anomaly event by the processor via a reverse path through the cellular protocol to a messaging client on the vehicle, in order to transmit it on a CAN bus to an ADAS controller of the vehicle in order to perform a control action on an ECU contained in the vehicle.

[0016] An advanced driver assistance system for controlling a vehicle is described, wherein it is a use case of the device and the driver assistance system comprises the device. The system includes a processor in an edge server that communicates with one or more vehicles in a fleet to: establish and maintain a continuous communication link between one or more vehicles in a fleet to receive a multitude of messages, wherein the multitude of messages contains a continuous stream of message data including state transition data of control units contained in each vehicle in a fleet of vehicles while vehicle operations are performed in an environment;to monitor the streamed message data from each vehicle in real time to capture updates of data on state transitions of ECUs used in vehicle operations, wherein the data of each ECU state transition is reported to the edge server by an advanced driver assistance system (ADAS) included in each vehicle; to crowdsource in real time a set of data of state transitions or different levels of operations of ECUs during the vehicle operations of each vehicle in communication with the edge server in order to group state transition data with a degree of commonality as crowdsource transition data, wherein the degree of commonality includes at least the vehicle location and the vehicle speed in the direction of proximity to the anomaly location;Processing the crowdsource transition data by applying a machine learning model to classify crowdsource transition data indicating an assisted driving system anomaly event; and transmitting a message in advance to an identified vehicle where the assisted driving system anomaly event is likely to occur, so that a driver or ADAS can take action. Fig. Figure 1 schematically shows a diagram of a crowdsourced communication system for collecting raw streaming signal data from a fleet of connected vehicles in accordance with various embodiments; Fig. Figure 2 shows an exemplary diagram of a system with a central gateway module connected to a vehicle, where primary anomaly detection takes place at an edge in a cellular network between vehicles in accordance with various embodiments; Fig. Figure 3 shows an exemplary diagram of the architecture of vehicle control components in communication with an edge server for the determination of anomalies based on the analysis of crowdsourced data in accordance with various embodiments; Fig. Figure 4 shows an exemplary flowchart of a method for predicting anomalies on a roadway in advance, based on communication between vehicles in a fleet by analyzing crowdsourced data in accordance with various embodiments; Fig. Figure 5 shows an exemplary block diagram of a system for predicting anomalies of the automated driving system by analyzing crowdsourced fleet data in a vehicle; and Fig. Figure 6 shows an exemplary flowchart for predicting automated driving system anomalies based on crowdsourced data analysis in accordance with various embodiments.

[0017] This description uses various terms; for example, a Controller Area Network (CAN) is a vehicle bus standard that allows devices and control units to communicate with each other without an identified computer. The CAN bus acts as a central networking system, enabling different electronic control units (ECUs) of vehicles to communicate with other ECUs. It is intended that the present system can be implemented not only with a CAN configuration for communication between vehicles, but also with CAN FD (flexible data rate CAN) in a similar manner. CAN FD increases the bandwidth requirements within vehicle networks. The CAN FD protocol brings the software closer to "real-time" by minimizing the delays between command and data transmission (latency) and by using higher bandwidths.

[0018] This description outlines various exemplary embodiments that provide processes using an in-vehicle processor that streams the vehicle bus (CAN / LIN / Flexray, etc.) to continuously transmit signals over the air through a vehicle modem. Furthermore, an edge-based system located at the edge of the telco carrier receives signals and detects or identifies anomalies, such as large changes in speed (hard braking), optionally using crowdsourcing algorithms. The edge-based system then builds graphs or models of vehicle relationships and continuously monitors for incidents, generating and sending appropriate alerts and / or commands to relevant vehicles near the incident. The vehicle's onboard processor receives the commands and instructs the ECU to display the FCA (Fuel Acceleration Warning) or to instruct the ADAS (Advanced Driver Assistance System) ECU to take action.

[0019] In various exemplary embodiments, this description describes the direct streaming of CAN / vehicle signals to edge compute for inference, where the centralized crowdsourced data collection takes place on an edge server and not locally between vehicle-to-vehicle systems. It also describes computation at edge server locations with low-latency carrier network access, synchronous fleet predictions using Kalman filtering, and contextual filtering of commands via artificial intelligence solutions, including neural networks, on the edge server to send relevant vehicles current navigation direction and historical vehicle operation data based on the vehicles' current navigation direction.

[0020] Fig. Figure 1 schematically shows a diagram of a crowdsourced communication system for collecting raw streaming signal data from a fleet of connected vehicles in accordance with various embodiments. Fig. 1. The communication system 100 of the connected fleet in the cellular domain is configured by an edge server 95 of a carrier network, which enables the collection and aggregation of real-time streaming raw signal data from a fleet of connected vehicles (i.e., connected vehicle 10, connected vehicle 90) within a specific control zone. The edge server 95 can analyze the collected data for various anomalies such as hard braking, wheel slippage, idling, etc. When an anomaly is detected, the communication system 100 is configured to send one instruction (i.e., a CAN message) or multiple instructions to relevant vehicles to warn the driver of each relevant vehicle of a specific driving warning, such as...the need for automatic emergency braking and the like, or to instruct advanced driver assistance systems (ADAS) to change a functional state of a control unit system in the vehicle at a suitable time and place, determined by the detected anomaly on a road segment and the current vehicle speed, location and path.

[0021] As in Fig. As shown in Figure 1, the raw CAN messages are continuously streamed (i.e., transmitted) from vehicle 10 to cellular network 20 and back to vehicle 90 via a continuously connected messaging protocol such as MQTT or DDS. The message from vehicle 90 is routed by cellular network 20 to the nearest edge server (i.e., edge server 95) that controls the geographic area where vehicle 90 is located. That is, once vehicle 90 moves outside the geographic area of ​​a particular edge server, the message from vehicle 90 would be forwarded to another edge server that controls the other geographic area where vehicle 90 is now located. The messages are then received or sent via the message gateway broker 30 (located in edge server 95), which acts as an interface between the edge server components and cellular network 20 (i.e., edge server 95).(the mobile phone mast). Each CAN message received via the message gateway broker 30 is decoded and / or encoded at the message decoder / encoder 40. When the message is decoded at the message decoder / encoder 40, the values ​​extracted from the message are digitally written to the memory of at least one set of twin vehicles in a fleet of vehicle and road models 50. That is, the fleet and road model 50 distributes the decoded messages to the relevant vehicles, with the relevance of a vehicle being determined based on an identified or parameters such as vehicle location, vehicle direction, time of the extracted message, type of information obtained from the decoded message, etc.For example, a set of key parameters of vehicles in the fleet can be predicted based on the data extracted and processed by the fleet and road model 50 60, or it can be continuously predicted in the absence of and / or between updates from the vehicle 90 (e.g., Kalman filtering), resulting in predictions about the current location of the vehicles, the road positions, and the directions of travel of the vehicles. The event detector 70 processes analyzed vehicle states to react to anomalies. This could be, for example, emergency braking or a standstill and the corresponding emergency braking or standstill state of the vehicle 10. Upon detection of a vehicle state (i.e.,Upon detection of a changed vehicle characteristic state, a notification can be generated and sent to a Command Manager 80, which in turn sends notifications to alert nearby vehicles in the vicinity via broadcasting CAN messages. This ensures they are informed in advance about a response to an anomaly and warns the driver of any potentially necessary action, if applicable to the vehicle. The Command Manager 80 checks nearby vehicles in its in-memory database using a spatial query and selects vehicles traveling on the same lane behind the anomaly event.The commands sent by the command manager follow a path or processing pipeline via the message decoder / encoder 40 to the message gateway broker 30, leave the edge server 95 via the cellular network 20, and communicate the command to the vehicle's ADAS or the driver of the vehicle 90. The vehicle 90 can also notify the driver or the ADAS upon receiving the CAN message.

[0022] Therefore, the ADAS is capable of executing a methodology for predicting the future feature state of an automated driving system (i.e., a control unit) to provide drivers with early feedback and improve the user experience. The methodology is functional for predicting anomalies based on crowdsourced fleet data, leading to improvements in the dynamic path and speed profiling of ADAS state analysis in an ADAS-equipped vehicle. The methodology can utilize a model trained on crowdsourced data collected from the automated driving fleet to identify micro-patterns at the road segment level and location-independent macro-patterns.The method can then model anomalies found through crowdsourcing in the vehicle's driving operation in future segments of the predicted vehicle path, and send early warnings via feature state transitions or CAN messages as the vehicle continues operation to the next road segments.

[0023] In various exemplary embodiments, key parameters (i.e., parameters defined in CAN messages) of vehicles in the fleet can be continuously predicted in the absence of or between updates by implementing a factorial formulation that allows inference on previously unobserved road segments, with the inference based on crowdsourced fleet data. For example, factorial hidden Markov models (FHMM) can be employed by treating sequences of individual feature states of ECUs responding to conditions such as traffic, weather, roadworks, and / or road segments as dependent only on the previous state of the respective feature, and the current observation as dependent only on the current state of all features. FHMM enables a distributed representation of features and allows prediction even with incomplete data, such as...This applies to driving on a previously unrecorded section of road or in unknown weather conditions. This Bayesian approach allows for the inherent capture of uncertainty due to missing or incomplete information. The output of a FHMM (Future Fleet Management Model) contains information about the degree of confidence the model has in each prediction based on current state fleet observations to determine probable future states.

[0024] Fig. Figure 2 shows a block diagram illustrating an exemplary implementation of a system with a central gateway module for predicting anomalies based on crowdsourced communication between vehicles, according to various embodiments. The central gateway module 200 is configured with various onboard systems, including an interface 220 (i.e., CAN bus) that receives CAN messages about the states of vehicle systems (e.g., wheel speed, location (GPS), brake pressure) cyclically placed on a CAN bus by various control units (210a, 210b) and / or brake control units or ADAS control units 260. The CAN messages are forwarded to a CAN message transmit / receive module 230, configured as a software component in the central gateway module 200, which encodes the CAN message for transmission to vehicles based on location, direction, anomaly type decoded from the CAN message, etc.are considered to belong to a group. In return, the CAN message is received and sent by a messaging client 240 via a telematics module 250 that is connected to or integrated with a vehicle, the vehicle being configured to transmit and receive radio signals using MQTT or DDS or an equivalent.

[0025] In vehicle operation scenarios where a reversing message is received via the telematics module 250, the message is first received by the telematics module 250 (from the edge server (not shown)) and transmitted to the messaging client 240 on the central gateway module 200 located in the vehicle. The (CAN) transmit / receive module 230 receives the message (decoded) from the messaging client 240 and sends a CAN message on a CAN bus 220 to the vehicle's ADAS controller 260. The CAN message can, for example, contain parameters for configuring the vehicle's ADAS by setting various control units (210a, 210b) to operating states at the instruction of the ADAS control unit 260. That is, the ADAS control unit 260 can react to the CAN message, or the other control unit can react to it and also issue a Human Machine Interface (HMI) warning.

[0026] In cases of vehicle operation where forward path notification is present, the CAN message(s) from the vehicle's ADAS control unit (310, 260) and state changes of the control units (210a, 210b) are sent or transmitted to the vehicle(s) via the telematics module (250). In this case, CAN messages are generated and sent by the ADAS control unit (260) on the CAN bus (220) to the message send / receive module (230) as a CAN message. These messages are then configured (i.e., encrypted) by the messaging client (240) to be sent by the central gateway module (200) to the telematics module (250). The telematics module (250) can then send the message to vehicles in the vicinity or to the edge server for crowdsourced analysis.

[0027] While the ADAS control unit 260 in each vehicle is likely also paired and functional to use and receive instructions based on data derived from the various sensors implemented in the vehicle itself to perceive the vehicle's environment, the responses instructed to the ADAS control unit 260 via the telematics module 250 are based on crowdsourced data analysis of groupings of vehicle systems (i.e., for example, clustering of state changes of sets of vehicle systems) and not on the individual raw data acquired by each set of vehicle sensors. For the described crowdsourced fleet communication system for anomaly detection in vehicle operation, there is neither a dependency on nor a use of the raw sensor data from each set of individual sensors in the vehicle when configuring the CAN message transmissions of the central gateway module 200.In this case, CAN messages are generated based on continuous monitoring of event data from vehicle states. This data is then analyzed in an aggregated manner (i.e., for crowdsourcing detection) for anomalies in vehicle operation while groups of vehicles are operating at a location. The analysis is performed at a central location. For example, the analysis can be performed at the nearest edge server, where the anomalies can be detected, distributed, and / or sent to other vehicles, or at other cloud-connected locations with analysis engines capable of communicating via the cellular protocols established between the Telematics Module 250 and the fleet vehicles.

[0028] Fig. Figure 3 shows an exemplary architecture of vehicle control components communicating with an edge server to determine anomalies based on crowdsourced data, according to various embodiments. The user interface 335 (i.e., HMI) can be a user input device, such as a screen, LED, audible alarm, or haptic seat located in the vehicle cabin and accessible to the driver. Alternatively, the user interface 335 can be a program running on an electronic device, such as a mobile phone, and communicating with the vehicle, for example, via a wireless network. The user interface 335 serves to collect instructions from a vehicle operator, such as initiating and selecting an ADAS function, the desired following distance for adaptive cruise control operation, selecting vehicle motion profiles for assisted driving, etc.In response to a selection by the driver, the user interface 335 can be used to couple a control signal or similar to the processor 340 to activate the ADAS function. Furthermore, the user interface can be operated to issue a user prompt or warning indicating an impending anomaly in vehicle operation on a vehicle path that affects the ADAS function, or a prompt such as the need for the user to take greater control of the vehicle due to a safety issue (e.g., unsafe weather conditions, etc.) or a likely anomaly of a driver assistance function.

[0029] The transceiver 333 sends and receives data over a wireless network (i.e., a cellular network) to an edge server 310 (i.e., a local central server or a cloud server connected to the cellular network). The transmitted CAN message data can include instances and locations where an anomaly-type event occurred during ADAS operation, based on analysis of communication between vehicles in a fleet. This CAN message data can be transmitted by the transceiver 333 in response to a request, periodically, or after one or more anomalies. The transceiver can remain operational to receive CAN message data from the edge server indicating anomaly locations, other ADAS operational state transitions, and / or other crowdsourced data (i.e., derived fleet CAN message data from various control units of vehicles in the fleet in the vicinity), such as...Weather, road conditions, obstacles, roadworks, traffic and the like, which can be used to predict an ADAS state transition, such as an anomaly event.

[0030] In an exemplary embodiment, the processor 340 is capable of receiving CAN message data streamed in real time from the vehicle-mounted transceiver 333 and receiving instructions for various ADAS operating state transition predictions based on analytical algorithms that perform predictive analysis based on current and historical crowdsourced data. The predicted transition states for the vehicle's ADAS are derived on a remote edge server 310. This is because the remote edge server is configured to continuously communicate and collect streamed data from multiple vehicles (i.e., real-time streamed crowdsourced vehicle data communication and collection).In one scenario, the Processor 315 on the Edge Server 310 is programmed to model vehicle operation and simulate or model the control of the vehicle as it travels a number of upcoming route segments. These segments are to be navigated by the vehicle's motion planning and autonomous driving systems, including ADAS (Advanced Driver Assistance System). The key parameters of each vehicle, required to perform autonomous or assisted driving operations for each vehicle in the fleet, are continuously predicted by the Processor 315 on the Edge Server. This processor acts as an analytics engine, continuously analyzing changes in vehicle states for anomalies through crowdsourced data clustering, other analytical solutions, or various machine learning applications on the identified or predicted route segments.Furthermore, in an alternative exemplary embodiment, the processor 340, which is contained in the vehicle, can be implemented to receive data actions from the in-memory database 320, which is transmitted by the processor 315 at the edge server 310. In response to the modeling, the processor 315 is operational to generate a prediction that indicates the probability of the vehicle encountering the anomaly in advance, based on factors including the time, distance, speed, weather conditions of the vehicle's surroundings, or the grouping of factors of several following vehicles.

[0031] If the probability of an anomaly is deemed high (e.g., a threshold may be associated with an anomaly event detection as an indicator of a probability high enough to alert the vehicle operator or instruct the ADAS), a user prompt or warning is generated in advance and transmitted to the user interface 335 via the cellular network. For example, if the processor 340 determines that an anomaly is likely within a certain distance of the cellular communication received via the transceiver 333, a user prompt can be sent to the user interface 335 to ensure the smooth operation of the vehicle's control systems and to give the vehicle operator sufficient time to safely activate or modify the relevant vehicle controls. This data can also be used by the processor 315 for the (supervised) training of a learned road segment model (i.e.,a machine learning model for instructing ADAS) based on crowdsourced data.

[0032] The vehicle control unit 330 can generate control signals for coupling with other vehicle system control units, such as a throttle control unit 355, a brake control unit 360, and a steering control unit 370, to control the operation of the vehicle in response to instructions provided by the ADAS algorithm. The vehicle control unit 330 can be operational to adjust the vehicle's speed by reducing the throttle valve via the throttle control unit 355 or to actuate the friction brakes via the brake control unit 360 in response to a control signal generated by the processor 340. The vehicle control unit 330 can also be operated to adjust the vehicle's direction by controlling the vehicle's steering via the steering control unit 370 in response to control signals generated by the processor 340.

[0033] Now to Fig. Figure 4, a flowchart showing an exemplary implementation of a method 400 for predicting anomalies on a roadway in advance based on communication between vehicles in a fleet via crowdsourced data in accordance with various embodiments. The method first serves to receive a route request via a user interface or via wireless transmission (e.g., cellular transmission) 410. The route request may specify a destination or a destination with a preferred route. The route request may further initiate an ADAS function, such as adaptive cruise control, or other ECU settings in response to a user request via a user interface or in response to CAN messages transmitted to the vehicle from the nearest edge server via a telematics module.

[0034] Next, procedure 420 determines the vehicle's current location. This location can be determined in response to a received GPS output and / or high-resolution map data, or similar information. Based on the vehicle's current location, the procedure can be operational to generate a route between the current location and the destination, using stored map data and data received over a wireless network. The map data and received data can include road information, traffic data, weather conditions, construction information, and similar information. The route can be divided into route segments, with the ADAS system navigating the vehicle sequentially through each of the route segments.

[0035] Next, the procedure performs an ADAS operational change (430) that responds to an environment, condition, etc., found in a (440) motion path modeled by a predictive navigation algorithm. During the ADAS operation, the procedure is ready to identify or predict an anomaly on a route while navigating a predicted navigation path on a route segment. For example, in response to an environmental condition in the navigated route segment, a specific command responding to the impending anomaly is pre-sent to a vehicle based on the analysis of crowdsourced vehicle data in an analysis engine on a remote edge server. This command then delivers a CAN message to various ECU systems of the vehicle's ADAS to correct or pre-respond to the anomaly predicted in the upcoming route segment.Therefore, the crowdsourcing of collected data from vehicles in the vicinity or from vehicles traveling the specified route is applied to the predictive model for the modeled movement path and used as a basis for predicting further anomalies for the vehicle during the journey and on the way to the next route segment in order to adapt various control units of the moving vehicle.

[0036] In various exemplary embodiments, the method can be implemented to determine a threshold value 445 for identifying an anomaly or for comparing an anomaly type to which a response must be made in the motion path, based on a clustering of a number of vehicles that cause a state change in various ECU systems for controlling the vehicle in the upcoming segment. The threshold value can be applied to how much an ECU must change state to react to the impending anomaly.

[0037] Additionally, the ADAS operation procedure employs a predictive navigation algorithm (using a machine learning model) to predict the likelihood of an anomaly in an upcoming route segment. The procedure receives a predictive (simulation) model generated from crowdsourced data (e.g., data streamed in real time from a number of vehicles in the fleet) regarding the upcoming route segments. Next, the procedure simulates virtual navigation of the vehicle on an upcoming segment to further predict anomalies in advance.

[0038] In an exemplary embodiment, the method uses received state data from control units and / or current streamed state data from control units ahead of the current vehicle to configure various control units of the vehicle or to warn the driver of the upcoming road segment. The method uses an FHMM model or other type of AI model on the edge server that incorporates features such as detected anomalies, location, weather, road segment, road type, map version, construction, ambient traffic, and road material. This model can be used to capture transitions between feature states along the road segment and state changes of different control units depending on these features as the vehicle traverses the road segment.

[0039] The method is used to generate or send a user alert 450 indicating the anomaly via messages. The user alert can be displayed through a user interface and can show a probability of the anomaly occurring and the vehicle's distance to the anomaly event. The user interface can, for example, consist of a variety of LEDs that change color depending on the probability of the anomaly occurring and / or the distance to the likely anomaly event. The method can couple this user alert, the score and / or the probability and location to the ADAS or the vehicle control system for use by the ADAS. The method can then be used to simulate the next section, with the number of simulated route segments ahead of the vehicle's location being dynamically determined, e.g.,due to distance and speed or another design requirement.

[0040] Additionally, the predictive information can be sent wirelessly (cellularly) to a local edge server when either the state changes or a certain distance / time has elapsed. If the state has changed, an efficient learning algorithm on the Edge Server 460 can update a state transition model in the data, which is then transmitted to other vehicles expecting to travel the same section of road. A cloud application on the edge server can simulate a vehicle traveling along the learned virtual road model to determine the likelihood of a state change. The cloud algorithm can then use the forward-backward algorithm for the FHMM to perform a belief propagation prediction for the next n road segments, where n can be dynamically determined, for example, by distance and speed.Due to the factorial nature of the FHMM, the cloud model can use partial knowledge to make predictions about state change probabilities on road segments that have not been encountered before. If the cloud application determines that a change in the ADAS state is likely in response to the segment conditions, it can update the information delivered to the vehicle indicating the probability of the control unit operating states changing.

[0041] In Fig. Figure 5 is a block diagram showing another exemplary implementation of a System 500 for predicting anomalies of the automated driving system using crowdsourced fleet data in a vehicle. The system may be an advanced driver assistance system for controlling an identified vehicle, comprising a transceiver 510, a processor 520, a user interface 530, and a vehicle controller 540.

[0042] The transceiver 510 can be a high-frequency transceiver, such as a device for a cellular network, capable of sending and receiving data over a cellular data network to a remote edge server. In this exemplary embodiment, the transceiver 510 is able to receive data indicating an anomaly event in the assisted driving system, provided by a first vehicle. The data can be generated in response to a large number of anomalies detected and transmitted by a multitude of vehicles. The model can then be used to predict an anomaly event in response to identified vehicle dynamics. In one exemplary embodiment, the anomaly event is detected in response to a request from an ADAS.

[0043] The exemplary system 500 further comprises a processor 520 that simulates an ADAS algorithm based on crowdsourced data analysis at the edge server 550 over multiple route segments to generate a predicted or simulated result, wherein the processor further predicts a predicted anomalous event within future route segments in response to either individual or crowdsourced vehicle state data and the simulation result, and generates a warning control signal in response to the predicted anomalous event.The processor 520 can further be operational to generate a route in response to a destination and an identified vehicle location, to determine the first route segment and any further or predicted route segment in response to the route, to generate a first motion path in response to the first route segment, and to couple the first motion path with the vehicle controller 540 to control the vehicle via the first route segment.

[0044] The exemplary system 500 can further include a user interface 530 to present a user warning or messages about the predicted anomaly event in response to the warning control signal, before the identified vehicle reaches the next track segment. The user interface 530 can be a screen in a vehicle cabin, perhaps one or more LEDs, a haptic seat, and / or an audible alarm.

[0045] In one exemplary embodiment, the prediction of an anomalous event can be performed using a machine learning model configured on the Edge Server 550. This model utilizes models such as a factorial hidden Markov model, a filtering model, a regression or classification model, or a neural network, continuously evaluating crowdsourced data transmitted by vehicles in the vicinity to the processors on the Edge Server. Furthermore, each machine learning model can be trained on crowdsourced data collected from an automated driving fleet, facilitating the detection of micro-patterns at the road segment level and location-independent macro-patterns. The Processor 520 is capable of simulating the operation of a virtual vehicle along a road segment and evaluating all models. Each model can provide insights into previously unobserved road segments.

[0046] The system can further include a vehicle controller 540 that controls an identified vehicle over the first route segment in response to an ADAS algorithm, such as an adaptive cruise control algorithm. The predicted anomalous event is forecast using a machine learning model that uses crowdsourced data and data from the vehicle controller 540 for analysis. The vehicle controller can be operated to transmit current feature data to the processor 520 and receive control instructions from an ADAS controller. In an exemplary embodiment, the processor 520 is also the ADAS controller. The vehicle controller can control the identified vehicle by controlling a steering controller, brake controller, and / or throttle controller.

[0047] Now to Fig.Figure 6, a flowchart illustrating an exemplary implementation of a system for predicting anomalies of the automated driving system based on crowdsourced data in an identified vehicle according to various embodiments. Exemplary method 600 first computes 610 a route between an identified vehicle location and a destination. The identified vehicle location can be determined in response to a measurement from the global positioning system indicating the current location of the identified vehicle. Alternatively, the identified vehicle location can be determined in response to map data stored in a memory within the identified vehicle. The destination can be determined in response to user input or in response to a signal received via a wireless network.The route can be calculated using map data, current traffic, weather, user preferences, vehicle characteristics, and similar factors.

[0048] Next, method 620 segments the route into at least one first route segment and a further or second route segment. The route can be segmented into a number of segments, the segment length being determined based on an identified vehicle speed, an identified vehicle location, road characteristics, and road conditions. In this exemplary embodiment, the first and second segments can be separated by an additional plurality of segments, the number of which can be determined in response to an identified vehicle speed, an identified vehicle location, road characteristics, and road conditions, thus providing a sufficient time interval between a warning of an anomalous event and the safe resumption of driving by a driver.

[0049] Next, procedure 630 generates an initial motion path for the first route segment and guides the identified vehicle along this segment. The initial motion path is generated by an ADAS algorithm and is the path along which the identified vehicle is guided through the first route segment. This initial motion path is generated based on the current identified location, the destination, the detection of nearby objects, map data, and similar information.

[0050] Next, procedure 640 generates a different or predicted motion path for the next route segment and simulates a simulated identified vehicle operation over the predicted route segment. The procedure is then able to predict an anomaly event in response to the simulated identified vehicle operation over the second route segment.

[0051] The procedure then provides a driver alert (650) indicating the anomalous event while the identified vehicle is being driven along the first section of the route. The driver alert can indicate the location of the anomalous event and / or its probability. Predicting the anomalous event can be done by determining its probability and comparing it to a threshold value; the probability exceeding the threshold value is the best prediction.

[0052] The method can further include receiving updated or continuous anomaly event data indicating a previous anomaly event within the predicted route segment, and the prediction of the anomaly event in response to the previous anomaly event, the identified vehicle location, and an identified vehicle speed. The event data can be a simulation model for predicting an anomaly event, the model being generated in response to crowdsourced ADAS operational state transitions compiled from a multitude of vehicles. In an exemplary embodiment, the anomaly event can be predicted in response to a machine learning model and the identified vehicle location and speed.In another exemplary embodiment, the anomalous event is predicted in response to the machine learning model, which was generated in response to a multitude of previous anomalous events within the second route segment. The prediction of the anomalous event can further be performed in response to map data, the identified vehicle location, and an identified vehicle speed.

Claims

[1] Device comprising: a processor (315) in an edge server (310) that communicates with one or more vehicles (10, 90) in a fleet, to: Communicating and maintaining a continuous communication link between one or more vehicles (10, 90) of the fleet to receive a multitude of messages, wherein the multitude of messages comprises a continuous stream of message data containing state transition data of electronic control units, ECUs, contained in each vehicle (10, 90) in a fleet of vehicles (10, 90) while performing vehicle operations in an environment; Monitoring the streamed message data from each vehicle (10, 90) in real time to detect updates of data about state transitions of ECUs used in vehicle operation, with the data of each state transition of the ECUs being reported to the edge server (310) by an advanced driver assistance system, ADAS, included in each vehicle (10, 90); Crowdsource, in real time, a set of data of the state transitions of ECUs or of different operating levels of ECUs during the vehicle operation of each vehicle (10, 90) in communication with the edge server (310), to aggregate state transition data with a certain degree of commonality as crowdsourced transition data; Processing, by applying a machine learning model, the crowdsource transition data to classify crowdsource transition data that indicates an anomaly event of an assisted driving system; and Sending, in advance, a message to an identified vehicle where the assisted driving system anomaly event is likely to occur, so that a driver or the ADAS can take action. [2] Device according to claim 1, further comprising: the processor (315) on the edge server (310) in communication with the one or more vehicles (10, 90) in a fleet, which is also operational, to: Simulating an algorithm for an assisted driving system over a predicted route segment for an identified vehicle to generate a simulation result; and Prediction of a predicted anomaly event of the assisted driving system within the predicted route segment based on an analysis of the crowdsourced data and the simulation result. [3] Device according to claim 2, further comprising: the processor (315) on the edge server (310) in communication with the one or more vehicles (10, 90) in a fleet, which is also operational, to: Sending a user alert about the predicted assisted driving system anomaly event before the identified vehicle triggers the predicted assisted driving system anomaly event while performing vehicle operations. [4] Device according to claim 3, further comprising: the processor (315) on the edge server (310) in communication with the one or more vehicles (10, 90) in a fleet, which is also operational, to: Sending a warning message in response to the predicted anomaly event of the assisted driving system to vehicles in the vicinity. [5] Device according to claim 4, further comprising: the processor (315) on the edge server (310) in communication with the one or more vehicles (10, 90) in a fleet, which is also operational, to: Monitoring one or more key parameters relating to the vehicle operation of each vehicle (10, 90) in the fleet. [6] Device according to claim 5, wherein the identified vehicle that is likely to cause the assisted driving system anomaly event is driving behind a vehicle that has already caused the assisted driving system anomaly event. [7] Device according to claim 6, further comprising: the processor (315) on the edge server (310) in communication with the one or more vehicles (10, 90) in a fleet, which is also operational, to: Receiving transition state data indicating a previous assisted driving system anomaly event of a vehicle (10, 90) of the fleet traveling on a nearby track segment, and wherein the assisted driving system anomaly event is predicted based on or in response to the previous assisted driving system anomaly event. [8] Device according to claim 7, wherein the message data are transmitted via an MQ Telemetry Transport, MQTT, and a Data Distribution Service, DDS, wireless protocol in a cellular network (20). [9] Device according to claim 8, further comprising: the processor (315) on the edge server (310) in communication with the one or more vehicles (10, 90) in a fleet, which is also operational, to: Sending the message about the anomaly event of the assisted driving system via a reverse path to a messaging client on the vehicle (10, 90) to transmit it on a Controller Area Network, CAN, bus to an ADAS controller of the vehicle (10, 90) to perform a control action on an ECU contained in the vehicle (10, 90). [10] Method (400) executed by a processor, comprising: Communicating a large number of messages between a processor (315) at an edge server (310) and one or more vehicles (10, 90) in a fleet; Maintaining a continuous cellular connection between the processor (315) at the edge server (310) and the one or more vehicles (10, 90) of a fleet for receiving the majority of messages, wherein the majority of messages comprise a continuous stream of message data containing state transition data of ECUs contained in each vehicle (10, 90) in a vehicle fleet during vehicle operation; Monitoring, in real time, the streamed message data from each vehicle (10, 90) to capture updates of data about ECU state transitions used in vehicle operation, wherein the data of each state transition of the ECUs is reported to the edge server (310) by an advanced driver assistance system, ADAS, included in each vehicle (10, 90); Crowdsourced, in real time, a set of data on state transitions or different levels of operations of ECUs during the vehicle operation of each vehicle (10, 90) in communication with the edge server (310) to aggregate state transition data with a degree of commonality as crowdsource transition data; Processing, by applying a machine learning model, the crowdsource transition data to classify crowdsource transition data that indicates an anomaly event of an assisted driving system; and Sending, in advance, a message to an identified vehicle where the assisted driving system anomaly event is likely to occur, so that a driver or the ADAS can take a vehicle control action.

Images