Method and system for monitoring driving experience of automatic driving system of vehicle

Abstract

Methods and systems for monitoring driving experience of an automated driving system of a vehicle are disclosed. The method includes receiving, by an ADS monitoring layer, a query that monitors an indication of occurrence of an event message trigger condition or an event record trigger condition. In response to the fulfillment of the event message trigger condition, a first trigger signal is transmitted to the ADS, an event message dataset is stored, the event message dataset including metadata associated with an event causing the fulfillment of the event message trigger condition, and the event message dataset is transmitted to a remote server. In response to the fulfillment of the event recording trigger condition, a second trigger signal is transmitted to the ADS, an event recording dataset is stored, the event recording dataset including perception data associated with events that cause the fulfillment of the event recording trigger condition and / or ADS output data associated with events that cause the fulfillment of the event recording trigger condition, the ADS output data including perception data associated with events that cause the fulfillment of the event recording trigger condition and / or ADS output data associated with events that cause the fulfillment of the event recording trigger condition. The event record dataset is transmitted to a remote server. The event record dataset is larger than the event message dataset.

Description

Technical Field

[0001] The disclosed technology relates to methods and systems for monitoring the driving experience of an automated driving system (ADS) of a vehicle and other related aspects. Specifically, but not exclusively, the disclosed technology relates to an architecture for collecting data from a fleet of vehicles equipped with an ADS, wherein the data is separated into several streams to enable efficient collection of relevant data sets without compromising understanding from the fleet. Background Art

[0002] Vehicles equipped with advanced driver assistance systems (ADAS) and / or automated driving (AD) systems, collectively referred to herein as automated driving systems (ADS), generate a large amount of sensor-related data while driving. Data from real-world scenarios is critical in data-driven development organizations where ADS performance scales with the amount of data collected. Current ADS solutions typically rely on data servers containing sensor data from a small number of dedicated test vehicles collected within a limited time frame. As a result, these data collections contain very limited scenarios and it is difficult to maintain adequate performance over time as ADS features are continuously developed and updated via over-the-air (OTA) updates. Therefore, it is foreseeable that the next generation of ADS solutions will leverage data collected from production fleets over time.

[0003] However, accessing data from an entire fleet (i.e., production vehicles or customer vehicles) and transferring the data to a central server is very costly and technically challenging due to storage limitations, bandwidth restrictions, and the huge post-processing requirements to extract the relevant datasets for each individual use case. Summary of the invention

[0004] The technology disclosed herein is intended to mitigate, alleviate or eliminate one or more of the above-mentioned defects and shortcomings in the prior art to address various problems related to technical constraints (bandwidth limitations, storage limitations, etc.) for collecting data from a fleet of vehicles equipped with ADS to estimate and improve the performance of ADS.

[0005] Various aspects and embodiments of the disclosed invention are defined in the following and accompanying independent and dependent claims.

[0006] The first aspect of the disclosed technology includes a computer-implemented method for monitoring the driving experience of an automated driving system (ADS) of a vehicle. The method includes receiving, by an ADS monitoring layer communicating with an ADS of the vehicle and running on a computing device of the vehicle, a query for an indication of monitoring the occurrence of one or more event message trigger conditions and the occurrence of one or more event record trigger conditions by monitoring ADS data. In addition, in response to the satisfaction of one or more event message trigger conditions, the method includes transmitting a first trigger signal by the ADS monitoring layer to the ADS, storing an event message data set at a data storage of the ADS, the event message data set including metadata associated with an event that caused the satisfaction of one or more event message trigger conditions, and transmitting the event message data set to one or more remote servers. In addition, in response to the satisfaction of one or more event record trigger conditions, the method includes transmitting a second trigger signal by the ADS monitoring layer to the ADS, storing an event record data set at a data storage of the ADS, the event record data set including sensory data associated with an event that caused the satisfaction of one or more event record trigger conditions and / or ADS output data associated with an event that caused the satisfaction of one or more event record trigger conditions, and transmitting the event record data set to one or more remote servers. Here, the event record data set is larger than the event message data set.

[0007] A second aspect of the disclosed technology includes a computer program product including instructions that, when executed by a computing device of a vehicle, cause the computing device to perform a method according to any one of the embodiments of the first aspect disclosed herein. For this aspect of the disclosed technology, there are similar advantages and preferred features to the other aspects.

[0008] A third aspect of the disclosed technology includes a (non-transitory) computer-readable storage medium including instructions that, when executed by a computing device of a vehicle, cause the computing device to perform a method according to any one of the embodiments of the first aspect disclosed herein. For this aspect of the disclosed technology, there are similar advantages and preferred features as the other aspects.

[0009] The term "non-transitory", as used herein, is intended to describe computer-readable storage media (or "memory") that do not include propagating electromagnetic signals, but is not intended to otherwise limit the type of physical computer-readable storage device encompassed by the terms computer-readable medium or memory. For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in non-transitory form on tangible computer-accessible storage media may also be transmitted via transmission media or signals (e.g., electrical, electromagnetic, or digital signals), which may be transmitted via communications media (e.g., networks and / or wireless links). Therefore, the term "non-transitory", as used herein, is a limitation on the medium itself (i.e., tangible, rather than signals), rather than a limitation on the persistence of data storage (e.g., RAM or ROM).

[0010] A fourth aspect of the disclosed technology includes a system for monitoring the driving experience of an automated driving system (ADS) of a vehicle. The system includes one or more memory storage areas containing program code, and the one or more memory storage areas and program code are configured to, using one or more processors, enable the system to receive at least a query for an indication of the occurrence of one or more event message trigger conditions and the occurrence of one or more event record trigger conditions by monitoring ADS data through an ADS monitoring layer that communicates with the ADS of the vehicle and runs on a computing device of the vehicle. In addition, in response to the satisfaction of one or more event message trigger conditions, the ADS monitoring layer causes the system to transmit a first trigger signal to the ADS, store an event message data set including metadata associated with an event that caused the satisfaction of one or more event trigger conditions at a data storage device of the ADS, and transmit the event message data set to one or more remote servers. In addition, in response to the satisfaction of one or more event record trigger conditions, the system transmits a second trigger signal to the ADS through the ADS monitoring layer, stores an event record data set at a data storage of the ADS, the event record data set including sensory data associated with the event that caused the satisfaction of the one or more event record trigger conditions and / or ADS output data associated with the event that caused the satisfaction of the one or more event record trigger conditions, and transmits the event record data set to one or more remote servers. Here, the event record data set is larger than the event message data set. For this aspect of the disclosed technology, there are similar advantages and preferred features as other aspects.

[0011] A fifth aspect of the disclosed technology includes a vehicle including an automated driving system (ADS) and a system according to any one of the embodiments disclosed herein. For this aspect of the disclosed technology, there are similar advantages and preferred features as the other aspects.

[0012] The disclosed aspects and preferred embodiments may be appropriately combined with each other in any manner obvious to a person skilled in the art, and thus one or more features or embodiments related to one aspect may also be considered to be related to another aspect or embodiment of another aspect.

[0013] An advantage of some embodiments is that collecting data from vehicles equipped with an ADS is more efficient and cost-effective.

[0014] An advantage of some embodiments is that an ADS-equipped fleet of vehicles can be understood through efficient use of processing, storage, and transmission resources.

[0015] An advantage of some embodiments is that it is no longer necessary to rely on dedicated test vehicles to understand how well an ADS works, as it is possible to collect data from a large fleet (ie, a production fleet).

[0016] An advantage of some embodiments is that data collection can be customized in a rapid manner to meet specific needs depending on the geographic scope of the fleet or the current data needs, thereby providing a more versatile tool for collecting large amounts of relevant data from ADS-equipped vehicles.

[0017] An advantage of some embodiments is that more efficient and secure ADS functionality may be released faster than prior known solutions.

[0018] Further embodiments are defined in the dependent claims. It should be emphasized that the term "comprises / comprising" when used in this description is used to specify the presence of stated features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or combinations thereof.

[0019] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above-described aspects, features, and advantages of the disclosed technology will be more fully understood by referring to the following illustrative and non-limiting detailed description of example embodiments of the invention, in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a schematic flow chart representation of a method for monitoring the driving experience of an automated driving system (ADS) of a vehicle, according to some embodiments.

[0022] Figure 2 is a schematic block diagram representation of a system for monitoring the driving experience of an automated driving system (ADS) of a vehicle, according to some embodiments.

[0023] Figure 3is a schematic diagram of a vehicle including a system for monitoring the driving experience of an automated driving system (ADS) of the vehicle, according to some embodiments. DETAILED DESCRIPTION

[0024] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be implemented in other forms and should not be construed as being limited to the disclosed example embodiments. The disclosed example embodiments are provided in order to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will appreciate that the steps, services, and functions explained herein may be implemented using separate hardware circuits, using software running in conjunction with a programmed microprocessor or general-purpose computer, using one or more application specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0025] It should also be understood that when the present disclosure is described in terms of a method, it may also be implemented in an apparatus comprising one or more processors, one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein.

[0026] It should also be understood that the terms used herein are only used to describe specific embodiments and are not intended to be limiting. It should be noted that, as used in the specification and the appended claims, the articles "one", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly indicates otherwise. Therefore, for example, in some contexts, mentioning "a unit" or "the unit" may refer to more than one unit, etc. In addition, the words "include", "include" do not exclude other elements or steps. It should be emphasized that the term "include / include" is used to specify the presence of stated features, integral bodies, steps or parts when used in this description. It does not exclude the presence or addition of one or more other features, integral bodies, steps, parts or their combinations. The term "and / or" should be interpreted as indicating "both" and each is optional. Similarly, at least one of the terms A, B and C will be interpreted as covering conjunction and disjunction realization, meaning that at least one of A, B and C will be understood as A and / or B and / or C, i.e., only A, only B, only C, only A and B, only A and C, or only B and C.

[0027] It should also be understood that, although the terms first, second, etc. can be used to describe various elements or features herein, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the embodiment, the first signal can be referred to as the second signal, and similarly, the second signal can be referred to as the first signal. Both the first signal and the second signal are signals, but they are not the same signal.

[0028] The term "Automated Driving System" (ADS) refers to a complex combination of various components and can be defined as a system in which electronic devices and machines replace human drivers for vehicle perception, decision-making and operation, as well as a system that introduces automation into road traffic. Therefore, the term "ADS" is used here as a general term for advanced driver assistance systems (ADAS) and automated driving (AD). According to the SAE J3016 Highway Vehicle Driving Automation Levels, ADS preferably includes ADS features of Level 3 or higher. ADS features can be, for example, traffic jam navigation, highway navigation or any other SAE J3016 Level 3+ ADS features.

[0029] Figure 1 is a schematic flow chart representation of a computer-implemented method S100 for monitoring driving experience of an ADS of a vehicle according to some embodiments. The method S100 is preferably a computer-implemented method S100 performed by a processing system of a vehicle equipped with an ADS. The processing system may, for example, include one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that, when executed by the one or more processors, cause the processing system to perform the steps, services, and functions of the method S100 disclosed herein.

[0030] Method S100 includes receiving S101 a query for an indication of the occurrence of one or more event message trigger conditions and the occurrence of one or more event record trigger conditions by monitoring S102 ADS data through an ADS monitoring layer that communicates with the vehicle's ADS and runs on a computing device of the vehicle. In some embodiments, the ADS data includes (a) ADS input data, (b) ADS internal data, and (c) ADS output data. In some embodiments, the vehicle's ADS includes safety-certified software, and the ADS monitoring layer is an uncertified software layer separated from the safety-certified software of the ADS. In some embodiments, the ADS monitoring layer is a software layer separated from the ADS.

[0031] In general, ADS software is required to comply with applicable standards, such as ISO standards, such as ISO 26262 and / or safety certification of the United Nations automotive industry regulations. Therefore, by separating the ADS monitoring layer from the (safety-certified) ADS software, updates related to the ADS monitoring layer can be pushed without undergoing troublesome, time-consuming and expensive updates associated with updating the ADS software. In other words, by separating the ADS monitoring layer from the (safety-certified) ADS software, ADS data analysts can query the fleet of ADS vehicles for specific events and then receive relevant data with lower latency, compared to having to wait for the next software version of ADS to add the function of monitoring specific events and then transmit relevant data. In addition, the ADS monitoring layer is preferably isolated from the ADS software. In other words, the ADS monitoring layer cannot change or delete any software of the ADS, and cannot control the ADS (e.g., causing the activation or deactivation of any ADS function or actuator of the vehicle).

[0032] In addition, in response to the satisfaction of one or more event message trigger conditions, the method S100 includes transmitting S103 a first trigger signal by the ADS monitoring layer to the ADS. The first trigger signal can, for example, be transmitted to the ADS log unit / module / system of the ADS. Then, an event message data set including metadata associated with the event that caused the satisfaction of one or more event message trigger conditions is stored S104 at a data storage device of the ADS (or ADS log system). In addition, the stored event message data set is transmitted S105 to one or more remote servers. The event message data set can be encrypted and / or compressed before storage S104 or before transmission S105.

[0033] In addition, in response to the satisfaction of one or more event record trigger conditions, method S100 includes transmitting S106 second trigger signal to ADS by ADS monitoring layer. Similarly, as mentioned above, the second trigger signal can be transmitted to ADS log system, for example. Then, the event record data set including the perception data associated with the event that causes the satisfaction of one or more event record trigger conditions and / or the ADS output data associated with the event that causes the satisfaction of one or more event record trigger conditions is stored S107 at the data storage device of ADS (or ADS log system). In addition, the stored event record data set is transmitted S108 to one or more remote servers. Here, the event record data set is larger than the event message data set. The event record data set can be, for example, several orders of magnitude larger than the event message data set (for example, 103 to 108 times larger). The event record data set can be encrypted and / or compressed before storage S107 or before transmission S108. In some embodiments, the event record data set can include ADS input data, ADS internal data and / or ADS output data.

[0034] The term "perception data" refers to information collected by sensors and other technologies that an ADS-equipped vehicle uses to detect and interpret its environment. This includes data collected from cameras, lidar, radar, and other sensors that help the vehicle "sense" its surroundings and make decisions based on that information. Perception data collected by a vehicle may include the location, speed, and direction of nearby objects, the location and type of road markings, the location and type of traffic signs, and other relevant information. This data may then be processed by the vehicle's onboard computer to help it make decisions about steering, acceleration, braking, and other actions necessary to safely navigate the environment. Thus, the term "perception" data may refer to "environment estimation" data, "spatial perception" data, "processed sensing" data, and / or "temporal correlation" data, while perception "data" may refer to perception "information" and / or "estimates." On the other hand, the term "obtained" from a perception module or perception system may refer to "derived" from a perception model and / or "based on output data" from a perception module or system. A perception block / system configured to "generate a perception data set" may refer to a perception module / system adapted and / or configured to "estimate the surroundings of the vehicle", "estimate at least a portion of the surroundings of the vehicle", "determine the surroundings of the vehicle", "interpret sensor information related to autonomous manipulation of the vehicle" and / or "estimate the surroundings of the vehicle and perform model predictions on future states of the surroundings of the vehicle".

[0035] In this context, a "query" may be interpreted as a signal comprising a request for information about one or more events from a remote entity (e.g., a remote server). In other words, receiving S101 a query may include receiving a signal from a remote server, wherein the signal comprises a request for information about one or more events. Information is then provided based on one or more event message trigger conditions and / or one or more event trigger conditions that are satisfied. Therefore, receiving S101's query instructs the ADS monitoring layer to monitor ADS data to determine whether one or more event message trigger conditions and / or one or more event trigger conditions are satisfied. In some embodiments, receiving S101's query instructs the ADS monitoring layer to monitor ADS input data, ADS internal data, and ADS output data to determine whether one or more event message trigger conditions and / or one or more event trigger conditions are satisfied.

[0036] The partitioning or separation of the data set into an "event message data set" and an "event record data set" provides an effective solution for monitoring the driving experience of a fleet equipped with an ADS, because it can collect high priority data with low bandwidth requirements (event message data set) and also collect low priority data with high bandwidth requirements (event record data set). As a result, data related to driving experience events can be accessed faster and, by extension, can provide a safer and higher performance ADS at a higher speed than other known solutions.

[0037] The metadata included in the event message data set may include, for example, the geographic location of the event that resulted in the satisfaction of one or more event trigger conditions, the timestamp of the event that resulted in the satisfaction of one or more event trigger conditions, the classification of the event that resulted in the satisfaction of one or more event trigger conditions, ADS hardware information, ADS software information, weather information, vehicle information, ADS processing load value, ADS RAM usage value, ADS perception output confidence, ADS road model output confidence. On the other hand, the event record data set may include ADS input data (e.g., sensor output data), ADS internal data (e.g., data transmitted between various modules or functions of the ADS, such as road models, target paths and / or target trajectories, etc.), and / or ADS output data (e.g., control signals to vehicle actuators).

[0038] As used herein, the term "in response to" may be understood as "when" or "at" or "if", depending on the context. Similarly, the phrase "in response to the satisfaction of a condition" may be interpreted to mean "when it is determined that the condition is satisfied" or "in the event of satisfaction of the condition", "upon determining the satisfaction of the condition" or "upon detecting and identifying the occurrence of an event that satisfies the condition" or "in response to detecting the occurrence of an event that satisfies the condition", depending on the context.

[0039] As a result, event record datasets are typically significantly larger than event message datasets. Due to the larger size, it is not always feasible to transmit event record datasets from vehicles. However, due to the separation of data streams, it is still cost-effective to collect data from the fleet to achieve fleet understanding. In more detail, the transmission and collection of fleet data is often subject to compliance with local rules and regulations (data privacy, data security, and data access rights) and / or bandwidth availability, which may limit the amount and / or type of data that can be transmitted and collected. However, with the help of the architecture proposed in this paper, valuable fleet understanding can still be achieved due to the generally non-sensitive nature of the data contained in the event message dataset defined in this paper.

[0040] In some embodiments, method S100 further includes generating S109 a unique identifier for an event message data set in response to the satisfaction of one or more event message triggering conditions. In addition, the storage S104 of the event message data set may accordingly include storing the unique identifier generated S109. In addition, in response to the satisfaction of one or more event record triggering conditions, method S100 may include generating S110 a corresponding unique identifier for the event record data set so as to enable pairing of event message data sets and event record data sets associated with the same event. Therefore, the storage S107 of the event record data set may include storing the corresponding unique identifier generated S110. By generating unique identifiers for an event message data set and an event record data set belonging to the same "event" that caused the triggering condition to be triggered, the two data sets can be related later in the database of the remote server, even if they are transmitted and subsequently received at different times.

[0041] Furthermore, in some embodiments, the one or more event message triggering conditions include one or more of the following triggering conditions:

[0042] Detection of autonomous activation or deactivation of an ADS safety function; examples of ADS safety functions include an emergency braking system (EBS), a lane keeping system (LKS), a lane departure warning system (LDWS), a minimum risk maneuvering (MRM) system, and / or any other ADS safety function readily recognized by a person skilled in the art;

[0043] Detection of autonomous activation or deactivation of an ADS driving function; examples of ADS driving functions include adaptive cruise control, automated lane change systems, valet parking systems, traffic jam pilot (TJP), highway pilot, and / or any other “autonomous driving” function;

[0044] Detection of predefined objects in the vehicle's surroundings. Examples of predefined objects include specific animals, specific vehicles, etc.; in addition, one or more event message trigger conditions may include a confidence level of the object detection being above or below a threshold;

[0045] Detection of predefined scenarios in the vehicle's surroundings; examples of predefined scenarios include accident scenarios, near-accident scenarios, specific weather conditions (e.g., snow or rain), specific road conditions (e.g., slippery and / or wet), the presence of dangerous road users (e.g., speeding vehicles, turning vehicles, etc.), and so on;

[0046] Detection of a difference between the map data and the sensor data; examples of a difference between the map data and the sensor data include if the map data indicates a first road geometry while the sensor data indicates a different road geometry, or if the map data indicates the presence of a landmark while the sensor data does not indicate the presence of a landmark, etc.;

[0047] Detection of error messages from ADS; examples of error messages from ADS include error reports from ADS, entries in error logs, etc.;

[0048] Detection of whether the processing load of the ADS meets a threshold; examples of the processing load of the ADS meeting the threshold include whether the CPU of the ADS reaches a processing load above 80%, 90%, 95% or 99%;

[0049] Detection of random access memory (RAM) usage meeting a threshold; examples of RAM usage meeting a threshold include whether the RAM usage of the ADS reaches a load of 80%, 90%, 95%, or 99% or more;

[0050] · Detection of one or more message drops in the ADS.

[0051] In some embodiments, the event record trigger condition includes a subset of the event message trigger condition. In more detail, it is conceivable that the event message data set will be sufficient for subsequent event analysis of a large number of detectable events experienced by the vehicle's ADS, so it will be inefficient or even infeasible to collect an event record data set for all possible triggering events experienced by the vehicle's ADS. Therefore, in some cases, an event or scene will satisfy the event message trigger condition and satisfy the event record trigger condition. However, in some cases, an event or scene will only satisfy the event message trigger condition, but not any event record trigger condition.

[0052] In addition, in some embodiments, one or more event message trigger conditions include one or more dynamically configurable event message trigger conditions, and / or one or more event record trigger conditions include one or more dynamically configurable event record conditions. In other words, the event message trigger conditions and the event record trigger conditions do not have to be static, but can be changed, new trigger conditions can be added, and / or existing trigger conditions can be removed, for example, through an over-the-air (OTA) update broadcast to multiple vehicles. As previously described, since the ADS monitoring layer can be separated (and isolated) from the (safety-certified) ADS, a higher degree of freedom can be achieved to deploy logic for event detection in the vehicle.

[0053] In some embodiments, the query further includes updating data / information, including one or more new or updated event message conditions and / or one or more new or updated event record conditions. Method S100 may further include: based on the updated data / information, updating S111 one or more dynamically configurable event message trigger conditions and / or one or more dynamically configurable event record trigger conditions. In this article, the step of updating S111 may include adding new event message trigger conditions and / or event record trigger conditions, changing existing event message trigger conditions and / or event record trigger conditions, and / or removing / deleting existing event message trigger conditions and / or event record trigger conditions.

[0054] In addition, in some embodiments, when the communication link established with the one or more remote servers meets the first set of link quality standards, the generated event message data set is transmitted S105 to the one or more remote servers, and when the communication link established with the one or more remote servers meets the second set of link quality standards, the generated event record data set is transmitted S108 to the one or more remote servers. Here, the second set of link quality standards may include a higher bandwidth requirement than the first set of link quality standards. However, additionally or alternatively, the first set of link quality standards includes a greater number of allowable communication link types than the second set of link quality standards. For example, the first set of link quality standards may allow the event message data set to be allowed to be transmitted S105 on a cellular network link (e.g., 4G, 5G, etc.) or via a Wi-Fi network link, while the second set of link quality standards may allow the event record data set to be transmitted only via a Wi-Fi network link S108. In other words, the first set of link quality standards may include a first set of allowable communication protocols, and the second set of link quality standards may include a second set of allowable communication protocols, wherein the second set of allowable communication protocols is less than the first set of allowable communication protocols.

[0055] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product, configured to be executed by one or more processors.

[0056] Figure 2 is a schematic block diagram of a system 10 for monitoring the driving experience of an automated driving system (ADS) of a vehicle according to some embodiments. In more detail, Figure 2 The system 10 is used to facilitate the above-described embodiments related to monitoring the driving experience by schematically illustrating the flow of information and various processing steps. The system 10 includes a control circuit (e.g., one or more processors) configured to perform the functions of the method S100 disclosed herein, wherein these functions may be included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by the control circuit. However, in order to better illustrate the embodiments, the control circuit is not Figure 2 The system is represented as various modules or blocks, each of which is linked to one or more specific functions of the system.

[0057] The system 10 includes one or more memory storage areas containing program codes, and the one or more memory storage areas and program codes are configured to use one or more processors to enable the system 10 to perform the method S100 according to any embodiment disclosed herein. Specifically, the one or more memory storage areas and program codes are configured to use one or more processors to enable the system 10 to receive a query for an indication of the occurrence of one or more event message trigger conditions and the occurrence of one or more event record trigger conditions by monitoring ADS data through an ADS monitoring layer 201 that communicates with the vehicle's ADS 310 and runs on a computing device of the vehicle 10. In some embodiments, the ADS data includes (a) ADS input data 251, (b) ADS internal data 252, and (c) ADS output data 253. The query can be transmitted, for example, from a remote server, wherein the operator can instruct the query engine 401 of the remote server 20 to transmit the query to one or more vehicles via the input device 30.

[0058] In some embodiments, the ADS input data 251 includes the output of the vehicle sensor system 320. The output of the sensor system 320 may include sensor output (e.g., camera output, radar output, lidar output, etc.) and / or post-processed sensor output (e.g., object-level data such as object classification, object detection, etc.). In some embodiments, the ADS internal data 252 includes information or data transmitted from one ADS function 321 to another ADS function 321, such as, for example, a road model, lane trajectory, etc. In some embodiments, the ADS output data 253 includes information or data transmitted from the ADS software stack 310 to the vehicle's operating system 328, such as, for example, control signals transmitted to the vehicle's actuators (e.g., steering actuators, throttle actuators, and brake actuators). In addition, in some embodiments, the ADS internal data 252 or the ADS output data 253 includes error messages, the processing load of the ADS, and the RAM usage (or memory usage) of the ADS.

[0059] Thus, the ADS monitoring layer 201 may include an event detector 202 configured to detect and subscribe to ADS data, such as, for example, ADS input data 251, ADS internal data 252, and ADS output data 253, to determine whether an "event of interest" has occurred. As described above, an "event of interest" may include events internal to the ADS (e.g., errors, activation or deactivation of ADS functions, processing load, memory usage, etc.) and / or events external to the ADS that occur in the vehicle's surroundings. The vehicle's surroundings may be understood as the general area around the vehicle, where objects (such as other vehicles, landmarks, obstacles, etc.) may be detected and identified by the vehicle's sensors (radar, lidar, cameras, etc.), i.e., within the vehicle's sensor range.

[0060] In addition, in response to the satisfaction of one or more event message trigger conditions, the one or more memory storage areas and program code are configured to, using one or more processors, cause the system 10 to transmit a first trigger signal to, for example, an ADS log system 205 of the ADS 310 via the ADS monitoring layer 201. The system 10 is also caused to store an event message data set including metadata associated with the event that caused the satisfaction of the one or more event trigger conditions at a data storage 209 of the ADS log system 205, and transmit the event message data set to one or more remote servers 20. The event message data set may be transmitted to the remote server 20, for example, using a communication system 326 including one or more transceivers and antennas.

[0061] In addition, in response to the satisfaction of one or more event record trigger conditions, the one or more memory storage areas and program code are configured to, using one or more processors, cause the system 10 to transmit a second trigger signal to, for example, the ADS log system 205 of the ADS through the ADS monitoring layer 201. The system 10 is also caused to store an event record data set at a data storage of the ADS log system of the ADS, the event record data set including the sensed data 251 associated with the event that caused the satisfaction of the one or more event trigger conditions and / or the ADS output data 253 associated with the event that caused the satisfaction of the one or more event trigger conditions, and transmit the event record data set to one or more remote servers 326. As previously described, the event record data set is larger than the event message data set.

[0062] The event message data set and the event record data set can be stored in the message data buffer 207 and the record data buffer 208, respectively, for example. In some embodiments, when the vehicle is running, the event message data set and the event record data set (or different types of data constituting different data sets) are continuously stored in the data buffers 207 and 208. Then, once the ADS log system 205 receives the corresponding trigger signal from the ADS monitoring layer 201, the data contained in the corresponding data buffers 207 and 208 are persisted in the permanent storage unit 209 of the ADS log system 205. The data set persisted in the permanent storage unit 209 can be encrypted and compressed, for example, before being persisted / stored. In addition, once the data sets are transmitted to the remote server 20, they can be deleted from the permanent storage unit 209.

[0063] The length of the data buffers 207, 208 may be any arbitrary size deemed feasible, for example, taking into account data capacity limitations and / or the characteristics of the ADS, and may, for example, range from less than a second to several minutes, or more preferably, from a few seconds to less than a minute. An advantage of utilizing the data buffers 207, 208 is that the entire temporal development of an "event of interest" may be captured, persisted, and subsequently transmitted to the remote server 20. The data buffers 207, 208 may be in the form of a first-in-first-out (FIFO) buffer (e.g., in the form of a fast access memory such as a random access memory RAM).

[0064] Thus, the event message data buffer 207 subscribes to (i.e., monitors and stores) a limited set of ADS data (e.g., ADS input data 251, ADS internal data 252, and / or ADS output data 253) so that a limited amount of historical data is available when an event that triggers an event message trigger condition occurs. Similarly, the event record data buffer 208 subscribes to (i.e., monitors and stores) a larger amount of ADS data (e.g., ADS input data 251, ADS internal data 252, and / or ADS output data 253) than the event message data buffer 207 so that a limited amount of historical data is available when an event that triggers an event record trigger condition occurs.

[0065] In addition, according to some embodiments, in response to the satisfaction of one or more event message triggering conditions, one or more memory storage areas and program codes are configured to use one or more processors to cause the system 10 to generate a unique identifier for the event message data set. Here, storing the event message data set includes storing the generated unique identifier. In addition, in response to the satisfaction of one or more event record triggering conditions, one or more memory storage areas and program codes are configured to use one or more processors to cause the system 10 to generate a corresponding unique identifier for the event record data set, thereby achieving pairing of the event message data set and the event record data set associated with the same event. Here, storing the event record data set includes storing the generated corresponding unique identifier.

[0066] In some embodiments, one or more event message trigger conditions include one or more dynamically configurable event message trigger conditions, and / or one or more event record trigger conditions include one or more dynamically configurable event record conditions. In addition, in some embodiments, the query further includes update data, and the update data includes one or more new or updated event message conditions and / or one or more new or updated event record conditions. Therefore, one or more memory storage areas and program codes can be configured to utilize one or more processors to enable the system 10 to update one or more dynamically configurable event message trigger conditions and / or one or more dynamically configurable event record trigger conditions based on the received update data. Therefore, the query engine 401 may include an event tuner module 402, through which an operator can add, change or remove trigger conditions. In other words, "events of interest" can be dynamically configurable.

[0067] In addition, in some embodiments, when the communication link established with the one or more remote servers 20 meets the first set of link quality standards, the generated event message data set is transmitted (e.g., via the communication system 326) to the one or more remote servers 20. Similarly, when the communication link established with the one or more remote servers meets the second set of link quality standards, the generated event record data set can be transmitted (e.g., via the communication system 326) to the one or more remote servers 20. Here, the second set of link quality standards can include higher bandwidth requirements than the first set of link quality standards. In other words, the event message data set can be transmitted over a communication link with a lower bandwidth capacity than the event record data set.

[0068] Figure 3 Schematic diagram of a vehicle 1 equipped with an ADS including such a system 10. As used herein, a "vehicle" is any form of motorized transport. For example, the vehicle 1 may be any road vehicle, such as a car (as shown herein), a motorcycle, a (freight) truck, a bus, etc.

[0069] The system 10 includes a control circuit 11 and a memory 12. The control circuit 11 may physically include a single circuit device. Alternatively, the control circuit 11 may be distributed over several circuit devices. As an example, the device 10 may share its control circuit 11 with other parts of the vehicle 1 (e.g., the ADS 310). The control circuit 11 may include one or more processors, such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 12 so as to perform various functions and operations of the vehicle 1 in addition to the methods disclosed herein. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 12. The memory 12 may optionally include a high-speed random access memory, such as a DRAM, SRAM, DDR RAM, or other random access solid-state storage device, and may optionally include a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 12 may include a database component, an object code component, a script component, or any other type of information structure for supporting various activities described herein.

[0070] In the example shown, the memory 12 also stores map data 308. The map data 308 can be used, for example, by the ADS 310 of the vehicle 1 in order to perform autonomous functions of the vehicle 1. The map data 308 may include high-definition (HD) map data. It is conceivable that the memory 12, even if shown as a separate element from the ADS 310, may also be provided as an integrated element of the ADS 310. In other words, according to an exemplary embodiment, any distributed or local storage device may be utilized in the implementation of the inventive concept. Similarly, any distributed or local control circuit device may be utilized in the implementation of the inventive concept. The ADS 310 is configured to perform functions and operations of autonomous or semi-autonomous functions of the vehicle 1. The ADS 310 may include multiple modules, wherein each module is responsible for different functions of the ADS 310.

[0071] The vehicle 1 includes various elements commonly found in autonomous or semi-autonomous vehicles. It should be understood that the vehicle 1 may have Figure 3 Any combination of the various elements shown. In addition, the vehicle 1 may include Figure 3. Although various elements are shown herein as being located inside the vehicle 1, one or more elements may be located outside the vehicle 1. For example, map data may be stored in a remote server and accessed by various components of the vehicle 1 via the communication system 326. In addition, even though various elements are described herein in a particular arrangement, the various elements may be implemented in a different arrangement, as will be readily appreciated by those skilled in the art. It should also be noted that the various elements may be communicatively connected to each other in any suitable manner. Figure 3 The vehicle 1 should be regarded as merely an illustrative example, as elements of the vehicle 1 may be implemented in several different ways.

[0072] The vehicle 1 further comprises a sensor system 320. The sensor system 320 is configured to acquire sensory data about the vehicle itself or its surroundings. The sensor system 320 may, for example, comprise a global navigation satellite system (GNSS) module 322 (e.g., GPS) configured to collect geographic location data of the vehicle 1. The sensor system 320 may further comprise one or more sensors 324. The sensor 324 may be any type of onboard sensor, such as a camera, a lidar and a radar, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, etc. It should be understood that the sensor system 320 may also provide the possibility of acquiring sensory data directly or through a dedicated sensor control circuit in the vehicle 1.

[0073] The vehicle 1 further includes a communication system 326. The communication system 326 is configured to communicate with external units, such as other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), a remote server (e.g., a cloud server), a database, or other external devices, i.e., a vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocol. The communication system 318 can communicate using one or more communication technologies. The communication system 318 may include one or more antennas (not shown). Cellular communication technology can be used for remote communications, such as a remote server or a cloud computing system. In addition, if the cellular communication technology used has low latency, it can also be used for V2V, V2I, or V2X communications. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., also including future cellular solutions. However, in some solutions, medium- and short-range communication technologies, such as wireless local area networks (LANs), such as solutions based on IEEE 802.11, can be used to communicate with other vehicles near the vehicle 1 or with local infrastructure elements. ETSI is developing cellular standards for vehicle communications, and 5G, for example, is seen as a suitable solution due to its low latency and efficient handling of high bandwidth and communication channels.

[0074] The communication system 326 may accordingly provide the possibility to send outputs (e.g. event message data sets and event log data sets) to a remote location (e.g. a remote operator or a control center) and / or receive inputs (e.g. queries) from a remote location by means of one or more antennas. In addition, the communication system 326 may be further configured to allow various elements of the vehicle 1 to communicate with each other. For example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth or similar medium / short range technologies.

[0075] The vehicle 1 further includes a steering system 320. The steering system 328 is configured to control the steering of the vehicle 1. The steering system 328 includes a steering module 330, which is configured to control the heading of the vehicle 1. The steering system 328 further includes a throttle module 332, which is configured to control the actuation of the throttle of the vehicle 1. The steering system 328 further includes a brake module 334, which is configured to control the actuation of the brakes of the vehicle 1. The various modules of the steering system 328 can also receive manual inputs from the driver of the vehicle 1 (i.e., from the steering wheel, the accelerator pedal, and the brake pedal, respectively). However, the steering system 328 can be communicatively connected to the ADS 310 of the vehicle to receive instructions on how the various modules of the steering system 328 should act. Therefore, the ADS 310 can control the steering of the vehicle 1, for example, via the decision and control module 318.

[0076] The ADS 310 may include a positioning module 312 or positioning block / system. The positioning module 312 is configured to determine and / or monitor the geographic location and heading of the vehicle 1 and may utilize data from a sensor system 320, such as data from a GNSS module 322. Alternatively or in combination, the positioning module 312 may utilize data from one or more sensors 324. The positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS for improved accuracy.

[0077] The ADS 310 may further include a perception module 314 or a perception block / system 314. The perception module 314 may refer to any known module and / or function, such as included in one or more electronic control modules and / or nodes of the vehicle 1, adapted and / or configured to interpret sensory data related to driving of the vehicle 1 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. Thus, the perception module 314 may be adapted to rely on and obtain inputs from multiple data sources, such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, etc., in combination with sensory data from, for example, the sensor system 320.

[0078] The positioning module 312 and / or the perception module 314 may be communicatively connected with the sensor system 320 to receive sensing data from the sensor system 320. The positioning module 312 and / or the perception module 314 may also transmit control instructions to the sensor system 320.

[0079] The present invention has been described above with reference to specific embodiments. However, other embodiments than those described above are possible and within the scope of the present invention. Within the scope of the present invention, method steps different from those described above may be provided, the method being performed by hardware or software. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium is provided storing one or more programs configured to be executed by one or more processors of a vehicle control system, the one or more programs comprising instructions for performing a method according to any of the above-described embodiments. Optionally, according to another exemplary embodiment, a cloud computing system may be configured to perform any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that collectively perform the methods presented herein under the control of one or more computer program products.

[0080] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or storage medium, such as electronic, magnetic or optical media, for example a disk or CD / DVD-ROM connected to a computer system via a bus. The terms "tangible" and "non-transitory" as used herein are intended to describe a computer-readable storage medium (or "memory") that does not include propagating electromagnetic signals, but are not intended to limit the type of physical computer-readable storage device encompassed by the term computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may also be transmitted via a transmission medium or signal (e.g., electrical, electromagnetic or digital signal), which may be transmitted via a communication medium (such as a network and / or wireless link).

[0081] The processor 11 (associated with the system 10) may be or may include any number of hardware components for performing data or signal processing, or for executing computer code stored in the memory 12. The device 10 has an associated memory 12, and the memory 12 may be one or more devices for storing data and / or computer code, which are used to complete or implement the various methods described in this description. The memory may include volatile memory or non-volatile memory. The memory 12 may include a database component, an object code component, a script component, or any other type of information structure for supporting the various activities described herein. According to an exemplary embodiment, any distributed or local storage device may be used with the system and method described herein. According to an exemplary embodiment, the memory 12 is communicatively connected to the processor 11 (e.g., via a circuit or any other wired, wireless or network connection) and includes computer code for executing one or more processes described herein.

[0082] It should be noted that any reference signs do not limit the scope of the claims, that the present invention may be implemented at least partially by means of hardware and software, and that several "means" or "units" may be represented by the same item of hardware.

[0083] Although a specific order of method steps may be shown in the figure, the order of the steps may be different from that described. In addition, two or more steps may be performed simultaneously or partially simultaneously. This variation will depend on the selected software and hardware systems and the designer's choice. All these variations are within the scope of the present invention. Similarly, software implementation methods can be completed with standard programming techniques, which have rule-based logic and other logic to complete various receiving steps, sending steps, storage steps, updating steps, and generating steps. The embodiments mentioned and described above are given as examples only and should not limit the present invention. For those skilled in the art, other solutions, uses, goals and functions within the scope of the present invention required by the following patent claims should be obvious.

Claims

1. A computer-implemented method (S100) for monitoring driving experience of an automated driving system (ADS) of a vehicle, the method comprising: receiving ( S101 ), by an ADS monitoring layer in communication with the ADS of the vehicle and operating on a computing device of the vehicle, a query for an indication of the occurrence of one or more event message triggering conditions and the occurrence of one or more event record triggering conditions by monitoring ( S102 ) ADS data; In response to the one or more event message triggering conditions being met: the ADS monitoring layer transmits (S103) a first trigger signal to the ADS; storing (S104) an event message data set at a data storage of the ADS, the event message data set comprising metadata associated with the satisfied event causing the one or more event message triggering conditions; transmitting (S105) the event message data set to one or more remote servers; In response to one or more event record trigger conditions being met: transmitting (S106) a second trigger signal by the ADS monitoring layer to the ADS; storing (S107) an event record data set at the data storage of the ADS, the event record data set comprising sensed data associated with the event causing the satisfaction of the one or more event record triggering conditions and / or ADS output data associated with the event causing the satisfaction of the one or more event record triggering conditions; and transmitting (S108) the event record data set to the one or more remote servers, The event record data set is larger than the event message data set.

2. The method (S100) according to claim 1, further comprising: In response to the one or more event message triggering conditions being satisfied: generating (S109) a unique identifier for the event message data set; wherein storing the event message data set includes storing the generated (S109) unique identifier; In response to the satisfaction of one or more event record trigger conditions: generating (S110) a corresponding unique identifier for the event record data set so as to be able to pair an event message data set and an event record data set associated with the same event; wherein, storing the event record data set includes storing the generated (S110) corresponding unique identifier.

3. The method (S100) according to claim 1, wherein: The one or more event message triggering conditions include detection of autonomous activation or deactivation of an ADS safety function, detection of autonomous activation or deactivation of an ADS driving function, detection of predefined objects in the vehicle's surroundings, detection of predefined scenes in the vehicle's surroundings, detection of differences between map data and sensor data, detection of error messages from the ADS, detection of a processing load of the ADS meeting a threshold, detection of random access memory (RAM) usage meeting a threshold, and / or detection of one or more message discards within the ADS.

4. The method (S100) according to any one of claims 1 to 3, wherein: The one or more event message triggering conditions include one or more dynamically configurable event message triggering conditions, and / or the one or more event record triggering conditions include one or more dynamically configurable event record triggering conditions.

5. The method (S100) according to claim 4, wherein: The query further includes update data, the update data including one or more new or updated event message trigger conditions and / or one or more new or updated event record trigger conditions; The method further comprises: updating (S111) the one or more dynamically configurable event message triggering conditions and / or the one or more dynamically configurable event record triggering conditions based on the update data.

6. The method (S100) according to claim 1, wherein: The metadata includes the geographic location of the event that causes the satisfaction of the one or more event trigger conditions, the timestamp of the event that causes the satisfaction of the one or more event trigger conditions, the classification of the event that causes the satisfaction of the one or more event trigger conditions, ADS hardware information, ADS software information, weather information, vehicle information, processing load value of the ADS, RAM usage value of the ADS, confidence of the perception output of the ADS, and confidence of the road model output of the ADS.

7. The method (S100) according to claim 1, wherein: transmitting the event message data set to the one or more remote servers when the communication link established with the one or more remote servers satisfies a first set of link quality criteria; wherein, when the communication link established with the one or more remote servers meets a second set of link quality criteria, transmitting the event record data set to the one or more remote servers; The second set of link quality standards includes a higher bandwidth requirement than the first set of link quality standards.

8. The method according to claim 1, wherein: The ADS of the vehicle includes safety-certified software, and wherein the ADS monitoring layer is an uncertified software layer separate from the safety-certified software of the ADS.

9. A computer program product having a computer program comprising program instructions thereon, which, when executed by a computing device of a vehicle, causes the computing device to perform the method (S100) according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing instructions, which, when executed by a computing device of a vehicle, cause the computing device to perform the method (S100) according to any one of claims 1 to 8.

11. A system (10) for monitoring the driving experience of an automated driving system (ADS) of a vehicle (1), the system comprising one or more memory storage areas (12), the one or more memory storage areas (12) comprising program code, the one or more memory storage areas and the program code being configured to, using one or more processors (11), cause the system (10) to at least: receiving, by an ADS monitoring layer (201) in communication with the ADS (310) of the vehicle and operating on a computing device of the vehicle (10), a query for monitoring indications of the occurrence of one or more event message trigger conditions and the occurrence of one or more event record trigger conditions by monitoring ADS data (251, 252, 253); In response to the one or more event message trigger conditions being met: the ADS monitoring layer (201) transmits a first trigger signal to the ADS; storing, at a data store of the ADS, an event message data set comprising metadata associated with the event resulting in the satisfaction of the one or more event triggering conditions; transmitting the event message data set to one or more remote servers (20); In response to one or more event record trigger conditions being met: the ADS monitoring layer (201) transmits a second trigger signal to the ADS; storing, at the data store of the ADS, an event record data set comprising sensed data associated with the event resulting in the satisfaction of the one or more event record triggering conditions and / or ADS output data associated with the event resulting in the satisfaction of the one or more event record triggering conditions; transmitting the event record data set to the one or more remote servers; The event record data set is larger than the event message data set.

12. The system (10) according to claim 11, wherein: The one or more memory storage areas and the program code are configured to, with the one or more processors (11), cause the system (10) to at least: In response to the one or more event message triggering conditions being satisfied: generating a unique identifier for the event message data set; wherein storing the event message data set includes storing the generated unique identifier; In response to the satisfaction of the one or more event record trigger conditions: generating a corresponding unique identifier for the event record data set so as to be able to pair an event message data set and an event record data set associated with the same event; wherein storing the event record data set includes storing the generated corresponding unique identifier.

13. The system (10) according to claim 11 or 12, wherein: The one or more event message triggering conditions include one or more dynamically configurable event message triggering conditions, and / or the one or more event record triggering conditions include one or more dynamically configurable event record triggering conditions.

14. The system (10) of claim 13, wherein: The query further includes update data, the update data including one or more new or updated event message conditions and / or one or more new or updated event record conditions; The one or more memory storage areas and program code are configured to utilize the one or more processors (11) to cause the system (10) to at least: Based on the received update data, the one or more dynamically configurable event message triggering conditions and / or the one or more dynamically configurable event record triggering conditions are updated.

15. A vehicle (1) comprising an automated driving system (ADS) and a system (10) according to any one of claims 11 to 14.

Images