Training a neural network to select objects by a computer-implemented method, vehicle

Abstract

A computer-implemented method of training a neural network to select objects, a vehicle is provided. A computer-implemented method of training a neural network to select objects in the vicinity of a target vehicle includes: aggregating (S110) sensor data related to a plurality of vehicles; filtering (S130) the sensor data according to one or more conditions, the conditions identifying actions of ADAS / AD systems of at least one of the plurality of vehicles; identifying (S150) one or more objects in the vicinity of a target vehicle based on the filtered sensor data; and using (S170) the identified one or more objects to train the neural network to determine potential objects that cause triggering of ADAS / AD systems of another vehicle.

Description

Technical Field

[0001] Example aspects herein generally relate to computer-implemented methods for training a neural network to select objects near a target vehicle, and vehicles, cloud servers, and transportation infrastructure units that employ the trained neural network to more efficiently use a vehicle communication network, and corresponding computer programs and computer-readable media. Background Art

[0002] Autonomous driving (AD) functionality, and more generally advanced driver assistance systems (ADAS) functionality, are features of modern vehicles that are attracting increasing interest. ADAS functionality can assist a driver in controlling a vehicle by assisting with driving, parking, and / or other functions of the vehicle. As a specific aspect of ADAS functionality, autonomous driving functionality can allow a driver of a host vehicle (i.e., a vehicle to be controlled) to hand over control of the acceleration and steering of the vehicle to an autonomous driving system that can be provided with, for example, a target speed and heading time or more detailed information to follow a desired route.

[0003] ADAS can be adapted to react appropriately to other road users, pedestrians, possible obstacles (e.g., collisions), and / or environmental conditions (e.g., heavy rain or snow). Therefore, ADAS is typically configured to obtain information from devices such as radar, sensors, cameras, inertial measurement units, etc. to collect data about the vehicle and its environment and generate a high-level environmental model that describes the road the vehicle is traveling on and the traffic on it.

[0004] Vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), car-to-cloud (C2C), or more generally, vehicle-to-everything (V2X) technologies that enable vehicles to be connected are entering the automotive market in large numbers. For example, V2V allows vehicles to link with each other (and with the cloud, towards the Internet of Everything), and vehicles can communicate with other vehicles (e.g., using dual-band IP radio via a secure mesh network, etc.) about accidents, traffic slowdowns, or other dangerous situations. Vehicles so notified can, for example, warn the driver, and / or the vehicle can autonomously slow down. In this way, connected vehicles are not limited to objects that can be detected independently (e.g., by radar, sensors, cameras). This increases awareness of real-time driving conditions because these technologies are not limited to the line of sight of the vehicle's individual sensors.

[0005] On the other hand, there are still outstanding issues and challenges that need to be addressed to enable the large-scale use of these V2X technologies. Besides the very important issues of radio communication and network security, one of the main questions is what data should be shared between vehicles. The answer to this question directly impacts two important technical issues: the bandwidth required for exchanging data and the usefulness of the information sent to other vehicles, such as in making ADAS-type decisions.

[0006] With advancements in ADAS / AD technology, current vehicle perception systems are capable of detecting objects, such as pedestrians or other vehicles, in a full 360-degree radius around the vehicle, typically extending to ±80 meters. This means that in a crowded city, each vehicle could easily detect over a hundred objects. If all vehicles within communication range attempted to broadcast all detections to all other vehicles, this would be extremely problematic due to the high bandwidth required to exchange such large amounts of data. For example, in a crowded urban environment, a vehicle could easily receive thousands of messages per second, or receive messages from hundreds of other vehicles.

[0007] With the introduction of 5G networks, the data bandwidth bottleneck between vehicles may be resolved. However, in reality, there is always more information than can be uploaded in a given radio bandwidth and / or processed in real time by other vehicles.

[0008] Therefore, it is necessary to overcome the technical limitations of data communication in vehicle communication in the prior art. Summary of the Invention

[0009] The above-mentioned technical problem is solved by the subject matter of the independent claims. The dependent claims describe further preferred embodiments.

[0010] According to an embodiment of the first aspect of the present disclosure, a computer-implemented method for training a neural network to select objects near a target vehicle is disclosed herein. The method includes aggregating sensor data related to a plurality of vehicles. The method also includes filtering the sensor data according to one or more conditions, the conditions identifying an action of an ADAS / AD system of at least one of the plurality of vehicles. The method also includes identifying one or more objects near the target vehicle based on the filtered sensor data; and using the identified one or more objects to train the neural network to determine potential objects that cause triggering of the ADAS / AD system of another vehicle.

[0011] Identifying one or more objects proximate to the target vehicle includes identifying a plurality of locations of the one or more objects.

[0012] Identifying one or more objects in the vicinity of the target vehicle may also include identifying a plurality of velocities and / or accelerations of the one or more objects.

[0013] The method according to the first aspect may further comprise using the plurality of positions and / or the plurality of velocities and / or accelerations to train the neural network.

[0014] The method according to the first aspect may further comprise tracing the identified one or more objects back to previous sensor readings to track the one or more objects over a plurality of time steps.

[0015] According to the method according to the first aspect, the one or more conditions for the identification action may be one or more of the following: detection of an anomaly in the vehicle group; detection of a difference from the traffic forecast; detection of a vehicle control unit alarm; detection of a path planning algorithm needing to correct its route due to an object not previously seen by the vehicle's perception system; detection of the vehicle's ADAS / AD system being disconnected and requesting driver intervention; detection of a sharp change in direction of the detected object; detection of a collision or detection of performing corrective action to avoid a collision; and / or detection of identification of a high-risk participant.

[0016] According to the method according to the first aspect, the aggregated sensor data may be synchronized sensor data.

[0017] According to the method according to the first aspect, the aggregated sensor data may come from multiple vehicles recording sensor data simultaneously, or from a virtual world simulation of multiple vehicles.

[0018] According to the method according to the first aspect, the trained neural network may be a neural network trained for a specific traffic scenario.

[0019] According to the method according to the first aspect, the output of the trained neural network may include the potential object and its position and / or velocity for reporting.

[0020] According to a second aspect of the present disclosure, a vehicle is disclosed. The vehicle includes: an acquisition unit configured to acquire sensor-based data; a processing unit configured to use the acquired sensor-based data as input to a neural network trained according to the method according to the first aspect of the present disclosure, wherein the trained neural network determines one or more objects that cause corrective action by an ADAS / AD system of another vehicle; and a communication unit configured to report the determined one or more objects to at least one other vehicle, a traffic facility, or a cloud server.

[0021] According to the vehicle according to the second aspect, the processing unit may be configured to mark the determined one or more objects in the traffic scene with binary information to report the object or not report the object.

[0022] According to the vehicle according to the second aspect, the number of the extracted one or more objects is less than the number of objects identified in the environment of the vehicle from the sensor-based data.

[0023] According to the vehicle according to the second aspect, the trained neural network outputs multiple confidence values ​​for the identified objects, each confidence value in the multiple confidence values ​​indicating the likelihood that the corresponding identified object will cause corrective action by the ADAS / AD system of another vehicle.

[0024] According to an embodiment of the third aspect of the present disclosure, a cloud server or a traffic facility unit is disclosed herein. The cloud server or the traffic facility unit includes: an acquisition unit configured to acquire sensor-based data; a processing unit configured to use the acquired sensor-based data as input to a neural network trained according to the method according to the first aspect of the present disclosure, wherein the trained neural network determines one or more objects that cause corrective action of an ADAS / AD system of another vehicle; and a communication unit configured to report the determined one or more objects to at least one other vehicle.

[0025] According to an embodiment of a fourth aspect of the present disclosure, disclosed herein is a cloud server or a traffic facility unit configured to implement the computer-implemented method according to the first aspect of the present disclosure.

[0026] According to an implementation of a fifth aspect of the present disclosure, disclosed herein is a computer program comprising instructions, which, when executed by a computer, causes the computer to perform the method according to the first aspect of the present disclosure.

[0027] According to an embodiment of a sixth aspect of the present disclosure, disclosed herein is a computer-readable storage medium comprising instructions, which, when executed by a computer, causes the computer to perform the method according to the first aspect of the present disclosure.

[0028] According to an embodiment of the seventh aspect of the present disclosure, disclosed herein is a computer program comprising instructions, which, when executed by a computer, cause the computer to acquire sensor-based data; use the acquired sensor-based data as input to a neural network trained according to the method according to the first aspect of the present disclosure, wherein the trained neural network determines one or more objects that cause corrective action of the ADAS / AD system of another vehicle; and report the determined one or more objects to at least one other vehicle or to a traffic facility or to a cloud server.

[0029] According to an embodiment of the eighth aspect of the present disclosure, disclosed herein is a computer-readable storage medium comprising instructions, which, when executed by a computer, causes the computer to acquire sensor-based data; use the acquired sensor-based data as input to a neural network trained according to a method according to the first aspect of the present disclosure, wherein the trained neural network determines one or more objects that cause corrective action of the ADAS / AD system of another vehicle; and report the determined one or more objects to at least one other vehicle or to a traffic facility or to a cloud server. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Embodiments of the present disclosure will now be explained in detail by way of non-limiting examples only, with reference to the accompanying drawings described below. The same reference numerals appearing in different drawings may indicate the same or functionally similar elements, unless otherwise specified.

[0031] Figure 1 A vehicle system and a cloud server according to an embodiment are illustrated.

[0032] Figure 2 An apparatus for determining or extracting one or more objects according to an example embodiment of the present disclosure is illustrated.

[0033] Figure 3 Multiple detection zones for a vehicle are illustrated.

[0034] Figure 4 illustrates a programmable signal processing device that can be configured to implement Figure 2 function of the device.

[0035] Figure 5 A flow chart illustrating a computer-implemented method of training a neural network to select objects near a target vehicle.

[0036] Figure 6 An exemplary traffic scenario is illustrated for explaining the concept of selecting an object near a target vehicle.

[0037] 7A to 7C A global map and a corresponding map as seen by each vehicle are illustrated separately to illustrate the concept of selecting objects near a target vehicle.

[0038] Figure 8A and Figure 8B The overall concept of the present disclosure is illustrated compared to conventional systems. DETAILED DESCRIPTION

[0039] Where reference numerals are attached to technical features in the drawings, detailed description, or any claims, such reference numerals are included solely for the purpose of enhancing the intelligibility of the drawings, detailed description, and claims. Therefore, neither the reference numerals nor their absence shall have any limiting effect on the scope of any claim element.

[0040] Modern vehicle perception systems are based on two or more sensor types (e.g., radar, camera, and / or lidar) and are able to correctly identify over 90% of all objects. This means that all information needs to be shared between all vehicles. The present disclosure is based on the overall concept of providing an object reduction selection mechanism for limiting the amount of exchanged data, for example, limiting the number of detected objects that need to be communicated to other vehicles. This object reduction selection mechanism can be easily applied to vehicles with V2V or other VTX connectivity capabilities.

[0041] This disclosure proposes an artificial intelligence (AI)-based method for object reduction selection using a trained neural network for efficient utilization of V2X networks, and proposes methods for training the neural network in an automated and systematic manner. This method allows for the efficient exchange of relevant road or traffic information between vehicles, allowing other vehicles to be alerted to potentially dangerous objects and take appropriate autonomous action. The advantage of this approach is that precise messages are sent only when they are truly needed.

[0042] Figure 1 The system of the vehicle 10 and the cloud server 20 according to the embodiment is illustrated. The vehicle 10 (for example, a fleet of ADAS / AD vehicles) is equipped with a communication interface to achieve communication connection with the cloud server 20. This communication connection can be achieved in different forms; Figure 1 As illustrated, the vehicle 10 may have, for example, an over-the-air (OTA) interface (e.g., a radio interface according to a wireless communication standard such as LTE, 4G, 5G, or 6G) with a base station 30. The base station 30 provides a communication connection to a cloud server 20 (e.g., via a core network, the Internet, etc.). Based on this communication connection, data related to each vehicle 20 may be uploaded to the cloud server 20 and / or may be transmitted to other vehicles 10, for example, in the form of software as a service (SaaS). In addition, direct V2V communication may be implemented, for example, based on the aforementioned communication standards, or via V2I communication using, for example, a roadside unit.

[0043] The vehicle 10, in particular an ADAS / AD vehicle, can be equipped with a wide range of sensor units for environmental perception (e.g., cameras and / or radars and / or lidars). These sensors allow the vehicle 10 to perceive its environment and, for example, detect lane markings, traffic signs, and other road users, including their dynamic characteristics (position and driving behavior, such as driving direction, attitude, yaw, speed or acceleration relative to the ADAS vehicle). This perception software (SW) forms the basis of modern ADAS systems and will be present (with varying degrees of maturity and quality) in various ADAS vehicles. Based on this, ADAS / AD vehicles are typically equipped with L2+ capabilities, i.e., they can, for example, perform lane keeping and distance keeping.

[0044] Figure 2 An apparatus 100 according to an example embodiment of the present disclosure is shown for determining or extracting one or more objects (e.g., another vehicle, a bicycle, a pedestrian, etc.) that trigger an action of an ADAS / AD system of another vehicle. The triggered action may be, for example, triggering a corrective action of the other vehicle, such as activating a specific ADAS / AD function (e.g., autonomous emergency braking, lane keeping assist), as will be described in detail below.

[0045] The device 100 can be set in the vehicle 10 and can preferably be installed on the vehicle 10 facing the travel direction D of the vehicle 10. Those skilled in the art understand that the device 100 is not required to face the travel direction; the device 100 can also face sideways or backward. The device 100 can be part of a radar sensor, a radar module, or a radar system. The device 100 can also be part of a lidar sensor, a lidar module, or a lidar system. The device can also be a camera sensor or a camera system. The device 100 can also be a device for collecting and processing multiple different sensor data (such as radar data, lidar data, camera data). The vehicle 10 can also have more than one such device 100.

[0046] The device 100 may be provided as part of a driver assistance system 200 of a vehicle 10 or as Figure 2 The illustrated embodiment interacts with a driver assistance system 200 of the vehicle 10. The driver assistance system of the vehicle 200 may include advanced driver assistance system (ADAS) functionality and / or autonomous driving (AD) functionality, ie, an ADAS / AD system.

[0047] The vehicle 10 can be any ground vehicle that is moved by mechanical power. The accompanying drawings illustrate the vehicle 10 as a car equipped with the device 100. However, the present disclosure is not limited thereto. Therefore, the device 100 can also be mounted on, for example, a van, a truck, a motorcycle, a bus, a robot, or any other mobile unit.

[0048] The device 100 may have multiple detection areas, for example oriented so that it has Figure 3 As a non-limiting example, an object may be considered to be around or near the vehicle 10 at least when the object is within one or more of the plurality of detection zones of the device 100.

[0049] like Figure 2 As shown, the device 100 includes an acquisition unit 120 and a processing unit 130. Furthermore, the device 100 may optionally include one or more sensors or sensor units (e.g., radar and / or lidar and / or camera sensors) 110, although one or more sensor units 110 may also be provided separately from the device 100. Furthermore, the device 100 may also be connected to a communication unit 140 (e.g., one or more antennas for transmitting and receiving information). Alternatively, the communication unit 140 may be an integral part of the device 100.

[0050] The acquisition unit 120 can be configured (e.g., programmed) to acquire and record sensor-based data of at least one sensor unit 110 of the vehicle 10, preferably in combination with positioning data associated with a positioning unit (e.g., a GPS unit, a DPGS unit, not shown) and time data of the vehicle 10. The sensor-based data can be radar data (from a radar sensor) and / or lidar data (from a lidar sensor) and / or camera data (from a camera system). Note that the sensor-based data and positioning data of the vehicle are real-world data and not simulated data. Acquisition of the sensor-based data from the vehicle's sensor unit 110 can be performed via an in-vehicle data communication interface, such as a CAN bus communication or Ethernet based on a local controller or domain controller of the vehicle 10.

[0051] The processing unit 130 can be configured (e.g., programmed) to use the acquired sensor-based data (e.g., a list of identified vehicles or objects, and corresponding positions, attitudes, yaws, velocities, and / or accelerations, sensor-based identified vehicle trajectories or tracks) as input to the trained neural network. The training of the neural network is described in detail below. The neural network is trained to analyze the positions (e.g., based on object detection and identification) and driving behaviors (e.g., based on vehicle trajectory analysis) of multiple vehicles in the vicinity or surrounding area of ​​a target vehicle, and the neural network determines or extracts one or more objects that can cause corrective action by the ADAS / AD system of another vehicle.

[0052] The communication unit 140 can be configured (e.g., programmed) to report the identified one or more objects to at least one other vehicle, a traffic facility, or a cloud server. By reporting only those vehicles that trigger or will trigger an ADAS / AD system action (e.g., corrective action to avoid a collision), only potentially dangerous objects are extracted and reported to other vehicles. In this way, not all sensor-based information is shared among all vehicles, thereby limiting the number of objects that need to be transmitted.

[0053] Here, the processing unit 130 is preferably configured to mark one or more identified objects in the traffic scene with binary information to indicate whether to report the object or not. In other words, the binary information can be easily used to determine whether to include or ignore the identified object in a report message to other vehicles and / or to a traffic facility and / or a cloud server.

[0054] Those skilled in the art will appreciate that the number of one or more objects determined (or extracted) is less than the number of objects actually identified in the vehicle environment based on the sensor-based data. That is, the neural network actively performs the selection of a subset of objects or vehicles, such as a subset of potentially dangerous vehicles, and only that subset (and corresponding information such as position, heading, attitude, yaw, and / or velocity data) is included in the report message.

[0055] The trained neural network may also preferably output multiple confidence values ​​for the identified objects, each confidence value indicating the likelihood that the corresponding identified object will cause corrective action by the ADAS / AD system of another vehicle. Such confidence values ​​may also be used by the processing unit or other vehicles (receiving such report messages) for appropriate processing. That is, the communication unit 140 may also be configured to report a subset of the identified objects having confidence values ​​greater than a predetermined threshold.

[0056] Figure 4 is a schematic diagram of a programmable signal processing device 300, which can be configured to implement Figure 2 The function of the device 100.

[0057] The signal processing device 300 has an interface module 310 that provides a means for sending and receiving information, such as one or more antennas or a wired connection, for example, for providing a communication connection to the cloud server 20 and / or other vehicles. The signal processing device 300 also has a processor 320, such as a CPU, for controlling the programmable signal processing device 300 to perform the functions of the device 100; a working memory 330, such as a random access memory; and an instruction storage unit 340 that stores a computer program 345 having computer-readable instructions that, when executed by the processor 320, cause the processor 320 to perform the functions of the device 100.

[0058] The instruction storage unit 340 may include a ROM, for example in the form of an electrically erasable programmable read-only memory (EEPROM) or flash memory, which is preloaded with computer-readable instructions. Alternatively, the instruction storage unit 340 may include a RAM or similar type of memory, and the computer-readable instructions may be input thereto from a computer program product (e.g., a computer-readable storage medium 350, such as a CD-ROM, etc.) or a computer-readable signal 360 carrying computer-readable instructions.

[0059] Apparatus 100 may alternatively be implemented in non-programmable hardware, such as an application specific integrated circuit (ASIC), or in any other suitable manner using any suitable combination of hardware and software components.

[0060] As described above, the present disclosure provides various techniques to accurately and reliably extract and report objects that cause triggering of another vehicle's ADAS / AD system.

[0061] The processing device 300 can also be implemented in the cloud server 20 to achieve Figure 2 In other words, the functions of the acquisition unit 120, the processing unit 130, and the communication unit 140 may also be implemented in the cloud server 20 or a traffic facility unit (such as a roadside unit).

[0062] Figure 5 A flow chart illustrating a computer-implemented method for training a neural network to select objects near a target vehicle. Here, the selection of objects near a target vehicle is performed to select, from a plurality of objects near the target vehicle (other vehicles, other road users, pedestrians), those objects that are likely to cause or activate an action (e.g., a corrective action) by an ADAS / AD system of another vehicle, such as changing direction and / or speed to avoid a collision.

[0063] To do this, a neural network can be trained to analyze the position and movement / driving behavior of multiple vehicles in the vicinity of a target or own vehicle.

[0064] According to step S110, sensor data related to the plurality of (connected) vehicles are aggregated. Thus, the plurality of (connected) vehicles are sensed using one or more of the above-mentioned sensors (radar sensor and / or lidar sensor and / or camera sensor), and the acquired sensor data can be processed and aggregated so that the corresponding sensor data from different sensors and for the plurality of vehicles and the target vehicle are synchronized in time and space, that is, in collecting and storing the spatiotemporal relationships between the plurality of vehicles, the detected plurality of vehicles or other objects in the vicinity of the target vehicle are appropriately associated with each other in terms of geographical location and time.

[0065] As explained, the aggregated sensor data can come from multiple vehicles simultaneously recording sensor data (e.g., a fleet of L2+ and / or L3+ connected vehicles simultaneously recording data). Alternatively, the aggregated sensor data can come from a virtual world simulation of multiple vehicles. In other words, the processed data used to train the neural network (see below) can come from actual real-world traffic recordings or from simulated traffic recordings.

[0066] According to step S130, the sensor data is filtered according to one or more conditions. Here, a condition can be defined to identify an action of the ADAS / AD system of at least one vehicle (around or near the target vehicle). Such an action can be an action that changes the driving behavior of the vehicle, such as deceleration or braking of the vehicle, changing the steering angle to avoid another vehicle or other obstacle, and sudden acceleration of the vehicle. Therefore, such a condition can be defined to identify a corrective action of the ADAS / AD system of at least one other vehicle around or near the target vehicle.

[0067] According to step S150, one of a plurality of objects or road users is identified based on the filtered sensor data, which may be one or more of a plurality of vehicles, but may also be other road users, such as bicycles, pedestrians, etc. In other words, an object that causes a corrective action of the ADAS / AD system in at least one vehicle and a vehicle having an ADAS / AD system that performs a corrective action are identified. Here, identification may include obtaining a plurality of positions of the identified object and a driving or movement behavior of the identified object. The driving or movement behavior may include, for example, a plurality of positions and / or attitudes and / or yaws and / or velocities and / or accelerations (vectors), based on which the driving and movement behavior may be analyzed over time.

[0068] Since the condition filters out only those vehicles in the traffic scene that are involved in the action (e.g. corrective action) of the ADAS / AD system of at least one vehicle, only a subset of the vehicles in the traffic scene can be identified and communicated or reported to the vehicles in the traffic scene.

[0069] According to step S170, the identified one or more objects are input into the neural network and thereby used to train the neural network to determine potential or candidate objects (near the vehicle) that may trigger the ADAS / AD system of another vehicle. The input may include a set of positions of the identified objects and a plurality of velocities and / or accelerations (vectors). The neural network is thus trained to analyze the positions and motion behaviors of multiple vehicles near the target vehicle.

[0070] In general, neural networks have become the state-of-the-art tool in many vehicle applications over the past few years. In computer vision in particular, state-of-the-art object detectors are based on neural networks, in particular deep neural networks and convolutional neural networks. In order to transfer their advantages in these fields to sensor processing and object detection for autonomous driving, similar components that can be seamlessly plugged into neural network systems are needed. The advantage of (supervised) neural networks is that they can learn a mapping f(x,θ)=y with input x, output y and learnable parameters θ based on data rather than based on a fully designed model. It is assumed that (supervised) neural networks can define the mapping in an abstract multidimensional way. The neural network component proposed here can be trained (i.e., trained by directly considering the inputs and outputs to optimize the network weights), i.e., the model dynamics are learned implicitly and directly from the input data.

[0071] In order to be easily pluggable into neural networks and trainable with gradient descent, the proposed algorithm (embodiment) preferably satisfies the following properties: (1) its output (the object that causes the action of the ADAS / AD system of another vehicle) is differentiable with respect to the input data (the objects identified over multiple time steps based on filtered sensor data, their positions, and driving behavior) and with respect to all learnable parameters; and (2) the gradient should not be zero almost everywhere. This can be achieved by applying the chain rule (e.g., computing sums, differences, etc.) on differentiable subfunctions and by avoiding hard YES / NO assignments that are generally non-differentiable.

[0072] This is important to ensure that the learnable components can be trained with gradient descent and thus adjusted by optimization algorithms. This is superior to prior art methods, which are either non-differentiable or involve hard assignments without non-zero gradients. In addition, the embodiments described here can themselves contain learnable parameters. Therefore, they can be partially or fully adjusted by data-driven optimization algorithms, which enables them to learn more abstract mappings.

[0073] A preferred embodiment of a neural network may be a recurrent neural network (RNN) to enable allocation and information collection over multiple time steps (over multiple frames), in particular the identification or allocation of data values ​​over multiple time steps at different grid positions (e.g., related to position, velocity). Thus, it is known that RNNs are able to collect information in a neural network over multiple time steps in a sequence, for example using a long-short-term memory (LSTM) with an internal memory state, see, for example, Hochreiter et al.: "Long short-term memory", Neural computation, 9(8), 1735-1780, 1997.

[0074] Referring back to the identification step, identification of one or more objects can advantageously be performed by tracking the one or more objects multiple times, for example, across multiple frames. A frame can be considered a sensor dataset for a particular time step, and thus multiple frames represent sensor datasets for multiple time steps. For example, a radar dataset for a particular time step (e.g., representing range data and Doppler data in the spatial dimensions of ±80 m and having a particular spatial resolution) can be considered a radar data frame.

[0075] Figure 6 An example illustrating this concept is shown. A target or host vehicle (denoted by Δ) has aggregated sensor data (e.g., defined by the sensor's detection range, such as ±80 meters for a radar sensor) about nearby vehicles A, B, and C (denoted by solid-line boxes). As shown, the ADAS / AD system of vehicle A performs corrective actions to avoid a collision with vehicle B. Vehicle B is illustrated not only for the current point in time (as a solid-line box) but also for four different previous points in time (as dashed-line boxes) to schematically illustrate the driving behavior of vehicle B that led to a potentially dangerous situation.

[0076] The corrective ADAS / AD action of vehicle A is illustrated as a change from the expected speed vector (dashed arrow) to the corrective speed vector (solid arrow). Those skilled in the art understand that the corrective speed vector points in a different direction than the expected speed vector to avoid a collision with vehicle B, and that the corrective speed is greater than the expected speed to avoid a collision. In addition, the action of the ADAS / AD system of vehicle B can also be identified, in particular, it has been identified earlier in time due to the difference in direction (the difference in the direction of the dashed bounding box of vehicle B). In contrast, the corrective action of vehicle C cannot be identified. Those skilled in the art understand that early action of detecting the ADAS / AD system of vehicle B to change the direction of travel (possibly overtaking vehicle C) can be advantageously communicated or reported to vehicle A to better avoid dangerous situations.

[0077] In a preferred embodiment, the identified one or more objects are traced back to previous sensor readings to track the one or more objects over multiple time steps. Figure 6 In , vehicle B is traced back to previous sensor readings to identify the trajectory (defined by position, and preferably also by velocity and / or acceleration) of vehicle B over time. Tracing one or more objects over multiple time steps can be performed in a finite amount of time, for example 5 seconds.

[0078] Here, in general, the algorithm can thus identify objects that may be "troublesome," which can be defined here as objects that force corrective action by the ADAS / AD system. This can be done by marking these objects at a given frame and projecting this information back to the previous sensor reading (e.g., minus 5 seconds, for which known object tracking algorithms can be used). This allows all objects in the scene to be labeled with binary information (report / broadcast or reduced selection), and the filtered results can then be used to train an AI-based object reduction selection neural network.

[0079] Logo Figure 6 The action of the ADAS / AD system of vehicle B in the vehicle B is conditioned upon detecting that vehicle B changes direction of travel abruptly (ie, within a predetermined amount of time) or changes direction of travel to an area where oncoming traffic can be expected.

[0080] In addition, the logo Figure 6 The condition for the action of the ADAS / AD system of vehicle A in FIG is to detect that vehicle A performs corrective action to avoid a collision.

[0081] The above examples are non-limiting examples of conditions for identifying the action of the ADAS / AD system of at least one of the multiple vehicles. Further preferred examples of conditions may be one or more of the following:

[0082] Anomalies in the vehicle group were detected;

[0083] Discrepancies from traffic forecasts are detected;

[0084] Detection of a vehicle control unit alarm (e.g. related to Autonomous Emergency Braking (AEB), Lane Keeping Assist (LKA));

[0085] Detection that the path planning algorithm needs to correct its course due to an object not previously seen by the vehicle's perception system

[0086] Detects that the vehicle's ADAS / AD system is disconnected and requests driver intervention;

[0087] Detecting a sudden change in the direction of a detected object (i.e., a change by a predetermined amount within a specific amount of time);

[0088] detecting a collision or performing corrective action to avoid a collision;

[0089] High-risk participants were identified.

[0090] In this context, a vehicle group can be considered to be a plurality of vehicles that appear to move together in a synchronized manner, for example, by maintaining a statistical average of the vehicle spacing over time. An example of such a vehicle group is a vehicle train, i.e., a plurality of vehicles aligned one behind the other and maintaining an average distance over time. A vehicle group anomaly can be considered to be a situation in which at least one vehicle begins to deviate from the established vehicle spacing or from the average vehicle speed of the group. Detecting such a vehicle group anomaly can indicate a potential situation that may require other vehicles to also perform corrective actions, such as preparing to move to fill the gap that opened up when a vehicle left the group.

[0091] In addition, traffic prediction can be performed based on a traffic prediction model, such as traffic modeling based on the Lattice Boltzmann Method (LBM). Such traffic modeling is known (e.g., et al., “Lattice Boltzmann Method for Heterogeneous Multi-Class Traffic Flow”, Journal of Computational and Theoretical Transport, Vol. 50, 2021, pages 27-51), and those skilled in the art understand that differences from such traffic predictions, for example, a difference of a predetermined amount in vehicle position, a difference of a predetermined amount in vehicle speed, etc., can be used as an indication of an action that has been or will be taken by the vehicle's ADAS / AD system.

[0092] Here, collisions can be identified relatively easily in the virtual simulation, and corrective actions to avoid collisions can readily occur and thus be identified for a fleet of connected vehicles that are simultaneously recording data.

[0093] Furthermore, high-risk participants may be scooters, motorcycles, bicycles, vehicles with trailers, emergency vehicles with traffic lights, etc., ie traffic participants associated with a higher risk of injury in the event of an accident or a collision with another object.

[0094] The above idea 7A to 7C Further examples are given in . Figure 7A An overall situation diagram (global diagram) is illustrated, and shows that vehicle C is entering a road intersection (which is a potentially dangerous situation) despite the red light. Figure 7B The objects detected by vehicle A are shown ("the world as seen from vehicle A"), and one skilled in the art will appreciate that not all vehicles are detected (due to, for example, occlusions, etc.). Here, vehicle C is not visible due to being obscured by a large truck / bus. Figure 7COn the second map shown ("The world as seen from vehicle B"), it can be noted that the number of vehicles detected by vehicle B is also limited due to the sensor line of sight. It can also be noted that some of the objects detected by A and B overlap. On the other hand, vehicle B should detect the dangerous action taken by vehicle C and should therefore inform the other vehicles at the road intersection about it. Using the trained neural network described above, vehicle B can determine vehicle C as an object that may cause the triggering of the ADAS / AD system of another vehicle (e.g. vehicle A) and will report (only) this object to the other vehicles as appropriate, either through direct V2V communication or via traffic infrastructure (V2X).

[0095] The output of the trained neural network includes the determined potential objects and their positions and / or speeds and / or accelerations for reporting to other vehicles. Those skilled in the art will appreciate that this allows a narrowing down of the selection from a plurality of objects in the area of ​​the target vehicle to only those objects that may have an impact on the ADAS / AD system of another vehicle. The method therefore limits the amount of traffic information that should be exchanged between vehicles (V2V) and / or between vehicles and facilities (V2X), and therefore provides more efficient network utilization. In addition, since only relevant objects are reported using this scheme, the saved bandwidth can be advantageously used to provide more accurate information about the relevant objects (multiple positions and / or speeds and / or accelerations), so that the report message can be a more accurate message and therefore better supports other vehicles in identifying the reported relevant objects.

[0096] Thus, the present disclosure teaches the use of AI-based algorithms to analyze the positions (e.g., through object detection) and behaviors (e.g., through vehicle trajectory analysis) of all detected objects around a vehicle with an ADAS / AD perception system to extract those objects that may be potentially dangerous to other vehicles. This advantageously allows limiting the number of objects that need to be broadcast to other vehicles.

[0097] and Figure 8A Compared with the conventional system exemplified in Figure 8B That is, although multiple sensor systems (e.g., cameras, radars, lidars, or any combination thereof) can be used to perform vehicle perception of other vehicles or objects in the vehicle's sensing area and thus report through any of V2V, V2I, or V2C communications, resulting in Figure 8A As shown, more data is transmitted, but not all of this more data is relevant (thereby generating unnecessary communication bandwidth and also generating unnecessary processing burden for other vehicles and / or transportation facilities to ignore the data). As explained, the present disclosure can successfully address this shortcoming of conventional systems by using a trained neural network to perform object reduction selection to report only those objects that are potentially dangerous.

[0098] Those skilled in the art will appreciate that the use of neural networks may require the preparation of extensive training and validation datasets that can be used to prepare the algorithm. Today, in most cases, this is prepared through lengthy manual annotation of the data performed by humans. In the case of training the aforementioned reduced selection algorithm, manual annotation is extremely difficult to perform (because there are typically a large number of objects in a given frame to be analyzed along with several seconds of history of their behavior, etc.). In contrast, the above-described embodiments provide a robust mechanism for training neural networks in a manner that avoids manual annotation, etc., and thus provide an automated method that can be used to prepare appropriate relevant datasets.

[0099] According to another preferred embodiment, the trained neural network can be a neural network trained for specific traffic scenarios, such as winding, multi-lane traffic roads, traffic areas with statistically increased likelihood of traffic accidents, etc. In other words, the neural network can be trained for specific traffic facilities (e.g., roadside units) located in such specific traffic scenarios, and can implement V2I communication with vehicles in the local geographic area of ​​the facility.

[0100] According to another embodiment, the cloud server or the traffic facility unit (such as a roadside unit) can also be configured as in Figure 4

[0014] As generally described in

[0014] , sensor-based data is acquired from multiple vehicles via an air interface. Preferably, the sensor-based data is aggregated to provide synchronized sensor-based data across the different vehicles. The processing unit of the cloud server or the traffic facility unit can use the acquired sensor-based data as input to a trained neural network, as described above. Furthermore, in this embodiment, the trained neural network determines one or more objects that cause corrective action by an ADAS / AD system of another vehicle, and the communication unit reports the determined one or more objects to at least one other vehicle.

[0101] According to another embodiment, the communication unit of the traffic facility unit or the cloud server may also receive a report message of the one or more objects determined or extracted thereby from the first vehicle, and send a report message to the second vehicle.

[0102] In the foregoing description, various aspects have been described with reference to various embodiments. Therefore, the description should be considered illustrative rather than restrictive. Similarly, the figures shown in the accompanying drawings, which highlight the features and advantages of the embodiments, are presented for illustrative purposes only. The architecture of the embodiments is sufficiently flexible and configurable that it can be utilized in ways other than those shown in the accompanying drawings.

[0103] In one exemplary embodiment, the software implementations presented herein may be provided as a computer program or software, such as one or more programs having instructions or instruction sequences, which are included or stored on an article of manufacture, such as a machine-accessible or machine-readable medium, instruction storage, or computer-readable storage device, each of which may be non-transitory. The program or instructions on the non-transitory machine-accessible medium, machine-readable medium, instruction storage, or computer-readable storage device may be used to program a computer system or other electronic device. Machine- or computer-readable media, instruction storage, and storage devices may include, but are not limited to, floppy disks, optical disks, and magneto-optical disks, or other types of media / machine-readable media / instruction storage devices suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may find applicability in any computing or processing environment. As used herein, the terms "computer-readable," "machine-accessible medium," "machine-readable medium," "instruction storage," and "computer-readable storage device" shall include any medium capable of storing, encoding, or transmitting instructions or instruction sequences for execution by a machine, computer, or computer processor, causing the machine / computer / computer processor to perform any of the methods described herein. Furthermore, it is common in the art to refer to software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, etc.) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that execution of the software by a processing system causes the processor to perform an action to produce a result.

[0104] Some embodiments may also be implemented by the preparation of application specific integrated circuits, field programmable gate arrays, or by interconnecting an appropriate network of conventional component circuits.

[0105] Some embodiments include a computer program product. A computer program product can be a storage medium or multiple storage media, one or more instruction stores, or one or more storage devices having instructions stored thereon or therein, which instructions can be used to control or cause a computer or computer processor to execute any of the programs of the exemplary embodiments described herein. The storage medium / instruction store / storage device can include, for example, but not limited to, an optical disk, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory, flash memory cards, magnetic cards, optical cards, nanosystems, molecular memory integrated circuits, RAID, remote data storage / archiving / warehousing, and / or any other type of device suitable for storing instructions and / or data.

[0106] Some implementations stored on a computer-readable medium or multiple media, one or more instruction stores, or one or more storage devices include hardware for controlling the system and software for enabling the system or microprocessor to interact with a human user or other mechanism using the results of the embodiments described herein. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Finally, such computer-readable media or storage devices also include software for executing the example aspects described above.

[0107] Included in the programming and / or software of the system are software modules for implementing the processes described herein. In some example embodiments herein, the modules include software, while in other example embodiments herein, the modules include hardware or a combination of hardware and software.

[0108] Although various embodiments of the present disclosure have been described above, it should be understood that they are presented as examples rather than limitations. It will be apparent to those skilled in the relevant art that various changes in form and detail may be made therein. Therefore, the exemplary embodiments described above are not restrictive.

[0109] Furthermore, the purpose of the abstract is to enable patent offices and the public, especially scientists, engineers, and practitioners in the field who are not familiar with patent or legal terminology or wording, to quickly ascertain the nature and essence of the technical disclosure of the application from a cursory inspection. The abstract is not intended to limit the scope of the embodiments presented herein in any way. It should also be understood that any process recited in the claims need not be performed in the order presented.

[0110] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of the claims, but rather as descriptions of specific features of the specific implementations described herein. Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. On the contrary, the various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a variant of a sub-combination or sub-combination.

[0111] In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0112] Now that some illustrative embodiments have been described, it will be apparent that the foregoing is illustrative and not restrictive, and has been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of devices or software elements, these elements can be combined in other ways to achieve the same objectives. Actions, elements, and features discussed in connection with only one embodiment are not intended to be excluded from similar roles in other embodiments or multiple embodiments.

[0113] The apparatus described herein may be embodied in other specific forms without departing from its characteristics. The above embodiments are illustrative rather than restrictive of the systems and methods described. The scope of the apparatus described herein is therefore indicated by the appended claims rather than the foregoing description, and changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein.

Claims

1. A computer-implemented method for training a neural network to select objects in the vicinity of a target vehicle, the method comprising the steps of: Aggregating ( S110 ) sensor data associated with a plurality of vehicles near the target vehicle; filtering ( S130 ) the sensor data according to one or more conditions to filter out sensor data related to at least one vehicle among the plurality of vehicles from the sensor data, the conditions identifying an action of an ADAS / AD system of the at least one vehicle among the plurality of vehicles; identifying ( S150 ) one or more objects near the target vehicle based on the filtered sensor data; as well as The neural network is trained using ( S170 ) the identified one or more objects to determine potential objects that cause triggering of an ADAS / AD system of another vehicle.

2. The method according to claim 1, wherein: The step of identifying (S150) includes identifying a plurality of locations of the one or more objects.

3. The method according to claim 2, wherein: The plurality of positions are used to train the neural network.

4. The method according to claim 1, wherein: The step of identifying (S150) includes identifying a plurality of velocities and / or accelerations of the one or more objects.

5. The method according to claim 4, wherein The plurality of velocities and / or accelerations are used to train the neural network.

6. The method according to claim 1 or 2, further comprising: The identified one or more objects are traced back to previous sensor readings to track the one or more objects over multiple time steps.

7. The method according to claim 1 or 2, wherein: The one or more conditions for identifying an action are one or more of the following: Anomalies in the vehicle group were detected; Discrepancies from traffic forecasts are detected; A vehicle control unit alarm is detected; Detection that the path planning algorithm needs to correct its course due to an object not previously seen by the vehicle's perception system; Detects that the vehicle's ADAS / AD system is disconnected and requests driver intervention; Detecting a sudden change in direction of the detected object; detecting a collision or performing corrective action to avoid a collision; High-risk participants were identified.

8. The method according to claim 1 or 2, wherein: The aggregated sensor data is synchronized sensor data.

9. The method according to claim 1 or 2, wherein: The aggregated sensor data comes from multiple vehicles recording sensor data simultaneously, or from a virtual world simulation of multiple vehicles.

10. The method according to claim 1 or 2, wherein: The trained neural network is a neural network trained for specific traffic scenarios.

11. The method according to claim 1 or 2, wherein: The output of the trained neural network includes the potential object and its position and / or velocity for reporting.

12. A carrier, comprising: an acquisition unit (120), the acquisition unit being configured to acquire sensor-based data; a processing unit (130) configured to use the acquired sensor-based data as input to a neural network trained according to the method of any one of claims 1 to 11, the trained neural network for determining one or more objects that cause corrective action of an ADAS / AD system of another vehicle; A communication unit (140) is configured to report the determined one or more objects to at least one other vehicle, to a transportation facility, or to a cloud server.

13. The carrier according to claim 12, wherein The processing unit (130) is configured to mark the determined one or more objects in the traffic scene with binary information to report the object or not report the object.

14. The carrier according to claim 12 or 13, wherein: The determined number of one or more objects is less than a number of objects identified in the vehicle's environment from the sensor-based data.

15. The carrier according to claim 12 or 13, wherein: The trained neural network outputs a plurality of confidence values ​​for the identified objects, each confidence value in the plurality of confidence values ​​indicating a likelihood that the corresponding identified object will cause corrective action by the ADAS / AD system of the other vehicle.

16. A cloud server or a transportation infrastructure unit configured to implement the computer-implemented method according to any one of claims 1 to 11.

Images