Method for training an autonomous driving system (ADS) of a vehicle

Abstract

A method for training an autonomous driving system (ADS) of a vehicle is provided, the method comprising monitoring a GPU utilization value, wherein the GPU utilization value relates to a current extent to which a GPU is occupied or to an extent to which the GPU will be occupied as reflected by currently queued data samples to be provided to the GPU. If the GPU utilization value is above a GPU utilization threshold, the following steps are performed: reading sensor data from a disk using a CPU, transferring the sensor data from the CPU to the GPU, training a neural network by the GPU based on the sensor data. If the GPU utilization value is below the GPU utilization threshold, the following steps are performed: generating synthetic sensor data using the GPU, and training the neural network by the GPU based on the synthetic sensor data.

Description

Technical Field

[0001] The disclosed technologies relate to methods and systems for training autonomous driving systems (ADS) for vehicles. In particular, but not exclusively, the disclosed technologies relate to how to more effectively balance the use of sensor data captured by the vehicle's sensors and synthetic sensor data generated by a graphics processing unit (GPU) when training a neural network for ADS. Background Technology

[0002] Most modern cars, and other types of vehicles, are equipped with an Automated Driving System (ADS), which can take the form of Autonomous Driving (AD) designed to control and navigate the car without supervision, or Advanced Driver Assistance Systems (ADAS) designed to assist the driver during driving. ADS is powered by sensor data from sensors located on the vehicle. These sensors are located in different parts of the vehicle, are of different types, and are activated and used under different conditions. Typical sensors used in this context include cameras (image sensors), LiDAR (Light Detection and Ranging) sensors, and radar sensors.

[0003] ADS typically comprises a neural network. Training of this network is generally performed on a server, i.e., a data processing device located outside the vehicle. This server has a GPU, CPU, and disk. By training on a server, sensor data from a wide range of vehicles can be easily collected, making a larger training dataset available.

[0004] Over the past few years, the number of sensors used in ADS has increased, leading to a surge in the amount of sensor data generated. Furthermore, sensors have evolved to offer increased resolution, which also contributes to the increased data volume. While more information is generally positive, the increased CPU power required to process this increased sensor data does impact ADS. For example, the increased data volume also increases the demand for operations related to reading and writing data to disk (tasks typically performed by the CPU). Therefore, the increased data volume has a greater impact on the CPU than on the GPU, potentially leading to suboptimal GPU utilization. Summary of the Invention

[0005] The techniques disclosed herein aim to mitigate, alleviate, or eliminate one or more of the aforementioned defects and disadvantages in the prior art to address various related problems, such as the impact of generating an increased amount of sensor data, where the CPU and / or disk may not be able to provide enough data to the GPU during neural network training, resulting in the GPU not being utilized at its full capacity.

[0006] The various aspects and implementations of the disclosed technology are defined below and in the appended independent and dependent claims.

[0007] A first aspect of the disclosed technology includes a method for training an autonomous driving system (ADS) for a vehicle, the training being performed using: a central processing unit (CPU) for processing input / output (I / O) operations associated with reading sensor data from and writing sensor data to a disk; a disk for storing sensor data, wherein the sensor data originates from sensors; and a graphics processing unit (GPU) for training a neural network of the ADS using sensor data provided to the GPU by the CPU, the method comprising: monitoring a GPU utilization value, wherein the GPU utilization value relates to the degree to which the GPU is currently occupied or to the degree to which the GPU will be occupied as reflected by currently queued data samples to be provided to the GPU; if the GPU utilization value is above a GPU utilization threshold, then reading sensor data from the disk using the CPU, transferring the sensor data from the CPU to the GPU, and training the neural network by the GPU based on the sensor data; if the GPU utilization value is below a GPU utilization threshold, then generating synthetic sensor data using the GPU, and training the neural network by the GPU based on the synthetic sensor data.

[0008] The second aspect of the disclosed technology includes an apparatus comprising: a disk for storing sensor data generated by sensors, wherein the sensor data reflects physical changes around the vehicle; a central processing unit (CPU) for processing input / output (I / O) operations associated with reading sensor data from and writing sensor data to the disk; and a graphics processing unit (GPU) for training a neural network of the vehicle's (100) ADS using sensor data provided by the CPU to the GPU; and control circuitry communicatively connected to the disk, CPU, and GPU, configured to monitor a GPU utilization value, and: if the GPU utilization value is above a GPU threshold, then reading sensor data from the disk using the CPU, transferring the sensor data from the CPU to the GPU, and training the neural network by the GPU based on the sensor data; and if the GPU utilization value is below a GPU threshold, then generating synthetic sensor data using the GPU, and training the neural network by the GPU based on the synthetic sensor data.

[0009] This aspect of the disclosed technology possesses similar advantages and preferred features as other aspects.

[0010] As used herein, the term "non-transient" is intended to describe computer-readable storage media (or "memory") that exclude the propagation of electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase computer-readable media or memory. For example, the terms "non-transient computer-readable media" or "tangible memory" are intended to encompass types of storage devices that include, for example, random access memory (RAM) that do not necessarily permanently store information. Program instructions and data stored in non-transient form on tangible computer-accessible storage media can be further transmitted via transmission media or signals such as electrical signals, electromagnetic signals, or digital signals, which can be transmitted via communication media such as networks and / or wireless links. Therefore, as used herein, the term "non-transient" is a limitation on the medium itself (i.e., tangible, not signal), not on the persistence of data storage (e.g., RAM and ROM).

[0011] The disclosed aspects and preferred embodiments may be suitably combined with each other in any manner that is obvious to those skilled in the art, such that one or more features or embodiments relating to one aspect may also be considered to relate to embodiments of another aspect or another aspect.

[0012] One advantage of some implementations is that by using the GPU according to the above tips, it can be utilized more efficiently, for example, with less idle time and by fully utilizing as many GPU cores as possible. Furthermore, the positive effect of having the GPU generate synthetic sensor data during periods when the CPU or disk cannot provide sensor data is that it allows the GPU to generate synthetic sensor data that can supplement the sensor data (i.e., data generated by the vehicle's sensors). By doing so, in addition to training the neural network more efficiently, the training can also cover edge cases, i.e., sensor combinations that are unlikely to appear in the sensor data generated by the sensors themselves, which has the positive effect of allowing for broader training of the neural network. Besides generating synthetic sensor data, other types of data can also be generated.

[0013] Further embodiments are defined in the dependent claims. It should be emphasized that when the term "comprising" and variations thereof are used in this specification, they are used to specify the presence of a described feature, integer, step, or component. They do not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.

[0014] These and other features and advantages of the disclosed technology will be further illustrated below with reference to the embodiments described herein. Attached Figure Description

[0015] The foregoing aspects, features, and advantages of the disclosed technology will be more fully understood when taken in conjunction with the accompanying drawings and by referring to the following illustrative and non-limiting detailed description of exemplary embodiments of the present disclosure, wherein:

[0016] Figure 1 The illustration shows an example of a vehicle equipped with ADS, which includes a neural network and an external data processing device for training the neural network of ADS.

[0017] Figure 2 This is a flowchart illustrating a method for training the neural network of an ADS for a vehicle.

[0018] Figure 3 Further detailed illustrations can be seen of the different parts that make up the vehicle. Detailed Implementation

[0019] This disclosure will now be described in detail with reference to the accompanying drawings, in which some exemplary embodiments of the disclosed technology are illustrated. However, the disclosed technology may be embodied in other forms and should not be construed as limited to the exemplary embodiments disclosed. The exemplary embodiments of the disclosure are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will recognize that the steps, services, and functions described herein can be implemented using separate hardware circuitry, using software that works in conjunction with a programmable microprocessor or general-purpose computer, using one or more application-specific integrated circuits (ASICs), using one or more field-programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0020] It will also be appreciated that when this disclosure is described in the form of a method, it can also be embodied in a device including one or more processors and one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the device to perform the steps, services, and functions disclosed herein.

[0021] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be noted that the articles “a,” “an,” “the,” and “said,” as used in the specification and appended claims, are intended to mean the presence of one or more elements unless the context clearly indicates otherwise. Thus, for example, in some contexts, a reference to “unit” or “the unit” may refer to more than one unit, etc. Furthermore, the word “comprising” and its variations do not exclude other elements or steps. It should be emphasized that when the term “comprising” and its variations are used in this specification, they are used to specify the presence of a described feature, integer, step, or component. It does not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof. The term “and / or” should be interpreted as also meaning “both” and each as an alternative.

[0022] It should also be understood that although the terms first, second, etc., may be used herein to describe various elements or features, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the embodiments, a first signal may be referred to as a second signal, and similarly, a second signal may be referred to as a first signal. Both the first signal and the second signal are signals, but they are not the same signal.

[0023] Figure 1 The general illustration depicts a vehicle 100 in the form of an automobile equipped with an Automated Driving System (ADS) 102, which can be used to control and navigate the vehicle without supervision, or to assist the driver during driving. The ADS 102 may include a neural network (NN) 104. As illustrated, the neural network 104 can be trained, for example, in an external data processing device 118 located outside the vehicle 100, such as a server. By training the neural network 104 outside the vehicle 100, additional computing power can be provided, and furthermore, sensor data from several vehicles can be retrieved and considered during training.

[0024] As illustrated, training of the neural network 104 can be achieved using a device 106 comprising a graphics processing unit (GPU) 108, a central processing unit (CPU) 110, and a disk 112. Control circuitry 115 can be communicatively connected to these components. In addition to controlling the operation of these components, this circuitry can also be arranged to monitor GPU utilization values, CPU utilization values, and / or disk utilization values. Although not illustrated, multiple GPUs, multiple CPUs, and / or multiple disks can be used.

[0025] In this context, the term "CPU" should be interpreted broadly as a device used to perform general computing tasks such as reading data from and writing data to a disk. The term "GPU" should be interpreted broadly as a device configured to process large amounts of data in parallel, thereby making it suitable for training neural networks.

[0026] Communication unit 116a can be provided in vehicle 100, and another communication unit 116b can be provided in external data processing device 118, so that neural network 104 can be transmitted from external data processing device 118 to vehicle 100 once trained. As illustrated, on-board GPU 120, on-board CPU 122, and on-board disk 124 can be provided in vehicle 100. Because less computing power is required during inference, the on-board GPU 120 can have less computing power compared to the GPU of the external data processing device used during training.

[0027] like Figure 1 As illustrated in the diagram, the advantage of training the neural network at external data processing device 118 is the ability to utilize GPUs with increased computing power. If high-resolution sensors are used, meaning the sensors are configured to generate large amounts of sensor data, the risk of CPU and I / O limitations increases. Training a neural network typically requires loading sensor data from disk and decoding it on the CPU before transferring it to the GPU for forward and backward propagation through the neural network. In onboard evaluation scenarios, small neural networks are often used—neural networks with, for example, fewer layers, fewer neurons per layer, fewer parameters, and a shallower architecture compared to those typically running on servers. Having such small networks reduces memory and computational requirements. Therefore, CPU and I / O limitations can become more pronounced when training such small networks with large amounts of sensor data.

[0028] If data loading and neural network operations are run asynchronously—that is, while the CPU is preparing the next data sample on the GPU while the GPU is running the neural network—IO constraints and CPU constraints can lead to frequent GPU idle time or other forms of underutilization. To use the GPU more effectively, this idle time can be used to generate synthetic sensor data.

[0029] During the training of neural network 104, it is possible to identify whether CPU-limited or I / O-limited conditions are relevant. If neither CPU-limited nor I / O-limited conditions exist, the neural network can be trained using sensor data captured by vehicle sensors such as sensors 114a to 114d. On the other hand, if the GPU is idle due to either CPU-limited or I / O-limited conditions, the GPU can be configured to generate synthetic sensor data and trained on this synthetic sensor data. For example, scene flow estimation can be used to generate the synthetic sensor data, which can be based on neural radiation fields (NeRF), Gaussian scatter plots, diffusion models, or general adversarial networks (GANs). In addition to being able to use the GPU more efficiently (i.e., reducing its idle time), the synthetic sensor data can also be generated to fill gaps. NeRF can also be generated without using scene flow estimation. In general, combinations of sensor data that are not adequately covered by the training data generated from the sensor data can be identified, referred to herein as gaps, and once these gaps are identified, gap analysis data (i.e., information about gaps in the training data) can be used as input when determining the combinations of sensor data to be covered by the synthetic sensor data. The most common cases of insufficient coverage are sensor combinations linked to traffic law violations, such as those involving a car driving against traffic on a one-way street. Another case involves traffic law violations not committed by the car's driver (which could be a human driver or a non-human driver in this context), where a pedestrian unexpectedly appears in front of the car. These edge cases, sometimes called outliers, can be considered by guiding the generation of synthetic sensor data to ensure their coverage.

[0030] Because monitoring GPU utilization can indirectly identify the aforementioned CPU-limited and I / O-limited states—that is, GPU limitations due to CPU-related or disk-related constraints—monitoring CPU utilization values ​​can determine whether neural networks are trained based on sensor data or synthetic sensor data. If the GPU utilization value is above a GPU utilization threshold, the sensor data can be used to train the neural network. In other words, if the GPU is utilized, it can be concluded that the CPU or disk is not limiting the GPU. However, if the GPU utilization value is below the GPU utilization threshold (also referred to as the GPU threshold in this paper), it can be concluded that the GPU is not fully utilized and is therefore limited by the CPU and / or disk. As an alternative to having the CPU utilization value relate to the degree to which the GPU is currently utilized, the value can relate to the degree to which the GPU will be utilized, as reflected by the currently queued data samples to be provided to the GPU (i.e., data samples to be loaded into the GPU).

[0031] See below for reference. Figure 2Further, to increase reliability and avoid the possibility that GPU utilization values ​​falling below a certain threshold are due to reasons other than CPU limiting the GPU, CPU utilization values ​​can also be monitored and compared to a CPU utilization threshold (referred to herein as the CPU threshold). By adding this extra step, if the GPU utilization value is below the GPU utilization threshold and the CPU utilization value is above the CPU utilization threshold, synthetic sensor data is generated. Similarly, disk utilization values ​​can be monitored and compared to a disk utilization threshold (referred to herein as the disk threshold). By adding this extra step, if the GPU utilization value is below the GPU utilization threshold and the disk utilization value is above the disk utilization threshold, synthetic sensor data is generated.

[0032] GPU utilization values ​​reflect the current level of GPU usage, but they can also reflect future GPU utilization, or in other words, the predictable utilization rate. This future GPU utilization can be determined or predicted by considering the currently queued data samples to be provided to the GPU. As an effect of considering future GPU utilization, a GPU threshold can be set above 100% of GPU capacity. The term "sensor data" as used in this paper should be understood to encompass sensor data originating from the vehicle, such as point clouds from LiDAR sensors or image data from cameras. As the name suggests, the term "synthetic sensor data" as used in this paper should be understood to encompass data in the same format as sensor data but generated by a data processing model. For example, synthetic sensor data can be generated indirectly using a structure-of-motion (SfM) model or a generative AI model. Furthermore, synthetic sensor data can be generated using neural radiation fields (NeRF) or Gaussian scatter plots. Sensor data can be retrieved from sensors provided in multiple vehicles, collected over time under various conditions.

[0033] The methods described above are not limited to a single GPU; multiple GPUs can also be used. Similarly, multiple disks and multiple CPUs can be used.

[0034] Figure 2 This is a flowchart illustrating a method 200 for training a neural network for ADS. Consistent with the above description and as... Figure 1 As illustrated in the diagram, the training data may involve a CPU for processing input / output (I / O) operations associated with reading sensor data from and writing sensor data to disk, a disk for storing sensor data, and a GPU for training the neural network of ADS using the sensor data provided to the GPU by the CPU.

[0035] The method may include monitoring a GPU utilization value 202, wherein the GPU utilization value relates to the degree to which the GPU is currently occupied or to the degree to which the GPU will be occupied, as reflected by the currently queued data samples to be provided to the GPU as described above. If the GPU utilization value is above a GPU utilization threshold, the following steps may be performed: reading sensor data 208 from disk 112 using CPU 110, transferring the sensor data from CPU 110 to GPU 108, and training a neural network 212 by GPU 108 based on the sensor data. In other words, the GPU is not limited by the CPU or disk, and for this reason, sensor data can be retrieved from the disk to the GPU via the CPU. On the other hand, if the GPU utilization value is below a GPU utilization threshold, the following steps may be performed: generating synthetic sensor data 214 using GPU 108, and training a neural network 216 by GPU 108 based on the synthetic sensor data. In other words, synthetic sensor data may be generated if there is free capacity available in the GPU.

[0036] Furthermore, to ensure that the generated synthetic sensor data can fill the gaps, as described above, the method may include determining 218 gap analysis data, wherein the gap analysis data can provide information about sensor combinations not fully covered by the current sensor data. For example, the aforementioned edge cases could constitute such a sensor combination.

[0037] As described above, method 200 may further include monitoring 220 CPU utilization values. By monitoring CPU utilization values, CPU limitation can be implicitly determined using GPU utilization values, and can also be explicitly determined using CPU utilization values. By combining both, a more reliable assessment of whether CPU limitation exists can be achieved. Therefore, as illustrated, if the GPU utilization value is below a GPU threshold and if the CPU utilization value is above a CPU threshold, steps 214 of generating synthetic sensor data and 216 of training a neural network based on the synthetic sensor data can be performed. Furthermore, in the same manner as I / O limitation can be implicitly determined by monitoring GPU utilization values, this can also be explicitly determined by monitoring 222 disk utilization values. To provide a more reliable assessment of I / O limitation, steps 214 of generating synthetic sensor data and 216 of training a neural network based on the synthetic sensor data can be performed if the GPU utilization value is below a GPU threshold and if the disk utilization value is above a disk threshold.

[0038] A CPU threshold can be a CPU capacity limit, i.e., maximum utilization. Alternatively, it can be set above the CPU capacity limit, i.e., above the maximum utilization. In the latter case, for example, if the CPU capacity threshold is set to 110% of the CPU capacity limit, the CPU will act as a bottleneck, and there will be CPU throttling, but to an acceptable degree. In other words, the CPU threshold can be set so that CPU throttling is not completely avoided, but rather its impact is kept within an acceptable level.

[0039] The advantage of having the synthetic sensor data generated by the GPU is that the synthetic sensor data can be dynamically generated and processed in memory during training, thus avoiding the steps of writing and reading sensor data from disk. For example, the data can be stored in volatile memory such as RAM, rather than non-volatile memory such as disk.

[0040] Method 200 is preferably derived from, for example Figure 1 The external data processing device 118 illustrated in the figure executes a computer-implemented method 200. The processing system may, for example, include one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that, when executed by the one or more processors, perform the steps, services, and functions of the method 200 disclosed herein.

[0041] Executable instructions for performing these functions may optionally be included in a non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0042] Return to reference Figure 1 These are schematic block diagram representations of apparatus 106. Apparatus 106 may include control circuitry (e.g., one or more processors) configured to perform the functions of the method 200 disclosed herein, wherein the functions may be contained in a non-transient computer-readable storage medium or other computer program product configured for execution by the control circuitry. In other words, apparatus 106 includes one or more memory storage regions comprising program code configured, together with one or more processors, to cause apparatus 106 to perform the method 200 according to any of the embodiments disclosed herein.

[0043] Figure 3 yes Figure 1 A more detailed schematic illustration of vehicle 100 equipped with ADS is shown in the figure. As used herein, “vehicle” means any form of motorized transport. For example, vehicle 100 can be any road vehicle such as (as illustrated herein) an automobile, motorcycle, (freight) truck, bus, etc.

[0044] Device 10 may include control circuitry 11 and memory 12. Control circuitry 11 may physically comprise a single circuit device. Alternatively, control circuitry 11 may be distributed across several circuit devices. As an example, device 10 may share its control circuitry 11 with other components of vehicle 100, such as ADS 102. Furthermore, device 10 may form part of ADS 102, i.e., device 10 may be implemented as a module or feature of ADS. Control circuitry 11 may include one or more processors, microcontrollers, or microprocessors, such as CPUs, GPUs, and disks. One or more processors may be configured to execute program code stored in memory 12 to perform various functions and operations of vehicle 100 in addition to the methods disclosed herein. Processors may be or may include any number of hardware components for performing data or signal processing or for executing computer code stored in memory 12. Memory 12 optionally includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices; and optionally includes non-volatile memory such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 12 may include a database component, an object code component, a script component, or any other type of information structure used to support the various activities of this specification.

[0045] In the illustrated example, memory 12 further stores map data 308. Map data 308 can be used, for example, by the ADS 102 of vehicle 100 to perform autonomous functions of vehicle 100. Map data 308 may include high-definition (HD) map data. Conversely, even though memory 12 is illustrated as a separate element from ADS 102, it can be provided as an integrated element of ADS 102. In other words, according to the exemplary embodiment, any distributed or local memory device can be used to implement the inventive concept. Similarly, control circuitry 11 can be distributed, for example, such that one or more processors of control circuitry 11 are provided as integrated elements of ADS 102 or any other system of vehicle 100. In other words, according to the exemplary embodiment, any distributed or local control circuitry device can be used to implement the inventive concept. ADS 102 is configured to perform autonomous or semi-autonomous functions and operations of vehicle 100. ADS 102 may include multiple modules, each responsible for a different function of ADS 102.

[0046] Vehicle 100 includes many components typically found in autonomous or semi-autonomous vehicles. It should be understood that vehicle 100 may have... Figure 3 Any combination of the various elements shown. Furthermore, vehicle 100 may include more than... Figure 3The elements shown herein are further elements. Although various elements are shown herein as being located inside vehicle 100, one or more elements may be located outside vehicle 100. For example, map data may be stored in a remote server and accessed by various components of vehicle 100 via communication system 326. Furthermore, although various elements are depicted herein in certain arrangements, as will be readily understood by those skilled in the art, various elements may be implemented in different arrangements. It should be further noted that various elements may be communicatively connected to each other in any suitable manner. Figure 3 The vehicle 100 should be considered only as an illustrative example, as the components of the vehicle 100 can be implemented in several different ways.

[0047] Vehicle 100 further includes a sensor system 320. Sensor system 320 is configured to acquire sensing data about the vehicle itself or its surroundings. Sensor system 320 may, for example, include a Global Navigation Satellite System (GNSS) module 322 (such as GPS) configured to collect geographic location data of vehicle 100. Sensor system 320 may further include one or more sensors 324. Sensors 324 may be any type of airborne sensor such as a camera, LiDAR and RADAR, ultrasonic sensors, gyroscopes, accelerometers, odometers, etc. It should be appreciated that sensor system 320 may also provide the possibility of acquiring sensing data directly or via dedicated sensor control circuitry within vehicle 100.

[0048] Vehicle 100 further includes a communication system 326. Communication system 326 is configured to communicate with external units such as other vehicles (i.e., via vehicle-to-vehicle (V2V) communication protocols), remote servers (e.g., cloud servers), databases, or other external devices, i.e., vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocols. Communication system 318 can communicate using one or more communication technologies. Communication system 318 may include one or more antennas (not shown). Cellular communication technologies can be used for remote communication, such as to remote servers or cloud computing systems. Furthermore, if the cellular communication technology used has low latency, it can also be used for V2V, V2I, or V2X communication. Examples of cellular radio technologies include GSM, GPRS, EDGE, LTE, 5G, 5G NR, and so on, including future cellular solutions. However, in some solutions, short-to-medium range communication technologies such as wireless local area networks (LANs) (e.g., solutions based on IEEE 802.11) can be used for communication with other vehicles near vehicle 100 or with local infrastructure components. ETSI is developing cellular standards for vehicle communications, and 5G is considered a suitable solution, for example, due to its high bandwidth, low latency, and efficient processing of communication channels.

[0049] The communication system 326 can accordingly provide the possibility of sending outputs to and / or receiving inputs from remote locations (e.g., remote operation or control centers) via one or more antennas. Furthermore, the communication system 326 can be further configured to allow various components of the vehicle 100 to communicate with each other. As an example, the communication system can provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. Local communication within the vehicle can also be a wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar medium / short-range technologies.

[0050] Vehicle 100 further includes a control system 328. The control system 328 is configured to control the handling of vehicle 100. The control system 328 includes a steering module 330 configured to control the heading of vehicle 100. The control system 328 further includes a throttle module 332 configured to control the actuation of the throttle valve of vehicle 100. The control system 328 further includes a braking module 334 configured to control the actuation of the brakes of vehicle 100. The various modules of the control system 328 can also receive manual input from the driver of vehicle 100 (i.e., from the steering wheel, accelerator pedal, and brake pedal, respectively). However, the control system 328 can be communicatively connected to the vehicle's ADS 102 to receive instructions on how the various modules of the control system 328 should operate. Therefore, ADS 102 can control the handling of vehicle 100, for example, via a decision and control module 318.

[0051] ADS 102 may include a positioning module 312 or a positioning block / system. The positioning module 312 is configured to determine and / or monitor the geographic location and heading of vehicle 100 and may utilize data from sensor system 320, such as data from GNSS module 322. Alternatively or in combination, the positioning module 312 may utilize data from one or more sensors 324. For improved accuracy, the positioning system may alternatively be implemented as Real-Time Kinematic (RTK) GPS.

[0052] ADS 102 may further include a perception module 314 or a perception block / system 314. The perception module 314 may refer to any known module and / or function, for example, included in one or more electronic control modules and / or nodes of vehicle 100, adapted and / or configured to interpret sensing data related to driving of vehicle 100 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. The perception module 314 may therefore be adapted to rely on and receive input from multiple data sources such as automotive imaging, image processing, computer vision, and / or in-vehicle networks, and to combine with sensing data, for example, from sensor system 320.

[0053] The positioning module 312 and / or the sensing module 314 can be communicatively connected to the sensor system 320 to receive sensing data from the sensor system 320. The positioning module 312 and / or the sensing module 314 can further transmit control commands to the sensor system 320.

[0054] The present invention has been presented above with reference to specific embodiments. However, other embodiments besides those described above are possible and within the scope of the invention. Within the scope of the invention, method steps that are different from those described above, performed by hardware or software, can be provided. Thus, according to an exemplary embodiment, a non-transient computer-readable storage medium is provided storing one or more programs configured to be executed by one or more processors of a vehicle control system, the one or more programs including instructions for performing the method according to any of the embodiments described above. Alternatively, according to another exemplary embodiment, a cloud computing system can be configured to perform any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that jointly perform the methods presented herein under the control of one or more computer program products.

[0055] Generally, computer-accessible media can include any tangible or non-transient storage medium or storage media such as electrical, magnetic, or optical media—for example, a hard disk or CD / DVD-ROM bus-connected to a computer system. As used herein, the terms “tangible” and “non-transient” are intended to describe computer-readable storage media (or “memory”) excluding those that propagate electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrases “computer-readable medium” or “memory.” For example, the terms “non-transient computer-readable medium” or “tangible memory” are intended to encompass types of storage devices that include, for example, random access memory (RAM) that do not necessarily permanently store information. Program instructions and data stored in non-transient form on tangible computer-accessible storage media can be further transmitted via transmission media or signals such as electrical, electromagnetic, or digital signals, which can be transported via communication media such as networks and / or wireless links.

[0056] The processor of the control circuitry 11 (as associated with device 106) may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in memory 12. Device 106 may have an associated memory 12, and memory 12 may be one or more means for storing data and / or computer code for performing or facilitating the various methods described herein. Memory may include volatile or non-volatile memory. Memory 12 may include database components, object code components, script components, or any other type of information structure for supporting the various activities of this specification. According to an exemplary embodiment, any distributed or local storage device may be used with the systems and methods of this specification. According to an exemplary embodiment, memory 12 (e.g., via circuitry or any other wired, wireless, or network connection) may be communicatively connected to processor 11 and includes computer code for performing one or more processes described herein.

[0057] Accordingly, it should be understood that the parts describing the solution can be implemented in vehicle 100, in a system located outside vehicle 100, or in a combination of inside and outside the vehicle; for example, a so-called cloud solution implemented in a server communicating with the vehicle. For instance, sensor data can be sent to an external system, and that system can perform steps to compare the sensor data (movement of another vehicle) with a predefined behavioral model. Different features and steps of the implementation can be combined in combinations other than those described.

[0058] It should be noted that any reference numerals in the drawings do not limit the scope of the claims. The invention can be implemented, at least in part, by both hardware and software means, and several “apparatus” or “units” can be represented by items of the same hardware.

[0059] Although the accompanying drawings may show a specific order of method steps, the order of steps may differ from what is depicted. Furthermore, two or more steps may be performed simultaneously or partially simultaneously. This variation will depend on the chosen software and hardware system and the designer's choices. All such variations are within the scope of this invention. Similarly, software implementation can be accomplished using standard programming techniques based on rule-based logic and other logics to perform various connection steps, processing steps, comparison steps, and decision steps. The embodiments mentioned and described above are given by way of example only and should not be limited to this invention. Other solutions, uses, purposes, and functions within the scope of this invention claimed in the patent claims described below will be apparent to those skilled in the art.

[0060] ADS Definition

[0061] In this context, an Automated Driving System (ADS) refers to a complex combination of hardware and software components designed to control and operate a vehicle without direct human intervention. ADS technology aims to automate various aspects of driving, such as steering, acceleration, deceleration, and monitoring of the surrounding environment. The primary goal of ADS is to enhance safety, efficiency, and convenience in transportation. The range of ADS can vary from basic driver assistance systems to highly advanced automated driving systems, depending on their level of automation, as classified by standards such as SAE J3016. These systems utilize various sensors, cameras, radar, lidar, and powerful computer algorithms to perceive the environment and make driving decisions. The specific capabilities and characteristics / functions of ADS can vary considerably, from systems providing limited assistance to systems that independently handle complex driving tasks under specific conditions.

[0062] Advanced Driver Assistance Systems (ADAS) are technologies that assist drivers during driving, although they do not necessarily provide complete autonomy. ADAS features are often used as building blocks for ADS. Examples include adaptive cruise control, lane keeping assist, automatic emergency braking, and parking assist. They enhance safety and convenience but typically require a certain level of human supervision and intervention. Autonomous Driving (AD), on the other hand, is a technology designed to control and navigate a vehicle without human supervision. Accordingly, the difference between ADAS and AD can be said to lie in the level of autonomy and control. ADAS systems are designed to assist and support the driver, while ADS aims for complete control of the vehicle without continuous human supervision. Accordingly, AD aims for a higher level of autonomy (such as Level 4 and 5 according to SAE international standards), where the vehicle can operate independently in most or all driving scenarios without human intervention. As mentioned earlier, the term "ADS" used in this paper is a collective term encompassing both ADAS and AD. In this context, ADS functions or ADS features can be understood as specific functions or features of the entire ADS stack, such as highway assist features, traffic jam assist features, and route planning features.

[0063] Surrounding environment

[0064] The surrounding environment of a vehicle can be understood as the general area around the vehicle, where objects (such as other vehicles, landmarks, obstacles, etc.) can be detected and identified by the vehicle's sensors (radar, LIDAR, cameras, etc.), i.e., within the sensor range of the vehicle.

[0065] If-term

[0066] As used herein, the term "if" or "in the case of" can be interpreted, depending on the context, as "when," "in the event of," or "in response to." Similarly, the phrases "if it is determined that," "when it is determined that," or "in the case of" can be interpreted, depending on the context, as "when it is determined that," "in response to determining that," "when the event is detected and identified," or "in response to the detection of the event." Accordingly, the phrase "if X equals Y" can be interpreted, depending on the context, as "when X equals Y," "when it is determined that X equals Y," "in response to X equals Y," or "in response to the detection / determination that X equals Y."

[0067] Sensor device

[0068] In the current context, "sensor device" or "sensor" refers to a specialized component or system designed to capture and collect information from the vehicle's surroundings. These sensors play a crucial role in enabling Autonomous Vehicles (ADSs) to perceive and understand their environment, make informed decisions, and navigate safely. Sensor devices are typically integrated into the hardware and software systems of autonomous vehicles to provide real-time data for various tasks such as obstacle detection, localization, road model estimation, and object recognition. Common types of sensor devices used in autonomous driving include LiDAR (Light Detection and Ranging), radar, cameras, inertial measurement units (IMUs), and ultrasonic sensors. LiDAR sensors use laser beams to measure distances and create high-resolution 3D maps of the vehicle's surroundings. Radar sensors use radio waves to determine the distance and relative speed of objects around the vehicle. Camera sensors capture visual data, allowing the vehicle's computer system to identify traffic signs, lane markings, pedestrians, and other vehicles. Ultrasonic sensors use sound waves to measure the proximity of objects. Various machine learning algorithms, such as artificial neural networks, can be used to process the output from sensors to understand the environment.

Claims

1. A method (200) for training an automated driving system (ADS) (102) for a vehicle (100), the training being performed using the following: A central processing unit (CPU) (110) is configured to process input / output (I / O) operations associated with reading sensor data from and writing sensor data to the disk (112). The disk (112) is used to store the sensor data, wherein, The sensor data originates from sensors (114a to 114d), and A graphics processing unit (GPU) (110) is used to train the neural network (104) of the ADS (102) using the sensor data provided to the GPU (108) via the CPU (110). The method includes: Monitor (202) GPU utilization values, wherein the GPU utilization values ​​relate to the degree to which the GPU is currently occupied or to the degree to which the GPU will be occupied as reflected by current queued data samples to be provided to the GPU; If the GPU utilization value is above the GPU utilization threshold, then: The sensor data is read (208) from the disk (112) using the CPU (110); The sensor data is transferred (210) from the CPU (110) to the GPU (108); and The neural network (104) is trained (212) by the GPU (108) based on the sensor data; and If the GPU utilization value is below the GPU utilization threshold, then: Synthesized sensor data is generated (214) using the GPU (108); and The neural network (104) is trained (216) by the GPU (108) based on the synthesized sensor data.

2. The method according to claim 1, wherein, The synthesized sensor data is dynamically generated and processed in memory during training, thereby avoiding steps involving writing the sensor data to the disk (112) and reading the sensor data from the disk (112).

3. The method according to claim 1, further comprising: Monitor (220) CPU utilization values, wherein the CPU utilization values ​​relate to the extent to which the CPU is occupied while decoding sensor data retrieved from the disk (112). If the GPU utilization value is below the GPU threshold and the CPU utilization value is above the CPU threshold, then the following steps are performed: Synthesized sensor data is generated (214) using the GPU (108); and The neural network (114) is trained (216) by the GPU (108) based on the synthesized sensor data.

4. The method according to claim 1, further comprising: Monitor (222) disk utilization values, wherein the disk utilization values ​​relate to the extent to which the disk is occupied by writing and reading sensor data; If the GPU utilization value is below the GPU threshold and the disk utilization value is above the disk threshold, then perform the following steps: Synthesized sensor data is generated (214) using the GPU (108); and The neural network (114) is trained (216) by the GPU (108) based on the synthesized sensor data.

5. The method according to claim 1, wherein, The GPU (108), the CPU (110), and the disk (112) are placed in the external processing device (118), and the neural network (114) is provided to the ADS (102) of the vehicle once trained.

6. The method of claim 1, further comprising: If the GPU utilization value is below the GPU threshold, then: (218) Gap analysis data is determined, wherein the gap analysis data represents combinations of sensor data that are not adequately covered by the training data generated based on the sensor data. The synthesized sensor data is generated by the GPU based on the gap analysis data.

7. The method according to claim 6, wherein, The sensor data combination relates to situations where the driver or other road user of the vehicle (100) violates traffic regulations.

8. The method according to claim 1, wherein, The synthesized sensor data is image data, and the synthesized sensor data is generated by the GPU using scene flow estimation.

9. The method according to claim 8, wherein, The synthesized sensor data is generated by the GPU using the scene flow estimation, wherein the scene flow estimation is a function of the gap analysis data.

10. The method according to claim 8 or 9, wherein, The scene flow estimation is based on Neural Radiation Field (NeRF), Gaussian scatter plot, diffusion model, or General Adversarial Network (GAN).

11. The method according to claim 1, wherein, The sensors (114a to 114d) include one or more of the following types: light detection and ranging (LiDAR) sensors, radio detection and ranging (radar) sensors, ultrasonic sensors, inertial measurement units (IMUs), and image sensors.

12. The method according to claim 1, wherein, The neural network is placed in the vehicle and forms part of an automated driving system (ADS), which is configured to control and operate the vehicle without direct human intervention.

13. An apparatus (106) comprising: A disk (112) is used to store the sensor data generated by the sensors (114a to 114d), wherein the sensor data reflects physical changes around the vehicle (100); A central processing unit (CPU) (110) is configured to process input / output (I / O) operations associated with reading sensor data from and writing sensor data to the disk (112); and Graphics processing unit (GPU) (108) for training a neural network (104) of the ADS (102) of the vehicle (100) using the sensor data provided to the GPU (108) via the CPU (110); and A control circuit, communicatively connected to the disk, the CPU, and the GPU, is configured to monitor GPU utilization values, and: If the GPU utilization value is above the GPU threshold, then: The sensor data is read from the disk (112) using the CPU (110); The sensor data is transmitted from the CPU (110) to the GPU (108); and The neural network (104) is trained by the GPU (108) based on the sensor data; and If the GPU utilization value is below the GPU threshold, then: Synthesized sensor data is generated using the GPU (108), and The neural network (104) is trained by the GPU (108) based on the synthesized sensor data.

14. The apparatus (106) according to claim 13, wherein, The sensors (114a to 114d) include one or more of the following types: light detection and ranging (LiDAR) sensors, radio detection and ranging (radar) sensors, ultrasonic sensors, inertial measurement units (IMUs), and image sensors.

15. A computer program product capable of being loaded into the memory of a processing unit and comprising instructions that, when executed by the processing unit, perform the steps of the method according to any one of claims 1 to 12.

Images