Active learning for object classification in visual perception tasks in vehicles

Abstract

The embodiment of the invention relates to active learning for object classification in a visual perception task in a vehicle. Automobile sensor data is provided as input for a machine learning algorithm trained to classify objects in the automobile sensor data. Based on the input, a machine learning algorithm determines a class of objects. Then, a potential spatial vector corresponding to the activation value within the potential space of the machine learning algorithm is obtained, and at least one potential spatial vector variance is calculated. If at least one potential spatial vector variance exceeds a variance threshold, determining a potential spatial vector distance quotient, the potential spatial vector distance quotient indicates a distance of the potential spatial vector to a closest object class potential spatial vector of the same object class relative to a distance of the potential spatial vector to a closest potential spatial vector of a different object class. Finally, automobile sensor data is provided to an object classifier based on the potential spatial vector distance quotient.

Description

Technical Field

[0001] The present invention relates generally to active learning, and more particularly to active learning in the context of perception tasks in a vehicle configured to provide at least partial driving automation. Background Art

[0002] In order to enable at least partial driving automation, the vehicle needs to accurately perform automotive perception tasks, such as object classification, object detection, or semantic segmentation. These perception tasks are usually performed by machine learning algorithms, which need to be trained on large data sets, and the machine learning algorithms can be further improved even when the vehicle is deployed in traffic. One method for ensuring that the machine learning algorithm accurately performs the automotive perception task is to train the machine learning algorithm with a large labeled data set (i.e., a data set indicating the results of the corresponding automotive perception task). However, since such labeling may be performed manually, it may be costly to generate a large labeled data set for training the machine learning algorithm to achieve at least partial driving automation. To overcome this problem, active learning can be used, that is, a given machine learning algorithm can be inferred on unlabeled data, and a subset of the unlabeled data can be required to be labeled based on active learning criteria. However, it is usually necessary to determine the active learning criteria in a way that improves the given machine learning algorithm, and in the context of at least partial driving automation, it is necessary to be able to achieve the accuracy level of the machine learning method required for at least partial driving automation.

[0003] It is therefore an object of the present disclosure to provide active learning criteria that enable training and improvement of machine learning algorithms configured to perform automotive perception tasks in a manner that ensures the accuracy required for at least partial driving automation. Summary of the invention

[0004] To achieve this purpose, the present disclosure provides a method configured to enable active learning of object classification in a visual perception task in a vehicle configured to provide at least partial driving automation. The method provides vehicle sensor data to a machine learning algorithm. The machine learning algorithm has been trained to classify objects in the environment of the vehicle based on the training vehicle sensor data. The vehicle sensor data is used as an input to the machine learning algorithm. The output of the machine learning algorithm corresponds to an object class determined by the machine learning algorithm based on the vehicle sensor data. The method also obtains a latent space vector from the machine learning algorithm. The latent space vector corresponds to an activation value within a latent space of the machine learning algorithm. The latent space includes all activation values ​​of the machine learning algorithm without activation values ​​corresponding to the input and output. The method also calculates at least one latent space vector variance of the latent space vector based on the object class latent space vector. The object class latent space vector corresponds to previous vehicle sensor data determined to correspond to the object class of the latent space vector. If at least one latent space vector variance exceeds a variance threshold, the method determines a latent space vector distance quotient. The latent space vector distance quotient indicates a distance of the latent space vector to a closest object class latent space vector in the object class latent space vector relative to a distance of the latent space vector to a closest latent space vector determined to correspond to an object class different from the object class of the latent space vector. Finally, the method provides the vehicle sensor data to an object classifier based on the latent space vector distance quotient.

[0005] The present invention also provides an automobile control unit. The automobile control unit includes at least one processing unit and a memory coupled to the at least one processing unit and configured to store machine-readable instructions. The machine-readable instructions cause the at least one processing unit to provide automobile sensor data to a machine learning algorithm. The machine learning algorithm has been trained based on the training automobile sensor data to classify objects in the environment of the vehicle. The automobile sensor data is used as an input to the machine learning algorithm. The output of the machine learning algorithm corresponds to an object class determined by the machine learning algorithm based on the automobile sensor data. The machine-readable instructions also cause the at least one processing unit to obtain a latent space vector from the machine learning algorithm. The latent space vector corresponds to an activation value within a latent space of the machine learning algorithm. The latent space includes all activation values ​​of the machine learning algorithm without activation values ​​corresponding to the input and output. The machine-readable instructions also cause the at least one processing unit to calculate at least one latent space vector variance of the latent space vector based on the object class latent space vector. The object class latent space vector corresponds to previous automobile sensor data determined to correspond to the object class of the latent space vector. If at least one latent space vector variance exceeds a variance threshold, the machine-readable instructions also cause the at least one processing unit to determine a latent space vector distance quotient. The latent space vector distance quotient indicates a distance of the latent space vector to a closest object class latent space vector in the object class latent space vectors relative to a distance of the latent space vector to a closest latent space vector determined to correspond to an object class different from the object class of the latent space vector. Finally, the machine-readable instructions further cause the at least one processing unit to provide the vehicle sensor data to the object classifier based on the latent space vector distance quotient.

[0006] The present disclosure also provides a vehicle including the above-mentioned automobile control unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Examples of the present disclosure will be described with reference to the following drawings, wherein like reference numerals represent like elements.

[0008] Figure 1 A flow chart illustrating a method according to an example of the present disclosure, the method being configured to enable active learning of object classification in a visual perception task in a vehicle configured to provide at least partial driving automation;

[0009] Figure 2 A vehicle according to an example of the present disclosure is illustrated; and

[0010] Figure 3 An automotive control unit according to an example of the present disclosure is illustrated.

[0011] It should be understood that the above-mentioned drawings are by no means intended to limit the present disclosure. On the contrary, these drawings are provided to help understand the present disclosure. It will be readily understood by those skilled in the art that the various aspects of the present invention shown in one figure may be combined with the various aspects of another figure, or may be omitted, without departing from the scope of the present disclosure. DETAILED DESCRIPTION

[0012] The present disclosure generally provides a method configured to enable active learning of object classification in a visual perception task in a vehicle configured to provide at least partial driving automation. In addition, the present disclosure generally provides an automotive control unit configured to execute instructions implementing the above method and a vehicle including the above automotive control unit.

[0013] To enable active learning of object classification in a visual perception task in a vehicle, a machine learning algorithm is first trained with training car sensor data to determine object classes in the car sensor data, i.e., data captured by various car sensors of the vehicle. During training of the machine learning algorithm, if the training data is fully or partially unlabeled, and during subsequent deployment of the machine learning algorithm during operation of the vehicle on a road, the machine learning algorithm may perform data collection for active learning. More specifically, the machine learning algorithm performs data collection for active learning using a two-step approach based on latent space vectors obtained from a latent space of the machine learning algorithm.

[0014] In a first step, at least one latent space vector variance of a latent space vector relative to a latent space vector is determined to correspond to the same object class as the latent space vector is calculated. For example, the at least one latent space vector variance can be the variance of the magnitude of the latent space vector, or the variance of some or all components of the latent space vector. If at least one latent space vector variance exceeds a variance threshold, the latent space vector is considered to be out of distribution (OOD).

[0015] If at least one latent space vector variance exceeds a variance threshold, a latent space vector distance quotient is calculated in a second step. The latent space vector distance quotient indicates whether the latent space vector is closer to a latent space vector determined to correspond to another object class than to a latent space vector determined to correspond to the same object class as the latent space vector. Based on the latent space vector distance quotient, the automotive sensor data corresponding to the latent space vector is provided to an object classifier, which labels the data and returns the automotive sensor data together with the label to the machine learning algorithm to improve the performance of the perception task performed by the machine learning algorithm.

[0016] By providing the object classifier with the automotive sensor data based on the latent space vector distance quotient, the automotive sensor data is selected for active learning, the data corresponding to an alternate region of an object class determined due to the distance of the closest latent space vector in the same object class as the latent space vector. In addition, due to this approach, the automotive sensor data is selected for active learning, the data corresponding to a decision boundary between object classes due to the proximity of the latent space vector to latent space vectors of different object classes.

[0017] In summary, performing active learning using a two-step approach can select automotive sensor data to be labeled by an object classifier that is object-oriented and can increase the coverage of alternate regions of determined object classes and can also increase the coverage at decision boundaries (i.e., boundaries between object classes).

[0018] This general concept will be explained with reference to the accompanying drawings, Figure 1 A flow chart of a method 100 is provided that is configured to enable active learning of object classification in a visual perception task in a vehicle that is configured to provide at least partial driving automation. In addition, Figure 2 A vehicle according to the present disclosure is illustrated, Figure 3 An automobile controller configured to perform method 100 is illustrated.

[0019] It should be understood that Figure 1 The dashed boxes in illustrative examples illustrate optional steps of method 100 .

[0020] The method 100 is configured to enable a vehicle, such as Figure 2 Active learning of object classification in a visual perception task in a vehicle 200 configured to provide at least partial driving automation.

[0021] In the context of the present disclosure, vehicle 200 refers to any type of motor vehicle configured to transport people and / or goods. The motor of vehicle 200 can be any type of motor, such as an electric motor or an internal combustion engine. Vehicle 200 can be, for example, a passenger car. However, it should be understood that vehicle 200 can also be a bus, a truck, or any other type of vehicle, which includes one or more sensors 210 and a vehicle control unit 300 that enables vehicle 200 to provide at least partial driving automation. That is, the vehicle control unit 300 and the one or more sensors 210 are configured to enable at least partial driving automation, i.e., level 2 defined in SAE International Standard J3016. Therefore, method 100 enables accurate execution of object classification in visual perception tasks within the vehicle control unit 400, so as to enable longitudinal and lateral safety control of vehicle 200. It should be understood that method 100, vehicle control unit 300, and one or more sensors 210 can enable higher levels of automation, for example, up to level 5 defined in SAE International Standard J3016.

[0022] One or more sensors 210 may be configured to capture automotive sensor data indicating the environment of the vehicle 200, which may provide environmental perception, thereby enabling at least partial driving automation. For example, one or more sensors 210 may provide information about the location and size of other vehicles, or about road markings, to the vehicle 200, which is extracted from the automotive sensor data based on object classification performed by a machine learning algorithm. To this end, one or more sensors 210 may be radar sensors, which may be configured to emit radio waves so as to determine the distance, angle, and speed of objects around the vehicle based on reflected radio waves. One or more sensors 210 may be light detection and ranging (LIDAR) sensors, which may be configured to emit laser beams so as to determine the distance, angle, and speed of objects around the vehicle 200 based on reflected laser beams. One or more sensors 210 may be cameras that capture images of the environment of the vehicle. One or more sensors 210 may be thermal imaging cameras that capture images of the environment of the vehicle 200 based on infrared radiation. It should be understood that LIDAR sensors, radar sensors, or cameras are provided only as examples of sensor types of one or more sensors 210. For example, one or more sensors 210 may also be ultrasonic sensors. More generally, the one or more sensors 210 may be any type of sensor capable of capturing sensor data indicative of the environment of the vehicle 200. It will also be understood that the one or more sensors 210 may include multiple sensors of various types of sensors. In addition, the same type of one or more sensors 210 may exhibit different properties, such as by being configured to capture sensor data at different ranges (such as short range, mid-range, and long range). For example, the vehicle 200 may include three short-range radar sensors located at the front and rear of the vehicle 200, respectively, a mid-range to long-range radar sensor located at the rear of the vehicle 200, a LIDAR sensor located at the front of the vehicle 200, a rear camera located at the rear of the vehicle 200, a front camera located at the front of the vehicle, a front camera located at the rearview mirror, and a rear short-range to mid-range radar sensor located in each door-mounted exterior mirror. It should be understood that the vehicle 200 may include more than one short-range radar sensor. Figure 2 More or fewer automotive sensors may be shown and discussed in the examples above.

[0023] The following will be combined Figure 3 The vehicle control unit 300 is discussed in more detail.

[0024] In the context of the present disclosure, a visual perception task refers to any type of task that identifies one or more object classes within the automotive sensor data captured by one or more sensors 210. The visual perception task may, for example, identify within the automotive sensor data provided by a camera included in the vehicle 200 whether the vehicle 200 is located on a controlled access highway, a limited access, a main road, a local road, or a parking lot. In this case, the one or more object classes correspond to the type of road on which the vehicle 200 may be located. In addition, the visual perception task may, for example, identify other vehicles and vehicle types, pavement markings and pavement marking types, road signs and road sign types, vulnerable road users (VRUs), and traffic lights and the indicated state of traffic lights within the automotive sensor data provided by a LIDAR sensor and multiple cameras included in the vehicle 200. Therefore, one or more object classes may correspond to any possible road user, road traffic control device and pavement marking, and any other type of possible element near the vehicle 200 that is related to providing at least partial driving automation. More generally, a visual perception task may thus be any perception task of determining a class of objects in the vicinity of the vehicle 200 , wherein the objects relate both to the determination of the general environment of the vehicle 200 and to the determination of individual elements in the vicinity of the vehicle 200 .

[0025] In the context of a visual perception task, active learning is therefore understood in the context of this disclosure to refer to a machine learning algorithm that performs a visual perception task and requests an object classifier to label the automotive sensor data, i.e. determine the object classes within the automotive sensor data.

[0026] The machine learning algorithm can be any type of machine learning algorithm that has been trained to classify objects in the vehicle's environment based on training vehicle sensor data, i.e., has been trained to perform the above-mentioned visual perception task. The training vehicle sensor data can be unlabeled, partially labeled, or fully labeled. In other words, in addition to the vehicle sensor data, the training vehicle sensor data can also include corresponding object classes. However, in view of the active learning functionality discussed above and described in detail below, the training vehicle sensor data does not need to be fully labeled. The machine learning algorithm can be a feature extractor configured to extract objects from the vehicle sensor data and determine the object class of the extracted object. To this end, the machine learning algorithm can be, for example, an artificial neural network (ANN) or an autoencoder.

[0027] In step 110, method 100 provides the vehicle sensor data to the machine learning algorithm. That is, the vehicle sensor data is used as an input to the machine learning algorithm, and the output of the machine learning algorithm corresponds to the object class determined by the machine learning algorithm based on the vehicle sensor data.

[0028] In step 120, method 100 obtains a latent space vector from a machine learning algorithm. The latent space vector corresponds to activation values ​​within the latent space of the machine learning algorithm. The latent space includes all activation values ​​of the machine learning algorithm without the activation values ​​corresponding to the input and output. Using ANN as an example of a machine learning algorithm, the latent space may, for example, include the activation values ​​of all neurons of all layers except the input layer and the output layer. In this example, the latent space vector may therefore include all activation values ​​of the intermediate layers of the ANN. The intermediate layer may, for example, be the layer of the ANN before the output layer. Using an autoencoder as another example of a machine learning algorithm, the latent space may again include all activation values ​​of all neurons between the input layer and the output layer, with the latent space vector corresponding to the code, i.e., the bottleneck, of the autoencoder.

[0029] Since the latent space vector corresponds to the activation values ​​in the latent space of the machine learning algorithm, it can be understood that the components of the latent space vector can be directly a subset of the activation values ​​in the latent space, such as the activation values ​​of all neurons of the layer of the ANN, or a value obtained based on the processing of a subset of the activation values ​​in the latent space. For example, the components of the latent space vector can reflect the activation probability of a given neuron, which can be derived from the corresponding activation value and the corresponding output of the machine learning algorithm and the deviation of the corresponding object class from the normal distribution of the determined object class. The lower the probability of a given neuron being activated, the higher the probability that the activation of the given neuron leads to the determination of the object class (which can be OOD) of the corresponding automobile sensor data. Therefore, in some examples of the present disclosure, the components of the latent space vector can correspond to the preprocessed activation values ​​of the latent space of the machine learning algorithm to improve the OOD detection of the automobile sensor data, and thereby improve the selection of automobile sensor data for active learning.

[0030] To this end, step 120 may include step 121 , in which method 100 derives a latent space vector from the activation values ​​in a latent space of the machine learning algorithm based on the activation probabilities of the activation values.

[0031] In step 130, method 100 calculates at least one latent space vector variance of the latent space vector based on the object class latent space vector. The object class latent space vector corresponds to previous automotive sensor data determined to correspond to the object class of the latent space vector. In other words, as part of step 130, method 100 may determine a latent space vector whose corresponding automotive sensor data is pre-classified as belonging to the same object class as the latent space vector during the visual perception task performed by the machine learning algorithm. The object class latent space vector may be a set of latent space vectors determined for each object class during the training of the machine learning algorithm, or may be continuously updated based on further object classification performed with automotive data continuously processed by the machine learning algorithm during operation of the vehicle 200. As part of step 130, method 100 may calculate the latent space vector variance based on the size of the latent space vector relative to the size of the object class latent space vector. As part of step 130, method 100 may also calculate the variance of some or all components of the latent space vector relative to corresponding some or all components of the object latent space vector. At least the calculation of the latent space vector variance enables the determination of whether the latent space vector is OOD. That is, step 130 corresponds to the first step of the two-step method of active learning selection described above. In the example of the present disclosure, where the latent space vector includes components derived from activation values ​​based on activation probabilities, as described above, step 130 and the components of the latent space vector derived accordingly together correspond to the first step of the two-step method of active learning selection described above.

[0032] Method 100 may include step 140, wherein method 100 generates a reduced latent space vector, the latent space vector including components of the latent space vector that exceed a variance threshold. The variance threshold may be any value indicating that a given latent space vector variance for a given object class is OOD. In other words, in addition to using at least one latent space vector variance for OOD detection, the at least one latent space vector variance may also be used to reduce the dimensionality of the latent space vector to a dimension determined to be OOD, thereby reducing the computational workload of subsequent steps of method 100.

[0033] In step 150, the method 100 determines the above-mentioned latent space vector distance quotient. The latent space vector distance quotient indicates the distance of the latent space vector to the closest object class latent space vector in the object class latent space vector relative to the distance of the latent space vector to the closest latent space vector corresponding to the object class different from the object class of the latent space vector. In other words, the method 100 determines the shortest distance, i.e., the closest distance, of the latent space vector to the object class latent space vector, i.e., the latent space vector whose corresponding automobile sensor data is pre-classified as belonging to the same object class as the latent space vector during the visual perception task performed by the machine learning algorithm. Obviously, the object class latent space vector is therefore one of the object class latent space vectors used to calculate at least one latent space vector variance in step 130. Similarly, the method 100 determines the shortest distance, i.e., the closest distance, of the latent space vector to the latent space vector whose corresponding automobile sensor data is pre-classified as belonging to the object class different from the latent space vector during the visual perception task performed by the machine learning algorithm. For example, both distances can be the Euclidean distance between the latent space vector and the closest object class latent space vector. Step 150 accordingly implements the second step of the two-step active learning method and thereby ensures that vehicle sensor data is selected for active learning which corresponds both to alternate regions of object classes determined due to the distance to the closest object class latent space vector and to decision boundaries between different object classes due to the proximity of the latent space vector to latent space vectors of different object classes.

[0034] To determine the latent space vector quotient, method 100 may include step 151, wherein method 100 may calculate the latent space vector distance quotient based on dividing the inner class distance by the outer class distance. The inner class distance may correspond to the distance between the latent space vector and the object class latent space vector in the object class latent space vector that is closest to the latent space vector. The outer class distance may correspond to the distance between the latent space vector and the different object class latent space vector that is closest to the latent null vector. The different object class latent space vector may correspond to an object class different from the object class corresponding to the latent space vector.

[0035] In an example of the present disclosure in which method 100 generates a reduced latent space vector, method 100 may determine a latent space vector distance quotient based on the reduced latent null vector. That is, in an example in which method 100 generates a reduced latent space vector, the latent space vector distance quotient indicates the distance of the reduced latent space vector to the closest reduced object class latent space vector in the object class latent space vector relative to the distance of the reduced latent space vector to the closest reduced latent space vector determined to correspond to an object class different from the object class of the latent space vector. In other words, the determination of the latent space vector distance quotient may be based solely on the dimensionality of the reduced latent space vector, which may reduce the computational workload of determining the latent space vector distance quotient.

[0036] Method 100 may include step 160, wherein method 100 normalizes the latent space vector distance quotient to a normalized latent space vector distance quotient having a value within a distance quotient range. Another method 100 may include step 170, wherein method 100 inverts the normalized latent space vector distance quotient. Normalizing the latent space vector quotient and inverting the normalized latent space vector distance quotient can improve the comparability and further processing of the latent space vector distance quotient. Normalizing the latent space vector distance quotient to the normalized latent space vector distance quotient can be based on one of the following: a sigmoid function, a hyperbolic tangent function, a Soboleva modified hyperbolic tangent function, or a Gaussian function.

[0037] In step 180, method 100 provides the car sensor data to the object classifier based on the latent space vector distance quotient. That is, if the latent space vector distance quotient indicates that the classification of the car sensor data appears to be OOD and corresponds to an area with low coverage and / or decision boundaries, the car sensor data can be selected for active learning, that is, the car sensor data can be selected to be labeled. The object classifier can be one of the user of the vehicle 200 and the cloud-based object classification service. More generally, if required, the object classifier can be any entity capable of labeling autonomous data, which can also be called a oracle or annotator. In other words, if the car sensor data is selected for active learning, the car sensor data can be displayed to the user of the vehicle 200, for example, to obtain the classification of the car sensor data from the user. For example. The car sensor data can be displayed to the user of the vehicle 200 on a screen integrated into the dashboard of the vehicle 200, while displaying a request, such as the request for confirming that the traffic control device visible in the car sensor data is indeed a stop sign, or in an alternative solution for indicating what is visible in the car sensor data. If the object classifier is a cloud-based object classification service, such as a classification service of the manufacturer of vehicle 200, the vehicle sensor data can be wirelessly sent to the cloud-based object classification service, which can classify one or more objects visible in the vehicle sensor data, and can provide the labeled vehicle sensor data not only to vehicle 200, but also to all vehicles of the manufacturer to improve the performance of the machine algorithm for visual perception tasks across all vehicles of the manufacturer.

[0038] Providing the vehicle sensor data to the object classifier can be based on a latent space vector distance quotient threshold. That is, if the latent space vector distance quotient exceeds a certain value, the value indicates that the vehicle sensor data needs to be marked because the latent space vector distance quotient indicates that the vehicle sensor data can be OOD and can increase the coverage of the spare area or decision boundary of the object class, then the vehicle sensor data can be provided to the object classifier. Providing the vehicle sensor data to the object classifier can also be based on selecting one or more sets of vehicle sensor data, which have corresponding latent space vector distance quotients, and the latent space vector distance quotient indicates that one or more sets of vehicle sensor data can be more OOD than other sets of vehicle sensor data and can increase the coverage of the spare area or decision boundary of the object class. To this end, method 100 may include steps 181 and 182. In step 181, method 100 may store the latent space vector distance quotient and the corresponding vehicle sensor data, as well as one or more previously determined latent space vector distance quotients and corresponding vehicle sensor data and one or more subsequently determined latent space vector distance quotients and corresponding vehicle sensor data. In step 181 , method 100 may provide one or more of a latent space vector distance quotient, one or more previously determined latent space vector distance quotients, one or more subsequently determined latent space vector distance quotients, and corresponding vehicle sensor data to an object classifier based on a ranking of the corresponding latent space vector distance quotients.

[0039] Figure 3 An automotive control unit 300 is shown configured to perform the method 100. The automotive control unit 300 may include a processor 310, a graphics processing unit (GPU) 320, an automotive processing system 330, a memory 340, a removable storage device 350, a storage device 360, a cellular interface 370, a global navigation satellite system (GNSS) interface 380, and a communication interface 390.

[0040] Processor 310 may be any type of single-core or multi-core processing unit that employs a reduced instruction set (RISC) or complex instruction set (CISC). Exemplary RISC processing units include ARM-based cores or RISC V-based cores. Exemplary CISC processing units include x86-based cores or x86-64-based cores. Processor 310 may execute instructions that cause automotive control unit 300 to perform method 100. Processor 310 may be directly coupled to any component of computing device 300, or may be directly coupled to memory 330, GPU 320, and a device bus.

[0041] GPU 320 can be any type of processing unit optimized for processing graphics-related instructions or more generally for parallel processing of instructions. Thus, GPU 320 can be configured to generate information displays, such as ADAS information or telemetry data, to the driver of the vehicle, for example via a head-up display (HUD) or a display arranged within the driver's field of view. GPU 320 can be coupled to the HUD and / or display via connection 320C. GPU 320 can also execute at least a portion of method 100 to achieve fast parallel processing of instructions related to method 300. It should be noted that in some embodiments, processor 310 can determine that GPU 320 does not need to execute instructions related to method 300. GPU 320 can be directly coupled to any component of the automotive control unit 300, or can be directly coupled to processor 310 and memory 330. In some embodiments, GPU 320 can also be coupled to a device bus.

[0042] The automotive processing system 330 may be any type of system-on-chip configured to provide trillions of operations per second (TOPS) to enable the automotive control unit 300 to implement one or more ADAS while driving. The automotive processing system 330 may interface only with the processor 310, or may interface with other devices via a system bus. For example, the automotive processing system 330 may execute a machine learning algorithm for performing a visual perception task.

[0043] The memory 340 may be any type of fast memory that enables the processor 310, GPU 320, and vehicle processing system 430 to store instructions for fast retrieval during instruction processing and to cache and buffer data. The memory 340 may be a unified memory coupled to the processor 310, GPU 320, and vehicle processing system 330 so that the memory 340 can be allocated to the processor 310, GPU 320, and vehicle processing system 330 as needed. Alternatively, the processor 310, GPU 320, and vehicle processing system 330 may be coupled to separate processor memories 340a, GPU memories 340b, and vehicle processor system memories 340c.

[0044] The removable storage device 350 may be a storage device that is removably coupled to the vehicle control unit 400. Examples include a digital versatile disc (DVD), a compact disc (CD), a universal serial bus (USB) storage device (such as an external SSD), or a magnetic tape. It should be noted that the removable storage device 350 may store data, such as instructions of the method 200 and / or vehicle sensor data, or may be omitted.

[0045] Storage device 360 ​​may be a storage device capable of storing program instructions and other data. For example, storage device 360 ​​may be a hard disk drive (HDD), a solid state disk (SSD), or some other type of non-volatile memory. Storage device 360 ​​may, for example, store instructions of method 100.

[0046] Removable storage device 350 and storage device 360 ​​may be coupled to processor 310 via a system bus. The system bus may be any type of bus system that enables processor 310 and optional GPU 320 and vehicle processing system 330 to communicate with other devices of vehicle control unit 300. The system bus may be, for example, a Peripheral Component Interconnect Express (PCIe) bus or a Serial AT Attachment (SATA) bus.

[0047] The cellular interface 370 may be any type of interface that enables the vehicle control unit 300 to communicate via a cellular network, such as a 4G network or a 5G network.

[0048] The GNSS interface 380 may be any type of interface that enables the vehicle control unit 300 to receive location data provided by a satellite network such as the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), or Galileo. The location data may be used in generating and updating the exit route map 320 and in determining and re-determining exit routes.

[0049] The communication interface 390 may enable the computing device 300 to interface with external devices, either directly or via a network, via connection 380C. The communication interface 380 may, for example, enable the computing device 300 to couple to a wired or wireless network, such as Ethernet, Wifi, a controller area network (CAN) bus, or any bus system suitable for a vehicle. For example, the automotive control unit 300 may be coupled to one or more sensors 210 to receive information about the environment of the vehicle 200 to classify objects near the vehicle 200. The communication interface 390 may also include a USB port or a serial port to enable direct communication with external devices.

[0050] The automotive control unit 300 may be integrated with the vehicle, for example under the cab, under the dashboard, or in the trunk of the vehicle 300 .

[0051] The present invention can be further illustrated by the following examples.

[0052] In one example, a method is configured to enable active learning of object classification in a visual perception task in a vehicle configured to provide at least partial driving automation. The example method provides vehicle sensor data to a machine learning algorithm. The machine learning algorithm has been trained to classify objects in the environment of the vehicle based on training vehicle sensor data. The vehicle sensor data is used as an input to the machine learning algorithm. The output of the machine learning algorithm corresponds to an object class determined by the machine learning algorithm based on the vehicle sensor data. The example method also obtains a latent space vector from the machine learning algorithm. The latent space vector corresponds to an activation value within a latent space of the machine learning algorithm. The latent space includes all activation values ​​of the machine learning algorithm without activation values ​​corresponding to the input and the output. The example method also calculates at least one latent space vector variance of the latent space vector based on an object class latent space vector. The object class latent space vector corresponds to previous vehicle sensor data determined to correspond to the object class of the latent space vector. If the at least one latent space vector variance exceeds a variance threshold, the example method determines a latent space vector distance quotient. The latent space vector distance quotient indicates a distance of the latent space vector to a closest object class latent space vector in the object class latent space vectors relative to a distance of the latent space vector to a closest latent space vector determined to correspond to an object class different from the object class of the latent space vector. Finally, the example method provides the automobile sensor data to an object classifier based on the latent space vector distance quotient.

[0053] In the example method, determining the latent space vector distance quotient may include calculating the latent space vector distance quotient based on dividing an inner class distance by an outer class distance, wherein the inner class distance may correspond to a distance between the latent space vector and an object class latent space vector that is closest to the latent space vector among the object class latent space vectors, and wherein the outer class distance may correspond to a distance between the latent space vector and a different object class latent space vector that is closest to the latent null vector, the different object class latent space vector corresponding to an object class different from the object class corresponding to the latent space vector.

[0054] The example method may also include generating a reduced latent space vector if at least one of the latent space vector variances exceeds a variance threshold, the reduced latent space vector including a component of the latent space vector that exceeds the variance threshold, wherein the latent space vector distance quotient is determined, the latent space vector distance quotient may indicate a distance of the reduced latent space vector to a closest reduced object class latent space vector among the object class latent space vectors relative to a distance of the reduced latent space vector to a closest reduced latent space vector determined to correspond to an object class different from the object class of the latent space vector.

[0055] In the example method, obtaining the latent space vector may include deriving the latent space vector from the activation values ​​in the latent space of the machine learning algorithm based on an activation probability of the activation values.

[0056] The example method may also include normalizing the latent space vector distance quotient to a normalized latent space vector distance quotient having a value within a distance quotient range; and inverting the normalized latent space vector distance quotient.

[0057] In the example method, the normalizing the latent space vector distance quotient to the normalized latent space vector distance quotient may be based on one of a sigmoid function, a hyperbolic tangent function, a Soboleva modified hyperbolic tangent function, or a Gaussian function.

[0058] In the example method, the machine learning algorithm may be an artificial neural network, and the latent space vector includes all activation values ​​of intermediate layers of the artificial neural network between an input layer and an output layer of the artificial neural network.

[0059] In the example method, the intermediate layer may be a layer of the artificial neural network before the output layer.

[0060] In the example method, the machine learning algorithm may be an autoencoder, and the latent space vector may correspond to a code of the autoencoder.

[0061] In the example method, providing the vehicle sensor data to the subject may be based on a latent space vector distance quotient threshold.

[0062] In the example method, providing the vehicle sensor data to the object classifier may include: storing the latent space vector distance quotient and the corresponding vehicle sensor data, as well as one or more previously determined latent space vector distance quotients and the corresponding vehicle sensor data, and one or more subsequently determined latent space vector distance quotients and the corresponding vehicle sensor data; and providing one or more of the following to the object classifier based on the ranking of the corresponding latent space vector distance quotients: the latent space vector distance quotient, the one or more previously determined latent space vector distance quotients, the one or more subsequently determined latent space vector distance quotients, and the corresponding vehicle sensor data.

[0063] In the example method, the object classifier may be one of a user of the vehicle and a cloud-based object classification service.

[0064] In one example, an automotive control unit includes at least one processing unit and a memory coupled to the at least one processing unit and configured to store machine-readable instructions. The machine-readable instructions cause the at least one processing unit to provide automotive sensor data to a machine learning algorithm. The machine learning algorithm has been trained to classify objects in the environment of the vehicle based on training automotive sensor data. The automotive sensor data is used as an input to the machine learning algorithm. The output of the machine learning algorithm corresponds to an object class determined by the machine learning algorithm based on the automotive sensor data. The machine-readable instructions also cause the at least one processing unit to obtain a latent space vector from the machine learning algorithm. The latent space vector corresponds to an activation value within a latent space of the machine learning algorithm. The latent space includes all activation values ​​of the machine learning algorithm without activation values ​​corresponding to the input and the output. The machine-readable instructions also cause the at least one processing unit to calculate at least one latent space vector variance of the latent space vector based on an object class latent space vector. The object class latent space vector corresponds to previous automotive sensor data determined to correspond to the object class of the latent space vector. If the at least one latent space vector variance exceeds a variance threshold, the machine-readable instructions also cause the at least one processing unit to determine a latent space vector distance quotient. The latent space vector distance quotient indicates a distance of the latent space vector to a closest object class latent space vector in the object class latent space vectors relative to a distance of the latent space vector to a closest latent space vector determined to correspond to an object class different from the object class of the latent space vector. Finally, the machine-readable instructions further cause the at least one processing unit to provide the vehicle sensor data to an object classifier based on the latent space vector distance quotient.

[0065] In the example automobile control unit, the machine-readable instructions may further cause the at least one processing unit to perform any one of the above example methods.

[0066] In one example, a vehicle includes any of the above-described example vehicle control units.

[0067] The foregoing description is provided to illustrate active learning for object classification in visual perception tasks in vehicles. It should be understood that this description is by no means intended to limit the scope of the present disclosure to the precise embodiments discussed throughout the description. Instead, those skilled in the art will appreciate that the examples of the present disclosure may be combined, modified, or condensed without departing from the scope of the present disclosure as defined by the appended claims.

[0068] Reference numerals list

[0069] 100: Methods

[0070] 110-182: Methods and Steps

[0071] 200: Car

[0072] 210: Automotive Sensors

[0073] 220: Light

[0074] 300: Automotive control unit

[0075] 310: CPU

[0076] 320: GPU

[0077] 320c: Connect

[0078] 330: Automotive Processing Systems

[0079] 340: Memory

[0080] 350: Removable storage device

[0081] 360: Storage Device

[0082] 370: Cellular interface

[0083] 380: GNSS interface

[0084] 390: Communication interface

Claims

1. A method (100) configured to enable active learning of object classification in a visual perception task in a vehicle (200), the vehicle being configured to provide at least partial driving automation, the method comprising: providing (110) the vehicle sensor data to a machine learning algorithm that has been trained to classify objects in the environment of the vehicle based on the training vehicle sensor data, wherein the vehicle sensor data serves as an input to the machine learning algorithm, and wherein an output of the machine learning algorithm corresponds to an object class determined by the machine learning algorithm based on the vehicle sensor data; obtaining (120) a latent space vector from the machine learning algorithm, the latent space vector corresponding to an activation value within a latent space of the machine learning algorithm, the latent space including all activation values ​​of the machine learning algorithm without activation values ​​corresponding to the input and the output; calculating (130) at least one latent space vector variance of the latent space vector based on an object class latent space vector corresponding to prior automotive sensor data determined to correspond to the object class of the latent space vector; If the at least one latent space vector variance exceeds a variance threshold, determining (150) a latent space vector distance quotient indicating a distance of the latent space vector to a closest object class latent space vector among the object class latent space vectors relative to a distance of the latent space vector to a closest latent space vector determined to correspond to an object class different from the object class of the latent space vector; and The vehicle sensor data is provided (180) to an object classifier based on the latent space vector distance quotient.

2. The method (100) of claim 1, wherein determining (140) the latent space vector distance quotient comprises: The latent space vector distance quotient is calculated (151) based on dividing the inner class distance by the outer class distance, wherein the intra-class distance corresponds to the distance between the latent space vector and the object class latent space vector that is closest to the latent space vector among the object class latent space vectors, and The external class distance corresponds to a distance between the latent space vector and a different object class latent space vector closest to the latent null vector, the different object class latent space vector corresponding to an object class different from the object class corresponding to the latent space vector.

3. The method (100) according to any one of the preceding claims, further comprising: If at least one of the latent space vector variances exceeds a variance threshold, generating (140) a reduced latent space vector comprising components of the latent space vector that exceed the variance threshold, wherein the latent space vector distance quotient is determined (150), the latent space vector distance quotient indicating the distance of the reduced latent space vector to the closest reduced object class latent space vector among the object class latent space vectors relative to the distance of the reduced latent space vector to the closest reduced latent space vector determined to correspond to an object class different from the object class of the latent space vector.

4. A method (100) according to any of the preceding claims, wherein obtaining (120) the latent space vector includes deriving (121) the latent space vector from the activation values ​​in the latent space of the machine learning algorithm based on the activation probability of the activation values.

5. The method (100) according to any one of the preceding claims, further comprising: normalizing (160) the latent space vector distance quotient to a normalized latent space vector distance quotient having a value within a distance quotient range; as well as The normalized latent space vector distance quotient is inverted (170).

6. The method (100) according to claim 5, wherein normalizing the latent space vector distance quotient to the normalized latent space vector distance quotient is based on one of the following: a sigmoid function, a hyperbolic tangent function, a Soboleva modified hyperbolic tangent function or a Gaussian function.

7. The method (100) according to any one of the preceding claims, wherein: The machine learning algorithm is an artificial neural network, and The latent space vector includes all activation values ​​of intermediate layers of the artificial neural network between an input layer and an output layer of the artificial neural network.

8. The method (100) of claim 6, wherein the intermediate layer is a layer of the artificial neural network before the output layer.

9. The method (100) according to any one of claims 1 to 6, wherein: The machine learning algorithm is an autoencoder, and The latent space vector corresponds to the code of the autoencoder.

10. The method (100) of any one of the preceding claims, wherein providing (180) the vehicle sensor data to the subject is based on a latent space vector distance quotient threshold.

11. The method (100) of any one of claims 1 to 9, wherein providing (180) the vehicle sensor data to the object classifier comprises: storing (181) the latent space vector distance quotient and the corresponding vehicle sensor data, and one or more previously determined latent space vector distance quotients and the corresponding vehicle sensor data, and one or more subsequently determined latent space vector distance quotients and the corresponding vehicle sensor data; and One or more of the following is provided (182) to the object classifier based on the ranking of the corresponding latent space vector distance quotients: the latent space vector distance quotient, the one or more previously determined latent space vector distance quotients, and the one or more subsequently determined latent space vector distance quotients, and the corresponding vehicle sensor data.

12. The method of any preceding claim, wherein the object classifier is one of a user of the vehicle and a cloud-based object classification service.

13. An automobile control unit (300), comprising: at least one processing unit (310); as well as a memory (340, 350, 360) coupled to the at least one processing unit (310) and configured to store machine-readable instructions, wherein the machine-readable instructions cause the at least one processing unit to: providing the vehicle sensor data to a machine learning algorithm that has been trained to classify objects in the environment of the vehicle based on the training vehicle sensor data, wherein the vehicle sensor data serves as an input to the machine learning algorithm, and wherein an output of the machine learning algorithm corresponds to an object class determined by the machine learning algorithm based on the vehicle sensor data; obtaining a latent space vector from the machine learning algorithm, the latent space vector corresponding to an activation value within a latent space of the machine learning algorithm, the latent space including all activation values ​​of the machine learning algorithm without activation values ​​corresponding to the input and the output; calculating at least one latent space vector variance of the latent space vector based on an object class latent space vector corresponding to prior automotive sensor data determined to correspond to the object class of the latent space vector; determining a latent space vector distance quotient if the at least one latent space vector variance exceeds a variance threshold, the latent space vector distance quotient indicating a distance of the latent space vector to a closest object class latent space vector among the object class latent space vectors relative to a distance of the latent space vector to a closest latent space vector determined to correspond to an object class different from the object class of the latent space vector; and The automobile sensor data is provided to an object classifier based on the latent space vector distance quotient.

14. The automotive control unit (300) according to claim 13, wherein the machine-readable instructions further cause the at least one processing unit (310) to execute the method according to any one of claims 2 to 11.

15. A vehicle (200) comprising an automotive control unit (300) according to any one of claims 13 and 14.

Images