Method and system for creating a training dataset and / or validation dataset for artificial neural networks

Abstract

To provide a method for creating a balanced training and / or validation dataset for artificial neural networks, a method (100) for creating a training and / or validation dataset for artificial neural networks is proposed, comprising the steps of: creating a plurality of datasets for a plurality of situations from sensor data of at least one sensor (12) of a data collector (10), computing feature vectors for each of the datasets using a data processing unit (14) of the data collector (10), computing hash values ​​H for each feature vector using the data processing unit (14), transmitting at least one of the hash values ​​H to a computing device (15) spaced apart from the data collector (10), determining frequencies HZ of the transmitted hash values ​​H using the computing device (15).and creation of a training dataset from the datasets, taking into account the frequencies HZ.,

Description

[0001] The present invention relates to a method for creating a training dataset and / or a validation dataset for artificial neural networks. Furthermore, the present invention relates to a system configured for carrying out a method for creating a training dataset and / or a validation dataset for artificial neural networks.

[0002] Artificial neural networks are needed, among other things, for autonomous driving. Training such networks requires a large dataset, which typically includes image data. This image data is captured by cameras installed in the vehicles and transmitted to a backend server for training the neural network. The amount of data transmitted and stored can be very large. Furthermore, the data must be prepared and labeled for training the neural network.

[0003] For an artificial neural network trained for autonomous driving to make correct and meaningful decisions, the training datasets must be diverse and balanced. The training datasets are generated as a subset of all stored data, and it is particularly important that frequently occurring situations that an autonomous vehicle might encounter are weighted appropriately compared to rarer situations.

[0004] Training datasets are generated from only a fraction of the stored data. If the underlying dataset is unbalanced, the training datasets will also be unbalanced. To validate or test artificial neural networks, validation datasets are generated as a further subset of all stored data. The artificial neural network therefore does not "see" the data in the validation datasets during training. Testing the artificial neural networks using validation datasets is intended to prevent over-fitting of the artificial neural networks to the training data.

[0005] US Patent 2022 / 0083820 A1 discloses a method for generating a training dataset, a validation dataset, and / or a test dataset for an AI module from measurement data. The method comprises splitting the measurement data based on time periods, applying a mathematical function to the split measurement data to obtain signatures representing the split measurement data, determining a measure for the frequency of occurrence of each signature, and generating a training dataset, a validation dataset, and / or a test dataset from the measurement data based on the frequency measure.

[0006] The present invention is based on the objective of providing a method for creating a balanced training dataset and / or validation dataset for artificial neural networks.

[0007] To solve the problem underlying the invention, a method for creating a training dataset and / or validation dataset for artificial neural networks is proposed, comprising the steps of: creating a plurality of datasets for a plurality of situations from sensor data of at least one sensor of a data collector, calculating feature vectors for each of the datasets using a data processing unit of the data collector, calculating hash values ​​H for each feature vector using the data processing unit, transmitting at least one of the hash values ​​H to a computing device located away from the data collector, determining frequencies HZ of the transmitted hash values ​​H using the computing device, and creating a training dataset from the datasets taking into account the frequencies HZ.

[0008] The method according to the invention can be used in particular in a method for controlling automatic vehicle functions of motor vehicles, as well as in autonomous driving. The use of automatic driving functions and autonomous driving should thereby become safer.

[0009] Furthermore, the process of providing suitable, balanced training and / or validation data can be optimized. The artificial neural networks trained using the inventive method can be used, in particular, to recognize the vehicle's surroundings more accurately. This knowledge of the environment allows for safer operation of automated driving functions. A training dataset describes a multitude of vehicle or traffic situations in the vehicle's surroundings, as they typically occur. The training dataset can be generated, in particular, from a base dataset, especially image data, for example, from camera images. However, the training dataset can also be artificially generated from synthetic data. One method for creating synthetic data is augmentation, in which values ​​contained in the data are slightly varied.A validation data set is taken from the same base data set and serves to prevent overfitting of the artificial neural network.

[0010] By procedure, sensor data from at least one sensor of a data collector is used to create a multitude of data records for a multitude of situations. In particular, each situation can be assigned to one of the multitude of data records.

[0011] In the context of the invention, these situations may be those that occur during the use of a motor vehicle. For example, these situations may include an empty country road in sunshine or a busy country road during heavy snowfall at night.

[0012] For each data record, a feature vector is calculated by a data processing unit of the data collector. The calculation of the feature vector thus takes place within the data collector, specifically in its data processing unit. The feature vector contains essential information about the respective data record in a highly compressed form. The feature vector allows, in particular, the classification of the underlying situation of the data record, but generally does not allow the corresponding data record to be reconstructed from the feature vector.

[0013] In a further process step, a hash value H is calculated for each feature vector using the data processing unit of the data collector. Within the scope of the invention, a hash value is generally understood to be a mapping that maps a large input set to a smaller target set. The hash value H need not be a cryptographic hash function. After the hash values ​​H have been calculated, at least one of the hash values ​​H is transmitted to a computing device located at a distance from the data collector, particularly spatially. For this purpose, the data collector can be equipped with a data transmission device. The computing device is arranged at a distance from the data collector and can, in particular, be located at a considerable distance from the data collector.

[0014] Preferably, it may be provided that at least one of the feature vectors and / or at least one of the data sets is also transferred to the computing device.

[0015] The computing device determines the frequency HZ of the transmitted hash values ​​H. Since the hash values ​​H depend on the original situations, the computing device thus determines the frequency HZ of the situations underlying the hash values ​​H. Subsequently, the computing device creates a training dataset and / or a validation dataset from the datasets, taking into account the frequencies HZ of the hash values ​​H.

[0016] It is also preferable to create a test data set.

[0017] Since the frequencies HZ of the hash values ​​H are taken into account when creating the training dataset and / or the validation dataset, the corresponding training dataset and / or validation dataset is balanced with regard to the situations represented therein, i.e., in the training dataset and / or validation dataset, more frequently occurring situations occur more often than less frequently occurring situations, but in a well-dosed manner.

[0018] Preferably, a plurality of data collectors is provided, each of which determines a plurality of data sets for a plurality of situations from sensor data from sensors, wherein the plurality of data collectors each calculates feature vectors for each of the data sets, wherein the plurality of data collectors furthermore calculates hash values ​​H for each of the feature vectors and transmits at least one of the hash values ​​H to the spaced computing device.

[0019] In a preferred embodiment, the data collector is a motor vehicle, and / or the at least one sensor is a camera and / or a speed sensor and / or an acceleration sensor and / or a steering angle sensor and / or a radar sensor and / or a lidar sensor and / or a GPS sensor and / or an electronic memory, and / or the computing device is a backend server.

[0020] If the data collector is a motor vehicle, preferably a plurality of motor vehicles, the method is particularly suitable for creating training datasets and / or validation datasets for autonomous driving.

[0021] Accordingly, the sensors can be a camera, a speed sensor, an acceleration sensor, a steering angle sensor, a radar sensor, a lidar sensor, a GPS sensor, and / or the vehicle's electronic storage device. Basically, any sensor suitable for acquiring data for autonomous driving can be used.

[0022] The computing device is preferably a cloud server or backend server that processes the hash values ​​H and / or feature vectors and / or datasets transmitted by the data collector(s). The computing device may further be configured to create the training and / or validation dataset from the datasets, taking into account the frequencies HZ.

[0023] The computing device preferably exchanges data wirelessly, for example via a mobile network, with the data processing units of the data collector(s).

[0024] A further advantage is that the situations can be designed to relate to current events and / or time intervals.

[0025] For example, the data set for the situation could be a single image or a single measurement from a sensor of the data collector. This is particularly the case if the situation is a momentary event. If the situation covers a time interval, the data set could include, for example, video sequences as well as time series of sensor data recorded within that time interval.

[0026] The length of the time interval can range from 1 millisecond to 100 seconds or more.

[0027] In particular, it is intended that the feature vectors include a variety of data characterizing the situations underlying the datasets.

[0028] For example, the data may include a maximum speed of the data collector, and / or a segmentation of an image or image sequence, and / or a distance measurement, and / or a position and / or an acceleration value and / or a steering angle of the data collector, and / or a memory content of the memory.

[0029] It is therefore possible for a feature to be, for example, the maximum speed of the vehicle during a given situation, particularly within a specific time interval, or for a feature to represent a percentage of image pixels of the "road" class. This percentage of image pixels can be determined by the data processing unit using a segmentation method. The feature vector data can also include the positional data of two opposite corners of a rectangular bounding box, derived, for example, from GPS positional data. Furthermore, the characterizing data may include minima, medians, or maxima of a vehicle speed and / or braking torques and / or their derivatives.

[0030] It is preferred that the hash values ​​H are integers.

[0031] The integer representation allows the hash values ​​H to be used in particular to characterize or classify the situations underlying the feature vectors.

[0032] It may preferably be provided that at least one data point, in particular all data points, of the feature vector is compared with threshold values, and an integer is assigned to the data point, in particular all data points, depending on the comparison.

[0033] In a non-restrictive example, a feature vector datum for a given situation could be the percentage of image pixels of the class "street," determined from image data using a segmentation method. For instance, it could be specified that if the percentage is less than the threshold of 30%, the datum would be assigned the integer 1. If the percentage lies between the thresholds of 30% and 60%, the datum could be assigned the integer 2. If the percentage exceeds the threshold of 60%, the datum could be assigned the integer 3. The aforementioned numerical examples are purely illustrative, and the conversion to an integer based on different thresholds can be individually configured for each datum.

[0034] A further advantage is that the transfer of the at least one hash value H to the computing device is carried out by means of a batch transfer.

[0035] Batch transmission allows for the at least one hash value H to be transmitted not individually and not immediately. For example, all hash values ​​H generated within a 10-second interval can be transmitted at once.

[0036] It is preferably provided that, in particular preferably by the data processing unit of the data collector, the calculated hash values ​​H are compared, and that a count of identical hash values ​​H is determined, and that if the count is greater than 1, a hash value H is transmitted only once, and that together with the hash value H the count of identical hash values ​​H is transmitted.

[0037] If a hash value H occurs multiple times, the hash value H and, if applicable, the associated feature vector do not need to be transmitted multiple times. Instead, only the hash value H and the count of the corresponding hash value are transmitted. An example of such hash values ​​H are those generated from images taken of a vehicle stopped at a red light. In such a situation, many similar images are captured, resulting in identical hash values ​​H, so these do not need to be transmitted to the processing unit multiple times.

[0038] Preferably, it may be provided that a corresponding data record is transmitted to the computing device for at least one of the transmitted hash values ​​H.

[0039] In principle, for every transmitted hash value H, a corresponding data record containing this hash value H can be transmitted to the computing device.

[0040] If a batch transfer is performed as explained above, it may be provided, in particular, that for a hash value H which has a count value greater than 1, only a representative data set is transferred to the computing device.

[0041] Accordingly, it may preferably be provided that, if the count value is greater than 1, data records and / or feature vectors, preferably all but one of the data records and / or feature vectors, are discarded with respect to the corresponding hash value H, in particular by the data processing unit of the data collector.

[0042] If a hash value H occurs multiple times, the data records and feature vectors from several, in particular all but one, of the situations exhibiting the identical hash value H can be discarded. This discarding of the data records and / or feature vectors can take place directly within the data processing unit of the data collector, so that the corresponding data records do not first need to be transmitted to the computing unit.

[0043] In the example above, involving a vehicle stopped at a traffic light, the calculated hash value H can be identical for a large number of generated images. For example, the count of the identical hash value is 10. In this case, in a particularly preferred embodiment, the corresponding hash value H is transmitted to the computing device only once, together with the count of 10. Furthermore, a data record determined for one of the ten images can be transmitted to the computing device, although transmission can also be omitted if a corresponding data record with the multiple occurrences of the hash value has already been transmitted to the computing device.

[0044] In particular, it can therefore be provided that during each batch transfer, only one such data record whose hash value H is new to the set of hash values ​​is transmitted to the computing device. If a hash value H already exists in the batch intended for the batch transfer, the corresponding data record with its feature vector can be deleted directly in the data collector, especially in the vehicle.

[0045] Preferably, it is provided that, particularly by the computing device, after receiving a hash value and optionally the count of identical hash values, the frequency value HZ of the corresponding hash value is incremented. If only the hash value is transmitted or the count is 1, the frequency value HZ is incremented by 1. If the count is greater than 1, the frequency value HZ is incremented by the count.

[0046] Preferably, data records transmitted to the computing device are stored in a database.

[0047] Therefore, it is preferable to store at least one corresponding data record in the database for each hash value.

[0048] A further advantage is that the computing device can preferably determine a maximum number of DZ for each hash value H. max is determined on stored data records, with the maximum number preferably DZ max = max (f * ceil(log 10 HZ), DZ start ), where DZ start an initial value, where f is a scaling factor, where HZ is the frequency of a hash value H.

[0049] Thus, for each hash value H, a maximum number of DZ is determined. max The number of data records to be stored is set. The initial value is DZ. startIt can be any starting value. For example, the initial value could be DZ. start For each of the hash values ​​H in a new measurement campaign, the value must be the same for all hash values ​​H. The logarithmic adaptation described above for the maximum number of DZ is preferred. max used. In principle, any other adaptation formulas can also be used. The scaling factor f is preferably between 0 and 100, preferably between 1 and 50, and more preferably between 1 and 10.

[0050] By setting the maximum number of double rooms max On the one hand, the required storage space is reduced for the data records to be stored for the respective hash values ​​H. Furthermore, the number of data records to be transmitted, which are generated from the sensor data of the data collector, is reduced.

[0051] In particular, the maximum number of DZ is preferred for creating the training dataset and / or the validation dataset for each hash value. max used on data sets and / or according to the maximum number of DZ max the data sets are weighted.

[0052] By setting the maximum number of double rooms max For each hash value H, a balanced training and / or validation dataset is thus provided, in which frequent situations occur more often than rare situations.

[0053] It can further be provided that, preferably by the computing device, a data record counter DZ is determined for each hash value H, wherein the data record counter DZ contains a number of the stored data records belonging to the hash value H, and wherein a corresponding data record is only transmitted to the computing device for a hash value H if DZ < DZ max .

[0054] The data record counter DZ is thus determined on the computing device side and corresponds to the number of data records stored for a specific hash value H at the respective time. When a new data record is transmitted, the data record counter is incremented by 1. It can be stipulated that a data record for a specific hash value H is only transmitted from the data collector to the computing device if DZ < DZ max is.

[0055] Thus, in particular after the data collector has transmitted at least one hash value H, in particular the hash values ​​H, and further in particular the hash values ​​H together with the count values ​​of the hash values, to the computing device, the computing device can, after evaluating the hash value H, transmit the instruction to the data transmission unit of the data collector that a data record should be transmitted or that transmission of the data record should not take place.

[0056] Another solution to the problem underlying the invention consists in a system for creating a training data set and / or validation data set for artificial neural networks, wherein the system is configured to carry out a method described above, comprising at least one data collector with at least one sensor and a data processing unit, and further comprising a computing device spaced apart from the data collector.

[0057] A further solution to the problem underlying the invention consists of a method for controlling automatic vehicle functions of motor vehicles, wherein an artificial neural network is used, wherein the artificial neural network was trained and / or validated with a training data set and / or validation data set generated in a previously described method.

[0058] All the embodiments, features and functions described above can also be transferred to the system in a correspondingly analogous manner.

[0059] The data collectors of the system are in particular at least one motor vehicle, preferably a large number of motor vehicles, especially a fleet of vehicles.

[0060] The invention is explained in more detail below with reference to the accompanying figures. These show: Fig. 1. A flowchart for a procedure for creating a training dataset and / or a validation dataset for artificial neural networks, and Fig. 2 a system for creating a training dataset and / or validation dataset for artificial neural networks.

[0061] The flowchart after Fig. One of the methods presented for creating a training dataset and / or validation dataset for artificial neural networks can be used with a Fig. The process is carried out using the system 200 shown in Figure 2. The system 200 comprises a multitude of data collectors 10, which are configured as motor vehicles 11. Each of the motor vehicles 11 includes sensors 12, such as cameras 13, distance sensors, and speed sensors. Each of the motor vehicles 11 also includes a data processing unit 14. The system 200 further comprises a computing device 15 spatially separated from the data collectors 10 or motor vehicles 11. The computing device 15 can be configured as a backend server 16 or a cloud server. For data transmission 19 with the computing device 15, the motor vehicles 11 have data transmission devices 17. The data transmission 19 can take place via a mobile network 18.

[0062] In a first process step 20 of the process 100 according to Fig. 1, data sets are generated for a multitude of situations from sensor data acquired by the sensors 12. These situations are those that can occur during the use of the motor vehicles 11. For example, one such situation could be an empty country road in sunshine.

[0063] In a second process step 21, the data processing units 14 of the motor vehicles 11 determine feature vectors for each of the created datasets. These feature vectors comprise the data characterizing the situations underlying the datasets. For example, the data could be the maximum speed of the respective motor vehicle 11 during a time interval or a percentage of image pixels of the class "road". The percentage of image pixels can be determined by the data processing units 14 using a segmentation method.

[0064] In a further step (22), a hash value H is calculated for each of the feature vectors. These hash values ​​H are integers. One possible method for calculating the hash values ​​H is to compare the data of the respective feature vector with threshold values ​​and assign an integer to each data point based on the comparison. For example, speed ranges can be defined, each with its own corresponding integer value. If the vehicle's speed is between 0 and 50 km / h, the integer 0 can be assigned to this data point in the feature vector. If the speed is between 50 km / h and 100 km / h, the integer 1 can be assigned to this data point in the feature vector. The data sets derived from these calculated feature vectors and the hash values ​​H are collected for a predetermined period, for example, 10 seconds.

[0065] In the following step 23, the data processing units 14 of the motor vehicles 11 compare the hash values ​​H calculated during the period. If a hash value H occurs multiple times, a count value is determined for the hash value H. For example, if a hash value occurs ten times, the corresponding count value is 10.

[0066] The data processing units 14 of the motor vehicles 11 discard all but one of the data records and / or feature vectors, to the hash values ​​H, which occur multiple times.

[0067] Subsequently, in a further step 24, data is transferred 19 from the data processing units 14 of the motor vehicles 11 to the computing device 15. To minimize the amount of data to be transmitted, each determined hash value H is transmitted only once. Simultaneously, the respective count value is transmitted to the computing device 15. A representative data set for the respective hash value H can be transmitted to the computing device 15 at the same time as the hash value H and the count value, provided that a maximum number of DZ max data records for the corresponding hash value H have not yet been received by the computing device 15 and stored in a database.

[0068] The maximum number of double rooms maxFor each individual hash value H, the computing device 15 determines in step 25, which can also be performed in parallel with the other process steps. The computing device determines a data record counter DZ for each hash value H, which corresponds to the number of stored data records associated with the hash value H. Furthermore, the computing device determines a frequency HZ for each hash value H, whereby the value of the frequency HZ is incremented by the count value for each hash value H received by the computing device 15. The maximum number of DZ max The calculating device 15 calculates according to DZ max = max (f * ceil(log 10 HZ), DZ start ), where DZ start Let f be an arbitrary initial value and f a scaling factor. For a hash value calculated by the data processing units, DZ < DZ max, then a data record corresponding to the hash value H is transmitted to the computing device 15, and the data record counter DZ is incremented by 1. If there are already more than the maximum number of DZ max If data records for a specific hash value H are stored in the database, the further transfer of the corresponding data records from the motor vehicles 11 to the computing device 15 is omitted.

[0069] The maximum number of double rooms max The respective hash values ​​H can then be used to create a balanced training dataset and / or validation dataset. This allows either the DZ to be used. max The stored data sets of the hash values ​​H can be used directly as a training data set and / or validation data set, or the data sets can be used to create the training data set and / or validation data set according to the maximum number of DZ. max weighted. Reference symbol list 100 procedures 200 System 10 data collectors 11 Motor vehicle 12 Sensor 13 Camera 14 Data processing unit 15 Calculating device 16 backend servers 17 Data transmission device 18 Mobile network 19 Data transmission 20-25 process steps QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2022 / 0083820 A1

[0005]

Claims

[1] Method (100) for creating a training dataset and / or validation dataset for artificial neural networks, comprising the steps of: creating a plurality of datasets for a plurality of situations from sensor data of at least one sensor (12) of a data collector (10), calculating feature vectors for each of the datasets using a data processing unit (14) of the data collector (10), calculating hash values ​​H for each feature vector using the data processing unit (14), transmitting at least one of the hash values ​​H to a computing device (15) spaced apart from the data collector (10), determining frequencies HZ of the transmitted hash values ​​H using the computing device (15), and creating a training dataset from the datasets taking into account the frequencies HZ. [2] Method (100) according to claim 1, characterized by, that the data collector (10) is a motor vehicle (11), and / or that the at least one sensor (12) is a camera (13) and / or a speed sensor and / or an acceleration sensor and / or a steering angle sensor and / or a radar sensor and / or a lidar sensor and / or a GPS sensor and / or an electronic storage device, and / or that the computing device (15) is a backend server (16). [3] Method (100) according to claim 1 or 2, characterized by that the situations concern current events and / or time intervals. [4] Method (100) according to any of the aforementioned claims, characterized by that the feature vectors comprise a variety of data characterizing the situations underlying the datasets. [5] Method (100) according to claim 4, characterized by, that the data include a maximum speed of the data collector (10), and / or a segmentation of an image or image sequence, and / or a distance measurement, and / or a position and / or an acceleration value and / or a steering angle of the data collector (10), and / or a memory content of the memory of the data collector (10). [6] Method (100) according to any of the aforementioned claims, characterized by that the hash values ​​H are integers. [7] Method (100) according to claim 6, characterized by , that at least one data point, in particular all data points, of the feature vector is compared with threshold values, and an integer is assigned to the data point, in particular all data points, depending on the comparison. [8] Method (100) according to any of the aforementioned claims, characterized by , that the transfer of the at least one hash value H to the computing device (15) is carried out by means of a batch transfer. [9] Method (100) according to any of the aforementioned claims, characterized by , that, preferably by the data processing unit (14) of the data collector (10), the calculated hash values ​​H are compared, and that a count of identical hash values ​​H is determined, and that if the count is greater than 1, a hash value H is transmitted only once, and that together with the hash value H the count of identical hash values ​​H is transmitted. [10] Method (100) according to any of the aforementioned claims, characterized by , that for at least one of the transmitted hash values ​​H a corresponding data record is transmitted to the computing device (15). [11] Method (100) according to claim 9 or 10, characterized by, that if the count value is greater than 1, data records and / or feature vectors, preferably all but one of the data records and / or feature vectors, are discarded to the corresponding hash value H, in particular by the data processing unit (14) of the data collector (10). [12] Method (100) according to any of the aforementioned claims, characterized by , that data records transmitted to the computing device (15) are stored in a database. [13] Method (100) according to any of the aforementioned claims, characterized by , that, preferably by the computing device (15), for each hash value H a maximum number of DZ max is determined on stored data records, with the maximum number preferably DZ max = max (f * ceil(log 10 HZ), DZ start where DZ start an initial value, where f is a scaling factor, where HZ is the frequency of a hash value H. [14] Method (100) according to claim 13, characterized by , that, preferably by the computing device (15), a data record counter DZ is determined for each hash value H, wherein the data record counter DZ contains a number of the stored data records belonging to the hash value H, wherein a corresponding data record is only transmitted to the computing device (15) for a hash value H if DZ < DZ max . [15] System (200) for creating a training data set and / or validation data set for artificial neural networks, wherein the system is configured to carry out a method (100) according to one of the preceding claims, comprising at least one data collector (10) with at least one sensor (12) and a data processing unit (14), further comprising a computing device (15) spaced apart from the data collector (10). [16] Method for controlling automatic vehicle functions of motor vehicles, wherein an artificial neural network is used, wherein the artificial neural network was trained and / or validated with a training data set and / or validation data set generated in a method according to one of claims 1 to 14.

Images