Method for training an artificial neural network

Abstract

The invention relates to a method for training an artificial neural network using training data, the training data comprising first image data of a first image and second image data of a second image of an infrastructure, wherein the first image comprises a first feature and wherein the second image comprises a second feature corresponding to the first image, wherein the training data comprises a target relative translation and a target relative rotation between the first feature and the second feature, wherein the training comprises: extracting the first feature from the first image and the second feature from the second image by means of an artificial neural network, wherein the extracted first feature is represented by first feature data having a first data amount, wherein the extracted second feature is represented by second feature data having a second data amount, determining the relative translation and the relative rotation between the extracted first feature and the extracted second feature by means of an artificial neural network, wherein the artificial neural network is trained until a loss function related to a pose of a feature and related to the first data amount and / or the second data amount has a minimum value or is smaller than or smaller than or equal to a predefined loss function threshold. The invention relates to an artificial neural network, a method for extracting a feature from an image of a surrounding of a motor vehicle, a device, a computer program and a machine readable storage medium.

Description

Background Technology

[0001] If the same scene is detected using corresponding environmental sensors from multiple motor vehicles, the images corresponding to that detection will typically overlap. The corresponding identical features in these images will often be offset.

[0002] In these motor vehicles, artificial neural networks can be used, for example, to extract features from the corresponding images.

[0003] The extracted features can be sent by the vehicles to a remote server, which may be part of a cloud infrastructure. This server can then align these features to, for example, create a digital map. Algorithms in the cloud can then use these features to align the individual journeys of the vehicles with each other based on the extracted features.

[0004] In this scenario, the features to be transmitted need to be suitable for efficient alignment. Furthermore, the amount of data in the transmitted features should not be too large, because bandwidth is typically limited. Summary of the Invention

[0005] The objective of this invention is to provide a design for efficiently training artificial neural networks.

[0006] This task is solved by means of the corresponding subject matter of the independent claim. The advantageous embodiments of the invention are the subject matter of the corresponding dependent claims.

[0007] According to the first aspect, a method for training an artificial neural network is provided, the method comprising the following steps:

[0008] Receive training data signals, which include training data for training an artificial neural network, wherein the training data includes first image data of a first image of the infrastructure and second image data of a second image of the infrastructure, wherein the first image includes a first feature and wherein the second image includes a second feature corresponding to the first image, wherein the training data includes target relative translation and target relative rotation between the first feature and the second feature;

[0009] The artificial neural network is trained based on this training data, and the training includes the following steps:

[0010] Using an artificial neural network, a first feature is extracted from a first image and a second feature is extracted from a second image, wherein the extracted first feature is represented by first feature data having a first data volume, and the extracted second feature is represented by second feature data having a second data volume.

[0011] By using an artificial neural network, the relative translation and relative rotation between the extracted first feature and the extracted second feature are determined.

[0012] The artificial neural network is trained until the loss function, which is related to the pose of the features and to the first and / or second data volume, has a minimum value or is less than or equal to a pre-given loss function threshold.

[0013] According to the second aspect, an artificial neural network is provided, which is trained by means of the method according to the first aspect.

[0014] According to a third aspect, a method for extracting features from an image of the environment surrounding a motor vehicle is provided, the method comprising the following steps:

[0015] Receive image data signals, which represent image data of the vehicle's surrounding environment;

[0016] Features are extracted from the image using an artificial neural network based on the second aspect;

[0017] Output feature signals, which represent the extracted features.

[0018] According to a fourth aspect, an apparatus is provided, which is configured to implement all the steps of the method according to the first aspect and / or the third aspect.

[0019] According to a fifth aspect, a computer program is provided, the computer program including instructions that, when executed by a computer, such as by a device according to a fourth aspect, cause the computer to perform the methods according to a first aspect and / or a third aspect.

[0020] According to a sixth aspect, a machine-readable storage medium is provided, on which a computer program according to a fifth aspect and / or an artificial neural network according to a second aspect are stored.

[0021] This invention is based on and includes the following understanding: the above-mentioned task can be solved in the following way: the loss function is not only related to the pose of the feature, but also to the amount of feature data describing the extracted feature.

[0022] This leads to the following technical advantages, for example: using trained artificial neural networks to extract features from images is optimized not only for the applicability of the extracted features to efficient alignment, but also for the smallest possible amount of data.

[0023] Therefore, bandwidth can be used efficiently to send the extracted features, while the remote server can efficiently align the extracted features.

[0024] Therefore, it brings the following technical advantages: it provides a design for efficiently training artificial neural networks.

[0025] One implementation specifies that the artificial neural network includes a CNN (Convolutional Neural Network) and a FCN (Fully Connected Network), wherein the output of the CNN is connected to the input of the FCN, and first image data and second image data are provided to the input of the CNN. FCN can be translated into German as "Voll-verbundenes neuronales Netzwerk". CNN can be translated into German as "Faltendes neuronales Netzwerk".

[0026] This results in the following technical advantages, for example: features can be extracted efficiently.

[0027] In one implementation, the extracted features are specified to be quantized such that both the first feature data and the second feature data are integers.

[0028] This results in, for example, the following technical advantages: data can be efficiently converted into a format that, for example, is used in one embodiment of the method according to the third aspect, to transmit the extracted features from a motor vehicle to a remote server via a communication network.

[0029] This leads to, for example, the following technical advantages: the amount of data for these feature data can be determined efficiently.

[0030] According to one implementation, this quantization includes soft quantization. In English, "softquantization" is used for "soft quantization". Examples of soft quantization can be found, for instance, in "Conditional Probability Models for Deep Image Compression" by F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and LV Gool, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[0031] In one implementation, the loss function is defined by the following mathematical equation: ,in This is the loss function, where It is a loss function related to the pose of the features, where It is a loss function that is related to the first data volume of the first feature data and / or the second data volume of the second feature data.

[0032] This results in, for example, the following technical advantages: the loss function is efficiently constrained.

[0033] In one implementation method, it is specified that: ,in The first parameter, where It is the expected value, where It is the first feature data and / or the second feature data, wherein It is the first data volume and / or the second data volume.

[0034] This results in, for example, the following technical advantages: the loss function is efficiently constrained with respect to the first and / or second data amounts.

[0035] In one implementation method, it is specified that: ,in It is the expected value, where It is a relative translation of the target, where It is a relative translation, in which These are hyperparameters, among which It is the relative rotation of the target, where It is a relative rotation.

[0036] This results in, for example, the following technical advantages: the loss function is efficiently constrained with respect to the pose.

[0037] In one implementation method, it is specified that: ,in It is the expected value, where It is a relative translation of the target, where It is a relative translation, in which It is the relative rotation of the target, where It is a relative rotation, where It is the second parameter, where It is the third parameter.

[0038] This results in the following technical advantages, for example: the loss function is efficiently constrained with respect to the pose. In particular, this loss function is especially robust.

[0039] In one implementation, training an artificial neural network includes training a second parameter and / or a third parameter.

[0040] This leads to the following technical advantages, for example: the second and / or third parameters can be determined efficiently.

[0041] According to one implementation, hyperparameters or parameters weight the components of the loss function L relative to each other. These hyperparameters or parameters are handled particularly during training. That is, these hyperparameters or parameters can be trained in particular.

[0042] In one implementation, the image is an element selected from the following image group: video image, ultrasound image, lidar image, radar image, infrared image, and magnetic field image.

[0043] According to one implementation, the infrastructure includes one or more infrastructure elements.

[0044] According to one implementation, infrastructure elements are selected from the following group of infrastructure elements: roads, sidewalks, buildings, traffic signal systems, streetlights, pillars, bridges, parking lots, and road signs.

[0045] According to one implementation, the infrastructure includes transportation infrastructure for vehicles, particularly motor vehicles.

[0046] According to one implementation, the feature is an infrastructure element.

[0047] According to one implementation, the feature is a natural element, such as a tree, shrub, or hedge.

[0048] The second feature corresponding to the first feature means that the same feature is involved.

[0049] According to one implementation method, the method according to the first aspect and / or the third aspect is a computer-implemented method.

[0050] According to one implementation method, the method according to the first aspect and / or the second aspect is implemented or performed by means of the device according to the fourth aspect.

[0051] The features of the equipment are derived similarly from the corresponding features of the method, and vice versa. That is, this in particular means that the technical functionality of the equipment according to the second aspect is derived similarly from the corresponding technical functionality of the method according to the first aspect and / or the third aspect, and vice versa.

[0052] The abbreviation "or (bzw.)" stands for "or (beziehungsweise)," which in particular means "respektive."

[0053] The word "respective" is especially used to mean "and / or".

[0054] The attitude is defined or specified, in particular, by relative rotation and relative translation. That is, this specifically means that the attitude depends on relative rotation and relative translation.

[0055] The pose determined by an artificial neural network is defined or specified, in particular, by a relative rotation determined by an artificial neural network and by a relative translation determined by an artificial neural network. That is, this specifically means that the pose depends on the relative rotation determined by an artificial neural network and on the relative translation determined by an artificial neural network.

[0056] The target's attitude is defined or specified, in particular, by the target's relative rotation and relative translation. That is, this specifically means that the target's attitude depends on the target's relative rotation and relative translation.

[0057] If the general term "pose" is used, it includes the following phrases: "target pose and / or pose determined by means of an artificial neural network".

[0058] If the general term "characteristic data" is used, it includes the following wording: "first characteristic data and / or second characteristic data".

[0059] If the general term "features" is used, it includes the following wording: "first feature and / or second feature".

[0060] If the general term "data volume" is used, it includes the following phrases: "first data volume and / or second data volume".

[0061] The amount of data can also be called entropy.

[0062] According to one implementation, the motor vehicle is configured to be driven at least partially automatically.

[0063] According to one embodiment, the vehicle includes one or more environmental sensors.

[0064] According to one implementation, the environmental sensor is one of the following: radar sensor, video sensor, ultrasonic sensor, lidar sensor, magnetic field sensor, infrared sensor.

[0065] The image of the vehicle’s surroundings is based on the detection of the vehicle’s surroundings by means of one or more environmental sensors. Attached Figure Description

[0066] Embodiments of the present invention are shown in the accompanying drawings and will be described in more detail in the following description.

[0067] in:

[0068] Figure 1 A flowchart of a method for training an artificial neural network is shown;

[0069] Figure 2 An artificial neural network is shown;

[0070] Figure 3 A flowchart is shown for a method of extracting features from an image of the environment surrounding a motor vehicle;

[0071] Figure 4 The device is shown;

[0072] Figure 5 A machine-readable storage medium is shown; and

[0073] Figure 6 A block diagram is shown.

[0074] In the following text, the same reference numerals may be used for the same features. Detailed Implementation

[0075] Figure 1 A flowchart of a method for training an artificial neural network is shown, which includes the following steps:

[0076] Receive 101 training data signals, which include training data for training an artificial neural network, wherein the training data includes first image data of a first image of the infrastructure and second image data of a second image of the infrastructure, wherein the first image includes a first feature and wherein the second image includes a second feature corresponding to the first image, wherein the training data includes target relative translation and target relative rotation between the first feature and the second feature;

[0077] Based on this training data, a 103 artificial neural network is trained, and the training includes the following steps:

[0078] Using an artificial neural network, 105 first features are extracted from a first image and second features are extracted from a second image. The extracted first features are represented by first feature data having a first data volume, and the extracted second features are represented by second feature data having a second data volume.

[0079] Using an artificial neural network, the relative translation and relative rotation of 107 between the extracted first feature and the extracted second feature are determined.

[0080] The artificial neural network is trained until the loss function, which is related to the pose of the features and to the first and / or second data volume, has a minimum value or is less than or equal to a pre-given loss function threshold.

[0081] Figure 2 An artificial neural network 201 is shown, which is trained by means of the method according to the first aspect.

[0082] Figure 3 A flowchart is shown for a method for extracting features from an image of the environment surrounding a motor vehicle, the method comprising the following steps:

[0083] Receive 301 image data signals, which represent image data of the surrounding environment of the motor vehicle;

[0084] 303 features are extracted from the image using the artificial neural network according to claim 8;

[0085] Output 305 feature signals, which represent the extracted features.

[0086] In one implementation, the extracted features are sent to a remote server via a communication network.

[0087] In one embodiment, the communication network includes a wireless communication network and / or a wired communication network.

[0088] In one implementation, the wireless communication network includes a WLAN communication network and / or a cellular network.

[0089] Figure 4 Device 401 is shown.

[0090] The device 401 is configured to implement all steps of the method according to the first aspect and / or the third aspect.

[0091] The device 401 includes an input terminal 403 configured to receive training data signals and / or image data signals.

[0092] The device 401 includes a processor 405 configured to train an artificial neural network based on training data and / or to extract features from an image using an artificial neural network.

[0093] The processor 405 is specifically designed to implement the steps involved in training an artificial neural network.

[0094] The device 401 includes an output terminal 407 configured to output feature signals and / or training result signals representing trained artificial neural networks.

[0095] Figure 5 Machine-readable storage medium 501 is shown.

[0096] A computer program 503 is stored on a machine-readable storage medium 501. The computer program 503 includes instructions that, when executed by a computer, cause the computer to implement the methods according to the first aspect and / or the third aspect.

[0097] An artificial neural network 505 according to the second aspect is stored on a machine-readable storage medium 501.

[0098] In one embodiment not shown, it is specified that only a computer program 503 or only an artificial neural network 505 is stored on the machine-readable storage medium.

[0099] Figure 6 Block diagram 601 is shown.

[0100] The block diagram 601 includes a first functional block 603 and a second functional block 605.

[0101] The two function blocks 603 and 605 are identical.

[0102] The first functional block 603 is implemented, for example, in the first device according to the third aspect.

[0103] The second functional block 605 is implemented, for example, in the second device according to the third aspect.

[0104] The first device, for example, is integrated into the first motor vehicle.

[0105] The second device, for example, is integrated into a second motor vehicle.

[0106] The two functional blocks 603 and 605 each include an artificial neural network 607.

[0107] The artificial neural network 607 was trained according to the method in the first aspect.

[0108] Artificial neural networks 607 include convolutional neural networks 609, which are referred to as CNNs below.

[0109] Artificial neural networks 607 include fully connected networks 611, hereinafter referred to as FCN.

[0110] The two function blocks 603 and 605 each include a quantizer 613.

[0111] CNN 609 includes a first input terminal 615 and a first output terminal 617.

[0112] FCN 611 includes a second input terminal 619 and a second output terminal 621.

[0113] The quantizer 613 includes a third input terminal 623 and a third output terminal 625.

[0114] The first output terminal 617 is connected to the second input terminal 619.

[0115] The second output terminal 621 is connected to the third input terminal 623.

[0116] First image data 627 is provided to the first input terminal 615 according to the first function block 603. The first image data 627 represents a first image of the first surrounding environment of the first motor vehicle.

[0117] Second image data 629 is provided to the first input terminal 615 according to the second function block 605. The second image data 629 represents a second image of the second surrounding environment of the second motor vehicle.

[0118] In one implementation, the first surrounding environment and the second surrounding environment can overlap.

[0119] The corresponding artificial neural network 607 extracts features from the two images respectively, and the features are correspondingly provided to the third input terminal 623 of the corresponding quantizer 613 according to the first functional block 603 and the second functional block 605.

[0120] The corresponding quantizer 613 quantizes the extracted features so that the first feature data and the second feature data representing the corresponding extracted features are both integers.

[0121] The quantized features are output using the output terminal 625 of the corresponding quantizer 613.

[0122] The first feature 631 extracted after quantization is output from the third output terminal 625 of the quantizer 613 of the first functional block 603.

[0123] The extracted second feature 633 is output from the third output terminal 625 of the quantizer 613 of the second function block 605.

[0124] A vertical dashed line 635 is drawn in this block diagram 601.

[0125] The vertical dashed line symbolically represents that two quantified extracted features 631 and 633 are sent from the corresponding motor vehicle to the remote server 637 via a communication network.

[0126] Server 637 includes another artificial neural network 639, which, in one embodiment, may be another FCN.

[0127] With the help of another artificial neural network 639, the two quantized extracted features 631 and 633 are aligned in order to determine the relative rotation and relative translation associated with the two features.

[0128] The determined relative translation and the determined relative rotation define the orientation of the feature 641.

[0129] The design described herein has the following main advantages: the extracted features can be represented compactly, which leads to lower requirements for data transmission.

[0130] Therefore, the following advantages can be seen in particular: it enables high data rates.

[0131] Furthermore, the design described herein has the particular advantage that the extracted features can be stored efficiently because they require less storage space than, for example, the original data, such as an original image representing the surrounding environment.

[0132] This results in, for example, the following technical advantages: servers that process and store the extracted features require less storage space and / or less computing power.

Claims

1. A method for training artificial neural networks (201, 505, 607), the method comprising the following steps: Receive (101) training data signal, the training data signal including training data for training the artificial neural network (201, 505, 607), wherein the training data includes first image data of a first image of the infrastructure and second image data of a second image of the infrastructure, wherein the first image includes a first feature and wherein the second image includes a second feature corresponding to the first image, wherein the training data includes target relative translation and target relative rotation between the first feature and the second feature; The artificial neural network (201, 505, 607) is trained (103) based on the training data, wherein the training includes the following steps: Using the artificial neural network (201, 505, 607), the first feature is extracted (105) from the first image and the second feature is extracted from the second image, wherein the extracted first feature is represented by first feature data having a first data volume, and wherein the extracted second feature is represented by second feature data having a second data volume. Using the artificial neural networks (201, 505, 607), the relative translation and relative rotation between the extracted first feature and the extracted second feature are determined (107). The artificial neural networks (201, 505, 607) are trained until the loss function, which is related to the pose of the features and to the first and / or second data amounts, has a minimum value or is less than or equal to a pre-given loss function threshold.

2. The method according to claim 1, wherein the artificial neural network (201, 505, 607) includes a convolutional neural network CNN (609) and a fully connected network FCN (611), wherein the output of the CNN (609) is connected to the input of the FCN (611), wherein the first image data and the second image data are provided to the input of the CNN (609).

3. The method according to claim 1 or 2, wherein the extracted features are quantized such that both the first feature data and the second feature data are integers.

4. The method according to claim 1 or 2, wherein the loss function is defined by the following mathematical equation: ,in It is the loss function, where It is a loss function related to the pose of the said feature, where It is a loss function that is related to the first data volume of the first feature data and / or the second data volume of the second feature data.

5. The method according to claim 4, wherein ,in The first parameter, where It is the expected value, where It is the first feature data and / or the second feature data, wherein It is the first data volume and / or the second data volume.

6. The method according to claim 4, wherein ,in It is the expected value, where It is the relative translation of the target, where It is the relative translation, where These are hyperparameters, among which The target is rotated relative to each other, where This refers to the relative rotation.

7. The method according to claim 4, wherein ,in It is the expected value, where It is the relative translation of the target, where It is the relative translation, where The target is rotated relative to each other, where It is the relative rotation, where It is the second parameter, where It is the third parameter.

8. A method for extracting features from an image of the environment surrounding a motor vehicle, the method comprising the following steps: Receive (301) image data signal, the image data signal representing image data of the surrounding environment of the motor vehicle; The artificial neural network (201, 505, 607) trained by any one of claims 1 to 7 extracts (303) features from the image; Output (305) a feature signal, which represents the extracted features.

9. A computing device (401) configured to implement all steps of the method according to any one of claims 1 to 7 and / or 8.

10. A computer program product comprising a computer program (503) including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7 and / or 8.

11. A machine-readable storage medium (501) having a computer program (503) stored thereon, wherein, The computer program (503) includes instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7 and / or 8.

Images