Computer-implemented methods for training an artificial neural network and driving a motor vehicle, data processing device and computer program product

Abstract

A teacher ANN is trained for a first and a second task. To train the teacher ANN for a third task, for each of a set of second training images (T2), a first set of features (F1) is generated by a first feature extraction module (E1) and a second set of features (F2) by a second feature extraction module (E2). An attention matrix is ​​generated based on the first and second sets of features (F1, F2). A third set of features (F3) is generated by a third feature extraction module (E3) based on the first and second sets of features (F1, F2) and the attention matrix. A prediction (P1) corresponding to the third task is generated. Parameters of the teacher ANN are updated based on the prediction (P1). A student ANN (4) is generated for the third task using knowledge distillation from the teacher ANN.

Description

[0001] The present invention is directed to a computer-implemented method for training a student artificial neural network (ANN), and to a computer-implemented method for computer vision, wherein such a computer-implemented method is carried out for training a student ANN. The invention further relates to a method for at least partially automatically driving a motor vehicle, wherein such a computer-implemented method for computer vision is carried out, a data processing device for carrying out the computer-implemented method, and a corresponding computer program product.

[0002] It is well known to use computer vision algorithms, especially trained ANNs, based on sensor data from a motor vehicle's environmental sensor systems to perform various driver assistance functions or other functions for automatic or partially automatic guidance of the vehicle. The performance of the computer vision algorithm may depend on the quality and / or quantity of the training data and, depending on the training method used, on the respective ground truth annotations of the training data.

[0003] For example, detecting unknown or novel objects can pose a significant challenge for common L2 / L2+ learning-based computer vision algorithms. Conventional computer vision algorithms may be trained on specific object categories, such as pedestrians, curbs, vehicles, cyclists, and so on. When confronted with objects that do not fall into these categories, such as a kangaroo or a horse-drawn carriage, these algorithms exhibit limitations in, for example, accurately detecting and / or classifying objects. This limits their ability to handle diverse real-world scenarios and necessitates the development of more robust and adaptable computer vision algorithms.

[0004] For example, it is possible to define a specific class for unknown objects and drastically increase the complexity of the computer vision algorithm to address these problems. Existing solutions require large-scale vision models, for example, based on vision transformer networks containing 500 million to 22 billion parameters, and state-of-the-art GPU servers with large memory ranging from 30 GB to 200 GB, such as Nvidia A100 or H100 GPU clusters, with high computing power, such as 2,000 TOPS or more. Such large-scale models are impractical for use in computing devices with limited resources, such as those found in automobiles or embedded systems. Vision transformer networks are presented in the paper by A. Dosovitskiy et al.: "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (arXiv:2010.11929v2).

[0005] Attention mechanisms for ANNs are also known, see for example the publication A. Vaswani et al: “Attention Is All You Need” (ArXiv:1706.03762v7).

[0006] Knowledge distillation is a process that uses a large model as a teacher ANN and transfers some of the teacher ANN's knowledge to a smaller student ANN, for example, by forcing the student ANN to learn to predict the same prediction data as the teacher ANN or by forcing the teacher ANN to learn the relevant feature representations in a latent space. In known approaches, the teacher ANN is trained with a very large amount of training data.

[0007] It is an object of the present invention to provide a method that uses knowledge distillation from a teacher ANN to train a student ANN to perform a computer vision algorithm, wherein the training effort, in particular for training the teacher ANN, is reduced.

[0008] This object is achieved by the subject matter of the independent claim. Further embodiments and preferred configurations are the subject matter of the dependent claims.

[0009] The invention is based on the idea of ​​training the teacher ANN to perform at least a first computer vision task, a second computer vision task, and a third computer vision task. The student ANN is trained using knowledge distillation from the teacher ANN for the third computer vision task, in particular only for the third computer vision task. When training the teacher ANN for the third computer vision task, an attention matrix, also referred to as an attention map, is generated based on the features of the already trained first computer vision task and the already trained second computer vision task. The attention matrix is ​​used to train the teacher ANN for the third computer vision task.

[0010] According to one aspect of the invention, a computer-implemented method for training a student ANN, in particular using knowledge distillation from a teacher ANN, is provided. The teacher ANN is trained for a first computer vision task based on a plurality of first training images, wherein the teacher ANN includes a first feature extraction module and a first classification module for performing the first computer vision task. The teacher ANN is trained for a second computer vision task based on the plurality of first training images, wherein the teacher ANN includes a second feature extraction module and a second classification module for performing the second computer vision task.

[0011] The teacher ANN is trained for a third computer vision task, in particular after the teacher ANN has been trained for the first computer vision task and the second computer vision task, for example, when the training of the teacher ANN for the first computer vision task and the second computer vision task is completed. The teacher ANN is trained for the third computer vision task based on a plurality of second training images, wherein the teacher ANN includes a third feature extraction module and a third classification module to perform the third computer vision task. To train the teacher ANN for the third computer vision task, the following steps i) to v) are performed for every second training image of the plurality of third training images. i) A first set of features is generated by the first feature extraction module depending on the respective second training image and a second set of features is generated by the second feature extraction module depending on the respective second training image. ii) A first attention matrix is ​​generated by an attention module, in particular a self-attention module, of the teacher ANN depending on the first set of features and the second set of features. iii) A third set of features is generated by the third feature extraction module depending on the first set of features and the second set of features and the first attention matrix. iv) A first prediction according to the first computer vision task is generated by the third classification module depending on the third set of features. v) Network parameters, such as weighting factors and / or bias factors, of the third feature extraction module and / or the third classification module are updated based on the first prediction. For example, the network parameters of the attention module can also be updated based on the first prediction.

[0012] The student ANN is trained for the third computer vision task based on a plurality of third training images using knowledge distillation from the teacher ANN, particularly after the teacher ANN has been trained for the third computer vision task, for example, when the training of the teacher ANN for the third computer vision task is completed.

[0013] Unless otherwise stated, all steps of the computer-implemented method for training the student ANN can be performed by a data processing device that includes at least one computing unit. In particular, the at least one computing unit is configured or adapted to perform the steps of the computer-implemented method for training the student ANN. For this purpose, the at least one computing unit can, for example, store a computer program that includes instructions that, when executed by the at least one computing unit, cause the at least one computing unit to execute the computer-implemented method for training the student ANN.

[0014] The Student ANN and the Teacher ANN are referred to as such according to the conventions in the field of knowledge distillation, particularly because the Teacher ANN effectively passes the knowledge about the third computer vision algorithm to the Student ANN. The Teacher ANN and the Student ANN may also be referred to as teachers and students, respectively, hereinafter. Knowledge distillation can be performed using known methods. However, specific methods are also described below for several embodiments of the computer-implemented method according to the invention, with which knowledge distillation can be performed in a particularly advantageous manner.

[0015] A computer vision task, also referred to as a visual perception task, can be understood, for example, as a task for extracting visual information from image data. In particular, the visual perception task can, in principle, be performed by a human capable of visually perceiving an image corresponding to the image data. In the present context, however, visual perception tasks are performed automatically without the need for human assistance. For example, an ANN for performing a computer vision task can be understood as an image processing algorithm or an image analysis algorithm.For example, the computer vision task may include an object detection task, an object classification task, a segmentation task, and / or a depth estimation task, an optical flow estimation task, and so on. Corresponding algorithms may also be performed based on input data other than visually perceivable images. For example, point clouds or thermal images from thermal imaging cameras, etc., can also be analyzed using appropriate computer algorithms. Here and in the following, the term "computer vision task" also includes such tasks.

[0016] In particular, here and below, an image can be a camera image, for example, from an RGB camera or a thermal imaging camera. A camera image contains a two-dimensional arrangement of image pixels for each of one or more channels, for example, color channels. However, an image can also be another data set in the aforementioned format based on raw data from another environmental sensor system, for example, a lidar sensor system, a radar sensor system, or an ultrasonic sensor system.

[0017] The form of the prediction according to a computer vision task depends on the specific underlying computer vision task. For example, a prediction of an object detection task may include one or more bounding boxes defining a spatial location and, optionally, orientation of one or more respective objects in the environment and / or corresponding object classes for the one or more objects. A prediction of a semantic segmentation algorithm may include a pixel level class for each pixel of an image, and so forth.

[0018] A variety of ANNs capable of performing such computer vision tasks are known, and corresponding architectures can be used for the student ANN and the teacher ANN in the present context, in particular for the architectures of the respective feature extraction and classification modules.

[0019] The plurality of first training images and the plurality of second training images generally differ from each other. However, in some embodiments, these pluralities may overlap, i.e., the plurality of first training images may include training images that are also included in the plurality of second training images, or vice versa.

[0020] Since the first attention matrix is ​​used to train the teacher ANN, the previously acquired knowledge of the teacher ANN about the first computer vision task or the second computer vision task with respect to the first set of features and the second set of features is exploited. The attention module effectively tells the third classification module which features of the first set of features and the second set of features should be used primarily for performing the third computer vision task, or where in the respective image the relevant information for performing the third computer vision task can be found.Consequently, the number of training images used to train the teacher for the third computer vision task can be significantly reduced compared to the number of training images used to train the teacher for the first computer vision task and the second computer vision task.

[0021] To train the teacher for the first computer vision task and the second computer vision task, known training procedures for the respective types of computer vision tasks can be used.

[0022] According to several embodiments, the plurality of first training images includes a first number N1 of first training images and the plurality of second training images includes a second number N2 of second training images, where N2 < N1.

[0023] This reduces the overall effort required to train the teacher ANN. In particular, the second number N2 can be significantly lower than the first number N1, for example, N1 / N2 > 100.

[0024] According to several embodiments, the plurality of third training images includes a third number N3 of third training images, where N2 < N3.

[0025] This improves the performance of the Student ANN after training. Note that the third number N3 can still be lower than the first number N1, thus balancing the effort required to train the Student ANN and its performance. In other words, N2 < N3 < N1 can hold, for example.

[0026] According to several embodiments, to train the teacher ANN for the third computer vision task, for each of the plurality of second training images, an encoded representation of the respective second training image is generated by applying a first vision transformer network of the teacher ANN to the respective second training image. For each of the plurality of second training images, the first set of features is generated by applying the first feature extraction module to the encoded representation of the respective second training image, and the second set of features is generated by applying the second feature extraction module to the encoded representation of the respective second training image.

[0027] To generate the coded representation of each second training image, the second training image can be treated as a plurality of patches that are coded separately. Consequently, the size of the first feature extraction module and the size of the first feature extraction module can be reduced. A large number of second training images can therefore be processed in a relatively short time.

[0028] According to several embodiments, to train the student ANN for the third computer vision task, for each of the third training images, a fourth set of features is generated by the first feature extraction module depending on the respective third training image, and a fifth set of features is generated by the second feature extraction module depending on the respective third training image. A second attention matrix is ​​generated by the attention module depending on the fourth set of features and the fifth set of features. A sixth set of features is generated by the third feature extraction module depending on the fourth set of features and the fifth set of features and the second attention matrix. A seventh set of features is generated by applying a second vision transformer network of the student ANN to the respective third training image.A similarity measure of the sixth set of features and the seventh set of features is calculated, and the network parameters of the student ANN, specifically the second vision transformer network of the student ANN, are updated depending on the similarity measure.

[0029] The similarity measure quantifies a similarity of the sixth set of features and the seventh set of features to each other.

[0030] In particular, a loss function for training the student ANN includes the similarity measure or depends on the similarity measure. The similarity measure can, for example, be a cosine similarity function or can be calculated using the cosine similarity function. In other embodiments, however, the similarity measure can also be a distance function, for example, a Euclidean distance, or can be calculated using the distance function. Depending on the type of similarity measure, it can be integrated accordingly into the loss function for training the student ANN.

[0031] By updating the network parameters of the student ANN depending on the similarity measure, the student ANN, especially the second vision transformer network, is forced to generate the seventh set of features similarly to the sixth set of features generated by the third feature extraction module of the teacher ANN. This achieves a particularly efficient implementation of knowledge distillation.

[0032] According to several embodiments, to train the student ANN for the third computer vision task, for each of the third training images, a fourth set of features is generated by the first feature extraction module depending on the respective third training image, and a fifth set of features is generated by the second feature extraction module depending on the respective third training image. A second attention matrix is ​​generated by the attention module depending on the fourth group of features and the fifth group of features. A sixth set of features is generated by the third feature extraction module depending on the fourth set of features and depending on the fifth set of features and depending on the second attention matrix. A seventh set of features is generated by applying a fourth feature extraction module of the student ANN to the respective third training image.A similarity measure of the sixth set of features and the seventh set of features is calculated, and the network parameters of the student ANN, especially the fourth feature extraction module of the student ANN, are updated depending on the similarity measure.

[0033] The similarity measure quantifies a similarity of the sixth set of features and the seventh set of features to each other.

[0034] By updating the network parameters of the student ANN depending on the similarity measure, the student ANN, specifically the fourth feature extraction module, is forced to generate the seventh set of features similarly to the sixth set of features generated by the third feature extraction module of the teacher ANN. This achieves a particularly efficient implementation of knowledge distillation.

[0035] According to several embodiments, a second prediction according to the third computer vision task is generated by a fourth classification module of the student ANN depending on the seventh set of features, and the network parameters of the student ANN are updated depending on the second prediction.

[0036] By updating the network parameters of the student ANN depending on the second prediction, the student ANN, specifically the fourth classification module of the student ANN, is trained to perform the third computer vision task based on the seventh set of features, the generation of which is forced to match the corresponding features of the teacher ANN.

[0037] According to several embodiments, the first computer vision task includes a depth estimation task and / or an image reconstruction task and / or an optical flow estimation task and / or an object detection task and / or a semantic segmentation task.

[0038] The latent features of the first computer vision task, which are used for the attention mechanism described above, are therefore particularly useful for training the teacher ANN for the third computer vision task and consequently also for training the student ANN.

[0039] According to several embodiments, the second computer vision task includes a depth estimation task and / or an image reconstruction task and / or an optical flow estimation task and / or an object detection task and / or a semantic segmentation task.

[0040] The latent features of the second computer vision task, which are used for the attention mechanism as described above, are therefore particularly useful for training the teacher ANN for the third computer vision task and consequently also for training the student ANN.

[0041] It should be noted that in some embodiments, the first computer vision task and the second computer vision task may also be the same type of computer vision task. In particular, both the first computer vision task and the second computer vision task may be, for example, depth estimation tasks, image reconstruction tasks, optical flow estimation tasks, object detection tasks, semantic segmentation tasks, and so on. Even in such cases, the first computer vision task and the second computer vision task are not necessarily identical. For example, the first computer vision task and the second computer vision task may be trained for an object detection task based on different classes, and so on.Alternatively or additionally, the first computer vision task and the second computer vision task can be trained based on images from different environmental sensor modalities, for example, different types of cameras, lidar sensor systems, radar systems, ultrasonic sensor systems, and so on.

[0042] However, in some embodiments, the first computer vision task and the second computer vision task are different computer vision tasks.

[0043] In this way, the latent features from the first computer vision task and the second computer vision task together cover a particularly wide range of features. This can improve the usefulness of the attention matrix for training the teacher ANN for the third computer vision task.

[0044] According to several embodiments, the third computer vision task includes an object detection task and / or a semantic segmentation task.

[0045] These tasks benefit particularly well from the use of latent features in the form of the attention matrix.

[0046] The third computer vision task and the second computer vision task may be different types of computer vision tasks, and the third computer vision task and the first computer vision task may also be different types of computer vision tasks. However, in some embodiments, the third computer vision task and the second computer vision task are of the same type and / or the third computer vision task and the first computer vision task are of the same type. Reference is made to the above explanations regarding the first computer vision task and the second computer vision task.

[0047] According to several embodiments, the teacher ANN includes at least 10 8 trainable parameters.

[0048] The trainable parameters include, in particular, weighting factors and / or bias factors of the teacher ANN, especially of all modules of the teacher ANN. The teacher ANN is therefore a very large ANN and thus potentially exhibits a very high degree of accuracy, robustness, and reliability with respect to the first computer vision task, the second computer vision task, and the third computer vision task. This makes it particularly suitable for knowledge distillation.

[0049] For example, the teacher ANN can have at least 5*10 8 up to 22*10 9 contain trainable parameters.

[0050] According to several embodiments, the ratio of the number of trainable parameters of the teacher ANN to a number of trainable parameters of the student ANN is at least 100.

[0051] In other words, the Student ANN is at least 100 times smaller than the Teacher ANN. Consequently, the Student ANN is particularly suitable for applications, especially in a motor vehicle or embedded computing environment, as it requires relatively few computational resources in the form of runtime and / or memory. For example, the ratio of the number of trainable parameters of the Teacher ANN to the number of trainable parameters of the Student ANN can be between 100 and 10,000 or between 100 and 1,000.

[0052] According to a further aspect of the invention, a computer-implemented method for computer vision is provided. A student ANN is trained using a computer-implemented method according to the invention for training a student ANN. A camera image is obtained, in particular from a camera, for example a camera of a motor vehicle, and a third prediction according to the third computer vision task is generated by applying the trained student ANN to the camera image.

[0053] The camera is, for example, a camera in the visible range, for example an RGB camera, or a thermal imaging camera or a time-of-flight camera.

[0054] Unless otherwise stated, all steps of the computer-implemented method for computer vision can be carried out by a further data processing device which includes at least one further computing unit and can, for example, be a further data processing device of the motor vehicle. In particular, the at least one further computing unit is set up or adapted to carry out the steps of the computer-implemented method for computer vision. For this purpose, the at least one further computing unit can, for example, store a further computer program which includes instructions which, when executed by the at least one further computing unit, cause the at least one further computing unit to carry out the computer-implemented method for computer vision.

[0055] All further computing units of the at least one further computing unit can be contained in the motor vehicle. However, it is also possible for all further computing units of the at least one further computing unit to be part of an external computing system external to the motor vehicle, for example a backend server or a cloud computing system. It is also possible for the at least one further computing unit to contain at least one vehicle computing unit of the motor vehicle and at least one external computing unit contained in the external computing system. The at least one vehicle computing unit can, for example, be comprised of one or more electronic control units (ECUs) and / or one or more zone control units (ZCUs) and / or one or more domain control units (DCUs) of the motor vehicle and / or the camera.

[0056] According to a further aspect of the invention, a method for at least partially automatically driving a motor vehicle is provided. A computer-implemented method for computer vision according to the invention is carried out. At least one control signal for at least partially automatically driving the motor vehicle is generated by at least one computing unit, for example of the motor vehicle, depending on the third prediction, and / or assistance information for supporting a driver of the motor vehicle in driving the motor vehicle is generated by the at least one computing unit depending on the third prediction.

[0057] The at least one control signal can, for example, be provided to one or more actuators of the motor vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least partially automatically.

[0058] The assistance information can be output via an output device of the motor vehicle, for example a display and / or an audio output system and / or a haptic output system.

[0059] For example, the motor vehicle may include an electronic vehicle guidance system adapted to carry out the method according to the invention for at least partially automatically guiding the motor vehicle. In particular, the electronic vehicle guidance system may include the aforementioned at least one computing unit and, in some embodiments, the camera.

[0060] An electronic vehicle guidance system can be understood as an electronic system designed to guide a vehicle fully automatically or autonomously, in particular without requiring driver intervention. The vehicle automatically performs all required functions, such as steering, braking, and / or acceleration maneuvers, monitoring and detecting road traffic, and corresponding reactions. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode of the motor vehicle according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be understood as an advanced driver assistance system (ADAS), which supports the driver in partially automated or semi-autonomous driving.In particular, the electronic vehicle guidance system can implement a partially automated or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and below, "SAE J3016" refers to the corresponding standard in the April 2021 version.

[0061] The at least partially automated vehicle guidance may therefore include driving the vehicle according to a fully automated or fully autonomous driving mode of Level 5 according to SAE J3016. The at least partially automated vehicle guidance may also include driving the vehicle according to a partially automated or semi-autonomous driving mode according to Levels 1 to 4 according to SAE J3016.

[0062] According to a further aspect of the invention, a data processing device with at least one computing unit is provided. The at least one computing unit is adapted to perform a computer-implemented method for training a student ANN according to the invention.

[0063] According to a further aspect of the invention, a further data processing device is provided, which includes at least one further computing unit. The at least one further computing unit is adapted to carry out a computer-implemented method for computer vision according to the invention.

[0064] A computing unit can be understood, in particular, as a data processing device that contains a processing circuit. The computing unit can therefore, in particular, process data to perform computing operations. This may also include operations for performing indexed access to a data structure, for example, a look-up table (LUT).

[0065] The computing unit may in particular contain one or more computers, one or more microcontrollers and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit may also contain one or more

[0066] Processors, for example, one or more microprocessors, one or more central processing units (CPU), one or more graphics processing units (GPU), and / or one or more signal processors, in particular one or more digital signal processors (DSP). The computing unit can also include a physical or virtual network of computers or other of the aforementioned units.

[0067] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0068] A memory unit can be a volatile data memory, for example a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory,MRAM (magnetoresistive random access memory) or phase-change random access memory (PCRAM).

[0069] According to a further aspect of the invention, a computer program is provided that includes instructions. When the instructions are executed by a data processing device, in particular by a data processing device according to the invention, the instructions cause the data processing device to perform a computer-implemented method according to the invention for training a student ANN.

[0070] The instructions may be provided, for example, as program code. The program code may be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0071] According to a further aspect of the invention, a further computer program is provided that includes further instructions. When the further instructions are executed by a further data processing device, in particular by a further data processing device according to the invention, the further instructions cause the further data processing device to perform a computer-implemented method for computer vision according to the invention.

[0072] The further instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0073] According to a further aspect of the invention, a computer-readable storage medium is provided on which a computer program according to the invention and / or another computer program according to the invention is stored.

[0074] The computer program, the further computer program and the computer-readable storage medium are each computer program products with the instructions and / or the further instructions.

[0075] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, the invention can also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention can encompass embodiments and combinations of features that go beyond the combinations of features set out in the backreferences to the claims or deviate from them.

[0076] The invention is explained in more detail below using specific embodiments and associated schematic drawings. In the figures, identical or functionally equivalent elements may be provided with the same reference numerals. The description of identical or functionally equivalent elements may not necessarily be repeated for different figures.

[0077] The figures show Fig. 1 schematically shows a motor vehicle with an exemplary embodiment of a data processing device according to the invention; Fig. 2 is a schematic block diagram of a teacher ANN for use in an exemplary embodiment of a computer-implemented method for training a student ANN according to the invention; Fig. 3 is a schematic block diagram of a teacher ANN and a student ANN for use in another exemplary embodiment of a computer-implemented method for training a student ANN according to the invention; Fig. 4 is a schematic block diagram of a teacher ANN and a student ANN for use in another exemplary embodiment of a computer-implemented method for training a student ANN according to the invention; Fig. 5 is a schematic block diagram of a teacher ANN for use in another exemplary embodiment of a computer-implemented method for training a student ANN according to the invention; and Fig. 6 is a schematic block diagram of a teacher ANN and a student ANN for use in another exemplary embodiment of a computer-implemented method for training a student ANN according to the invention.

[0078] Fig. 1 schematically shows a motor vehicle 1 with an exemplary embodiment of a data processing device according to the invention. The data processing device, which can be part of an electronic vehicle guidance system 5 of the motor vehicle 1, includes at least one computing unit 3 that stores a student ANN 4 trained by means of a computer-implemented method according to the invention for training a student ANN 4. The motor vehicle 1, in particular the electronic vehicle guidance system 5, includes a camera 2 that is configured to generate a camera image representing an environment of the motor vehicle 1 and to provide the camera image to the at least one computing unit 3.

[0079] The at least one computing unit 3 performs a third computer vision task, for example an object recognition task or a semantic segmentation task, in order to generate a third prediction according to the third computer vision task, for example one or more bounding boxes enclosing respective objects in the environment represented by the camera image or a semantically segmented image by applying the trained student ANN 4 to the camera image.

[0080] The at least one computing unit 3 generates at least one control signal for at least partially automatically guiding the motor vehicle 1 as a function of a result of the third computer vision task, in particular as a function of the third prediction, and / or generates assistance information for supporting a driver of the motor vehicle 1 in guiding the motor vehicle 1 as a function of the result of the computer vision task, in particular as a function of the third prediction.

[0081] The student ANN 4 is trained by a further data processing device adapted to carry out a computer-implemented method according to the invention for training a student ANN, in particular using knowledge distillation from the teacher ANN.

[0082] Fig. 2 shows a schematic block diagram of a teacher ANN for use in an exemplary embodiment of a computer-implemented method for training the student ANN 4 according to the invention.

[0083] The teacher ANN is trained based on a plurality of first training images for a first computer vision task, wherein the teacher ANN includes a first feature extraction module E1 and a first classification module C1 for performing the first computer vision task. The teacher ANN is trained based on the plurality of first training images for a second computer vision task, wherein the teacher ANN includes a second feature extraction module E2 and a second classification module C2 for performing the second computer vision task. After training the teacher ANN for the first computer vision task and the second computer vision task, the teacher ANN is trained based on a plurality of second training images T2 for a third computer vision task, wherein the teacher ANN includes a third feature extraction module E3 and a third classification module C3 for performing the third computer vision task.

[0084] To train the teacher ANN for the third computer vision task, the following steps are performed for each of the plurality of second training images T2. A first set of features F1 is generated by the first feature extraction module E1 depending on the respective second training image, and a second set of features F2 is generated by the second feature extraction module E2 depending on the respective second training image. A first attention matrix is ​​generated by an attention module A of the teacher ANN depending on the first set of features F1 and the second set of features F2. A third set of features F3 is generated by the third feature extraction module E3 depending on the first set of features F1 and the second set of features F2 and the first attention matrix.A first prediction P1 according to the first computer vision task is generated by the third classification module C3 based on the third set of features F3. Network parameters of the third feature extraction module E3 and / or the third classification module C3 are updated based on the first prediction P1. For example, network parameters of the attention module A can also be updated based on the first prediction P1.

[0085] When the training of the teacher ANN for the third computer vision task is completed, the student ANN 4 is trained based on a plurality of third training images using knowledge distillation from the teacher ANN for the third computer vision task.

[0086] For example, the teacher ANN may include a first vision transformer network TR1. In such embodiments, the first vision transformer network TR1 may be applied to the respective second training image to generate an encoded representation of the respective second training image. The first set of features F1 is generated by applying the first feature extraction module E1 to the encoded representation of the respective second training image, and the second set of features F2 is generated by applying the second feature extraction module E2 to the encoded representation of the respective second training image. In other embodiments, the first feature extraction module E1 and the second feature extraction module E2 may be applied directly to the respective second training image to generate the first set of features F1 and the second set of features F2.

[0087] Fig. Figure 3 shows a schematic block diagram of a teacher ANN, for example the teacher ANN from Fig. 2, and a student ANN 4 for use in a further exemplary embodiment of a computer-implemented method for training a student ANN 4 according to the invention. The teacher ANN was trained as in Fig. 2 explained.

[0088] To train the student ANN 4 for the third computer vision task, for each of the third training images T3, a fourth set of features F4 is generated by the first feature extraction module E1 based on the respective third training image, and a fifth set of features F5 is generated by the second feature extraction module E2 based on the respective third training image. A second attention matrix is ​​generated by the attention module A based on the fourth set of features F4 and the fifth set of features F5. A sixth set of features F6 is generated by the third feature extraction module E3 based on the fourth set of features F4 and the fifth set of features F5 and the second attention matrix.

[0089] A seventh set of features F7 is generated by applying a fourth feature extraction module E4 of the student ANN 4 to the respective third training image. A similarity measure, e.g., a cosine similarity, of the sixth set of features F6 and the seventh set of features F7 is calculated by an attention block S1, and the network parameters of the student ANN 4 are updated depending on the similarity measure.

[0090] If the teacher ANN includes the first vision transformer network TR1, the first vision transformer network TR1 can be applied to the respective third training image to generate an encoded representation of the respective third training image. The fourth set of features F4 is generated by applying the first feature extraction module E1 to the encoded representation of the respective third training image, and the fifth set of features F5 is generated by applying the second feature extraction module E2 to the encoded representation of the respective third training image. In other embodiments, the first feature extraction module E1 and the second feature extraction module E2 can be applied directly to the respective third training image to generate the fourth set of features F4 and the fifth set of features F5.

[0091] Fig. Figure 4 shows a schematic block diagram of a teacher ANN, for example the teacher ANN from Fig. 2, and a student ANN 4 for use in a further exemplary embodiment of a computer-implemented method according to the invention for training a student ANN 4. The teacher ANN was trained as described with respect to Fig. 2 explained.

[0092] To train the student ANN 4 for the third computer vision task, for each of the third training images T3, a fourth set of features F4 is generated by the first feature extraction module E1 based on the respective third training image, and a fifth set of features F5 is generated by the second feature extraction module E2 based on the respective third training image. A second attention matrix is ​​generated by the attention module A based on the fourth set of features F4 and the fifth set of features F5. A sixth set of features F6 is generated by the third feature extraction module E3 based on the fourth set of features F4 and the fifth set of features F5 and the second attention matrix.

[0093] A seventh set of features F7 is generated by applying a second vision transformer network TR2 of the student ANN 4 to the respective third training image. A similarity measure, e.g., a cosine similarity, of the sixth set of features F6 and the seventh set of features F7 is calculated by the similarity block S1, and the network parameters of the student ANN 4 are updated depending on the similarity measure.

[0094] If the teacher ANN includes the first vision transformer network TR1, the first vision transformer network TR1 can be applied to the respective third training image to generate an encoded representation of the respective third training image. The fourth set of features F4 is generated by applying the first feature extraction module E1 to the encoded representation of the respective third training image, and the fifth set of features F5 is generated by applying the second feature extraction module E2 to the encoded representation of the respective third training image. In other embodiments, the first feature extraction module E1 and the second feature extraction module E2 can be applied directly to the respective third training image to generate the fourth set of features F4 and the fifth set of features F5.

[0095] With reference to the embodiments of Fig. 3 and Fig. 4, a second prediction P2 according to the third computer vision task may be generated by a fourth classification module C4 of the student ANN 4 depending on the seventh set of features F7, and the network parameters of the student ANN 4 are updated depending on the similarity measure and the second prediction P2.

[0096] Fig. Figure 5 shows a schematic block diagram of a teacher ANN for use in an exemplary embodiment of a computer-implemented method for training the student ANN 4 according to the invention. The teacher ANN is based on the teacher ANN described in Fig. 2 is described.

[0097] The teacher ANN is trained for the first and second computer vision tasks as in Fig. 2. Furthermore, the teacher ANN is trained for a further computer vision task based on the plurality of first training images, in particular before the teacher ANN is trained for the third computer vision task, wherein the teacher ANN includes a fifth feature extraction module E5 and a fifth classification module C5 to perform the further computer vision task.

[0098] For example, the first feature extraction module E1 can be an image reconstruction encoder, and the first classification module C1 can be an image reconstruction decoder. The first computer vision task in this case is therefore an image reconstruction task. For example, the second feature extraction module E2 can be an optical flow and / or motion estimation encoder, and the second classification module C2 can be an optical flow and / or motion estimation decoder. The second computer vision task in this case is therefore an optical flow and / or motion estimation task. For example, the fifth feature extraction module E5 can be a depth estimation encoder, and the fifth classification module C5 can be a depth estimation decoder. The further computer vision task in this case is therefore a depth estimation task.

[0099] In some embodiments, when training the teacher ANN for the third computer vision task, the first attention matrix may be generated by the attention module 6 depending on a further set of features generated by the fifth feature extraction module E5 depending on the respective second training image. In particular, the first attention matrix may be generated by the attention module 6 depending on the first set of features F1 and the second set of features F2, or it may be generated depending on the first set of features F1 and the further set of features, or it may be generated depending on the second set of features F2 and the further set of features, or it may be generated depending on the first set of features F1 and the second set of features F2 and the further set of features.This concept can be analogously extended to further computer vision tasks, for which the teacher ANN is trained based on the large number of initial training images, especially before the teacher ANN is trained for the third computer vision task.

[0100] The attention module 6 can, for example, contain a first attention block A1 and a second attention block A2. This structure can be understood such that the first attention block A1 specifies which features from the first, second, and / or further computer vision task are primarily required to perform the third computer vision task. The respective sets of keys K1, queries, and values ​​V1 are thus contained in the features generated by the first encoder module E1, the second encoder module E2, and / or the fifth encoder module E5. The second attention block A2 indicates where the objects relevant to the third computer vision task are located in the respective training image.The corresponding values ​​V2 are obtained as output of the first attention block A1, the respective keys K2 are, for example, extrinsic and / or intrinsic camera calibration parameters CC of camera 2, and the queries Q are obtained as embeddings.

[0101] For example, the third feature extraction module E3 can be a semantic segmentation encoder for image-level feature extraction, and the third classification module C3 can be a semantic segmentation decoder for classification. Thus, the third computer vision task in this case is a semantic segmentation task.

[0102] In some embodiments, the teacher ANN includes, in addition to or instead of the third feature extraction module E3 and the third classification module C3, a sixth feature extraction module E6 and a sixth classification module C6. For example, the sixth feature extraction module E6 may include a BEV projection module followed by a BEV encoder (BEV: bird's-eye view), and the sixth classification module C6 may be a semantic segmentation decoder for classification in the BEV perspective. The sixth feature extraction module E6 and the sixth classification module C6 may therefore be used to perform a further third computer vision task, which is semantic segmentation in the BEV perspective.As input data, the sixth feature extraction module E6 receives the attention matrix from the attention module 6 as well as the extrinsic and / or intrinsic camera calibration parameters CC of the camera 2.

[0103] The above statements regarding the third computer vision task and the training of the teacher ANN for the third computer vision task can be applied analogously to the further third computer vision task and the training of the teacher ANN for the further third computer vision task.

[0104] Fig. Figure 6 shows a schematic block diagram of a teacher ANN, for example the teacher ANN of Fig. 5, and a student ANN 4 for use in a further exemplary embodiment of a computer-implemented method according to the invention for training a student ANN 4. The student ANN 4 can be based on the student ANN 4 from Fig. 4. The teacher ANN was trained as to Fig.5. The second vision transformer network TR2 may, for example, include an image encoder module TRE1 and an image decoder module TRD2.

[0105] In embodiments in which the teacher ANN includes the sixth feature extraction module E6 and the sixth classification module C6, the student ANN 4 can include a seventh feature extraction module E7 and a seventh classification module C7 to perform the further computer vision task. The above explanations regarding the third computer vision task and the training of the student ANN 4 for the third computer vision task can be applied analogously to the further third computer vision task and the training of the student ANN 4 for the further third computer vision task.

[0106] As described, in particular with reference to the figures, the invention provides a way to use knowledge distillation from the teacher ANN to train a student ANN to perform a computer vision algorithm, in particular reducing the effort for training the teacher ANN.

[0107] In some embodiments, a multimodal knowledge distillation process is implemented, enabling the use of models requiring fewer than 5 TOPs.

[0108] In some embodiments, an optimized workflow is provided that facilitates the creation of fully data-driven neural network models that can adapt their design to different tasks, for example, object detection or segmentation, sign detection, wheel stop detection, and so on.

[0109] In some embodiments, the teacher ANN is capable of handling multiple computer vision tasks with very high accuracy. It may initially be trained on fully labeled data using a single-prompt approach. The teacher ANN may also be a pre-trained large vision model.

[0110] In some embodiments, one or more auxiliary tasks are integrated into the teacher ANN to facilitate the learning of a variety of features. The auxiliary tasks may result in further training of the feature space for the required tasks of the teacher ANN, such as depth prediction or pose detection, and so on.

[0111] In some embodiments, the Student ANN uses the features extracted by the Teacher ANN to perform further computer vision tasks, such as object detection or segmentation, specifically focusing on previously "unseen" objects.

[0112] In some embodiments, a multimodal, multiheaded self-attention module is used. The features obtained from the branches of the teacher ANN can be fused with this module, leveraging multimodal information and self-attention mechanisms.

[0113] In some embodiments, the knowledge distillation approach replicates the behavior of the teacher ANN and transfers its knowledge to the student ANN. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] A. Dosovitskiy et al.: “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” (arXiv:2010.11929v2

[0004] A. Vaswani et al: “Attention Is All You Need” (ArXiv:1706.03762v7

[0005] computer vision task depends on the specific underlying computer vision task

[0017]

Claims

[1] Computer-implemented method for training a student artificial neural network, ANN, (4), where - a teacher ANN is trained based on a plurality of first training images for a first computer vision task, wherein the teacher ANN includes a first feature extraction module (E1) and a first classification module (C1) for performing the first computer vision task; - the teacher ANN is trained based on the plurality of first training images for a second computer vision task, wherein the teacher ANN includes a second feature extraction module (E2) and a second classification module (C2) for performing the second computer vision task; - the teacher ANN is trained based on a plurality of second training images (T2) for a third computer vision task, wherein the teacher ANN includes a third feature extraction module (E3) and a third classification module (C3) for performing the third computer vision task; - to train the teacher ANN for the third computer vision task, for each of the plurality of second training images (T2), i) a first set of features (F1) is generated by the first feature extraction module (E1) depending on the respective second training image and a second set of features (F2) is generated by the second feature extraction module (E2) depending on the respective second training image; ii) a first attention matrix is ​​generated by an attention module (A) of the teacher ANN depending on the first set of features (F1) and the second set of features (F2); iii) a third set of features (F3) is generated by the third feature extraction module (E3) depending on the first set of features (F1) and the second set of features (F2) and the first attention matrix; iv) a first prediction (P1) according to the third computer vision task is generated by the third classification module (C3) depending on the third set of features (F3); and v) network parameters of the third feature extraction module (E3) and / or the third classification module (C3) are updated depending on the first prediction (P1); and - the student ANN (4) is trained for the third computer vision task based on a plurality of third training images (T3) using knowledge distillation from the teacher ANN. [2] The computer-implemented method of claim 1, wherein the plurality of first training images includes a first number N1 of first training images and the plurality of second training images (T2) includes a second number N2 of second training images (T2), where N2 < N1. [3] The computer-implemented method of claim 2, wherein the plurality of third training images (T3) includes a third number N3 of third training images (T3), where N2 < N3. [4] Computer-implemented method according to one of the preceding claims, wherein for training the teacher ANN for the third computer vision task for each of the plurality of second training images (T2) - a coded representation of the respective second training image is generated by applying a first vision transformer network (TR1) of the teacher ANN to the respective second training image; and - the first set of features (F1) is generated by applying the first feature extraction module (E1) to the coded representation of the respective second training image and the second set of features (F2) is generated by applying the second feature extraction module (E2) to the coded representation of the respective second training image. [5] Computer-implemented method according to one of the preceding claims, wherein for training the student ANN (4) for the third computer vision task for each of the plurality of third training images (T3) - a fourth set of features (F4) is generated by the first feature extraction module (E1) depending on the respective third training image and a fifth set of features (F5) is generated by the second feature extraction module (E2) depending on the respective third training image; - a second attention matrix is ​​generated by the attention stage (A) depending on the fourth set of features (F4) and the fifth set of features (F5); - a sixth set of features (F6) is generated by the third feature extraction module (E3) depending on the fourth set of features (F4) and the fifth set of features (F5) and the second attention matrix; - a seventh set of features (F7) is generated by applying a second vision transformer network (TR2) of the student ANN (4) to the respective third training image; - a similarity measure of the sixth set of features (F6) and the seventh set of features (F7) is calculated and the network parameters of the student ANN (4) are updated depending on the similarity measure. [6] Computer-implemented method according to one of claims 1 to 4, wherein for training the student ANN (4) for the third computer vision task for each of the third training images (T3) - a fourth set of features (F4) is generated by the first feature extraction module (E1) depending on the respective third training image and a fifth set of features (F5) is generated by the second feature extraction module (E2) depending on the respective third training image; - a second attention matrix is ​​generated by the attention stage (A) depending on the fourth set of features (F4) and the fifth set of features (F5); - a sixth set of features (F6) is generated by the third feature extraction module (E3) depending on the fourth set of features (F4) and the fifth set of features (F5) and the second attention matrix; - a seventh set of features (F7) is generated by applying a fourth feature extraction module (E4) of the student ANN (4) to the respective third training image; - a similarity measure of the sixth set of features (F6) and the seventh set of features (F7) is calculated and the network parameters of the student ANN (4) are updated depending on the similarity measure. [7] Computer-implemented method according to one of claims 5 or 6, wherein a second prediction (P2) according to the third computer vision task is generated by a fourth classification module (C4) of the student ANN (4) depending on the seventh set of features (F7) and the network parameters of the student ANN (4) are updated depending on the second prediction (P2). [8] A computer-implemented method according to any one of claims 5 to 7, wherein the similarity measure is calculated as cosine similarity. [9] Computer-implemented method according to one of the preceding claims, wherein - the first computer vision task includes a depth estimation task and / or an image reconstruction task and / or an optical flow estimation task and / or an object recognition task and / or a semantic segmentation task; and / or - the second computer vision task includes a depth estimation task and / or an image reconstruction task and / or an optical flow estimation task and / or an object recognition task and / or a semantic segmentation task. [10] A computer-implemented method according to any one of the preceding claims, wherein the third computer vision task includes an object recognition task and / or a semantic segmentation task. [11] A computer-implemented method according to any one of the preceding claims, wherein the teacher ANN comprises at least 10 8includes trainable parameters. [12] Computer-implemented method according to one of the preceding claims, wherein a ratio of the number of trainable parameters of the teacher ANN to a number of trainable parameters of the student ANN (4) is at least 100. [13] Computer-implemented method for computer vision, wherein - a student ANN (4) is trained using a computer-implemented method according to one of the preceding claims; and - a camera image is obtained and a third prediction according to the third computer vision task is generated by applying the trained student ANN (4) to the camera image. [14] Method for at least partially automatically driving a motor vehicle (1), wherein a computer-implemented method according to claim 13 is carried out and - at least one control signal for at least partially automatically guiding the motor vehicle (1) is generated by means of at least one computing unit (3) depending on the third prediction; and / or - Assistance information for supporting a driver of the motor vehicle (1) in driving the motor vehicle (1) is generated by means of the at least one computing unit (3) depending on the third prediction. [15] Data processing device comprising at least one computing unit (3) adapted to carry out a computer-implemented method according to one of claims 1 to 13. [16] A computer program product including instructions which, when executed by a data processing device, cause the data processing device to perform a computer-implemented method according to any one of claims 1 to 13.

Images