Training an artificial neural network for carrying out a computer vision task

Abstract

For training a student ANN (5), first features (18) are generated by applying a first feature module (9) of a teacher ANN (6) to a first dataset (7) corresponding to a first sensor type, second features (19) are generated by applying a second feature module (10) of the teacher ANN (6) to a second dataset (8) corresponding to a second sensor type, the first and the second features are fused, third features (21) are generated by applying a feature module (14) of the student ANN (5) to the first dataset (7), prediction data (16) is generated by applying a decoder module (15) of the student ANN (5) to the third features (21 ). Arguments of a loss function (17) for updating the student ANN (5) include the prediction data (16), ground truth data (22) for the first dataset (7), the fused features (20) and the third features (21).

Description

[0001] Training an artificial neural network for carrying out a computer vision task

[0002] The present invention is directed to a computer-implemented training method for training a student artificial neural network, ANN, for carrying out a computer vision task, wherein a trained teacher ANN is provided, which is trained to carry out the computer vision task. The invention is further directed to a computer-implemented method for carrying out a computer vision task, wherein such a computer-implemented training method is carried out, and to a method for guiding a motor vehicle at least in part automatically, wherein such a computer-implemented training method is carried out. The invention is also directed to a data processing apparatus, which is adapted to carry out such a computer- implemented training method or computer-implemented method. The invention is also directed to a corresponding electronic vehicle guidance system and to a corresponding computer program product.

[0003] Computer vision tasks, such as object detection, semantic segmentation, depth estimation and so forth, are widely used to perceive the environment of motor vehicles in order to implement various driver assistance functions or other functions for guiding the motor vehicle at least in part automatically. Electronic vehicle guidance systems of the motor vehicle may implement such functions. Algorithms for carrying out such computer vision tasks maybe based on trained artificial neural networks, ANNs, and can be applied to input data from different sensor types, also denoted as different sensor modalities, such as visual range cameras, thermal cameras, lidar sensor systems and so forth.

[0004] Any ANN trained with unimodal data, that is data from only one sensor modality, may have limited performance with respect to the nature of sensor modality. For example, an ANN trained with only visible range camera images may have a reduced performance at low-light scenarios or during nighttime, while an ANN trained with only thermal camera images may have a reduced performance regarding the extraction of semantic information or for objects with a large distance from the thermal camera.

[0005] To overcome this, it is well known to train ANNs with multimodal data, that is data from two or more different sensor modalities. However, deploying such a multimodal ANN on a target device may have drawbacks for various reasons. For example, using multiple sensor modalities increases the overall complexity and costs of the electronic vehicle guidance system. Furthermore, the target device may have limited computing resources, in particular in case of embedded systems for motor vehicles, while the use of a multimodal ANN typically requires more computational resources than using a unimodal ANN. Also the latency of the computer vision task may be larger for multimodal ANNs. Furthermore, it may also be required that safety relevant functions rely only on a single sensor.

[0006] It is an objective of the present invention, to overcome the limitations of a unimodal ANN due to the specific characteristics of the corresponding sensor modality at least partially.

[0007] This objective is achieved by the subject matter of the independent claim. Further implementations and preferred embodiments are subject matter of the dependent claims.

[0008] The invention is based on the idea to provide a teacher ANN, which is trained to carry out the computer vision task based on multimodal input data. A unimodal student ANN is then trained using at least one loss function, whose arguments include features from a latent space of the teacher ANN. In this way, a novel approach for knowledge distillation is implemented.

[0009] According to an aspect of the invention, a computer-implemented training method for training a student artificial neural network, ANN, to carry out a computer vision task is provided. Therein, a trained teacher ANN is provided, for example stored on a storage device. The teacher ANN is trained, in particular pre-trained, to carry out the computer vision task based on multimodal input data. For a training cycle of the student ANN, in particular for each training cycle of a plurality of consecutive training cycles to train the student ANN, at least the following steps i) to vi) are carried out. i) A first set of features is generated by applying a first feature extraction module, in particular a first encoder module, of the teacher ANN to a first training dataset, which corresponds to a first type of environmental sensor system. ii) A second set of features is generated by applying a second feature extraction module, for example a second encoder module, of the teacher ANN to a second training dataset, which corresponds to a second type of environmental sensor system, which is, in particular, different from the first type of environmental sensor system. iii) A set of fused features is generated by fusing the first set of features with the second set of features. iv) A third set of features is generated by applying a feature extraction module of the student ANN, for example a encoder module of the student ANN, to the first training dataset. v) Prediction data according to the computer vision task is generated by applying a decoder module of the student ANN to the third set of features. vi) Network parameters, for example weighting factors and / or bias factors, of the student ANN are updated depending on a result of an evaluation of at least one loss function. Therein, arguments of the at least one loss function, which may also be denoted as independent variables of the at least one loss function, include the prediction data, ground truth data for the first training dataset, the set of fused features and the third set of features.

[0010] Unless stated otherwise, all steps of the computer-implemented training method may be performed by a data processing apparatus, which comprises at least one computing unit. In particular, the at least one computing unit is configured or adapted to perform the steps of the computer-implemented training method. For this purpose, the at least one computing unit may for example store a computer program comprising instructions which, when executed by the at least one computing unit, cause the at least one computing unit to execute the computer-implemented training method.

[0011] The student ANN and the teacher ANN are denoted as such, since the teacher ANN effectively transfers knowledge to the student ANN. This technique may generally also be denoted as knowledge distillation. The teacher ANN may also be denoted as multimodal ANN and the student ANN may also be denoted as unimodal ANN.

[0012] The training of the teacher ANN is not necessarily a part of the computer-implemented training method according to the invention. The teacher ANN may be trained by means of a conventional method for training an ANN for the corresponding computer vision task based on multimodal input data. Here and in the following, "unimodal" refers to the use of input data generated only by a single type of environmental sensor system or, in other words, by only a single sensor modality. Analogously, "multimodal" refers to the use of input data generated by a two or more types of environmental sensor system or, in other words, by two or more different sensor modalities.

[0013] A computer vision task, also denoted as visual perception task, may for example be understood as a task for extracting visual information from image data. In particular, the visual perception task may in several cases be performed by a human in principle, who is able to visually perceive an image corresponding to the image data. In the present context, however, visual perception tasks are performed automatically without requiring the support by a human. For example, an ANN for carrying out a computer vision task may be understood as an image processing algorithm or an algorithm for image analysis. For example, the computer vision task may include an object detection task, a classification task, a segmentation task, and / or a depth estimation task. Corresponding algorithms may analogously be performed based on input data other than images visually perceivable by a human. For example, point clouds or thermal images of thermal cameras et cetera may also be analyzed by means of corresponding computer algorithms. Here and in the following, the expression computer vision task also includes such tasks.

[0014] The form of the prediction data according to the computer vision task depends on the specific underlying computer vision task. For example, prediction data of an object detection task may include one or more bounding boxes defining a spatial location and, optionally, orientation of one or more respective objects in the environment and / or corresponding object classes for the one or more objects. Prediction data of a semantic segmentation algorithm may include a pixel level class for each pixel of an image, and so forth.

[0015] A variety of ANNs, which are able to carry out such computer vision tasks is known and corresponding architectures can be used for the student ANN and the teacher ANN in the present context.

[0016] For fusing of the first set of features with the second set of features, known methods for feature fusion may be used, which include for example averaging, concatenation or summation. However, also more complex methods for feature fusion may be used. Also for updating the network parameters of the student ANN, known methods may be used, such as for example backpropagation.

[0017] An environmental sensor system can be understood as a sensor system, which is able to generate sensor data or sensor signals, which depict or represent an environment of the environmental sensor system. In particular, the ability to capture or detect electromagnetic or other signals from the environment, cannot be considered a sufficient condition for qualifying a sensor system as an environmental sensor system. For example, cameras, lidar systems, radar systems or ultrasonic sensor systems may be considered as environmental sensor systems.

[0018] The first training dataset and the second training dataset may for example be provided in form of respective pixel values for each pixel of two-dimensional arrangements of pixels. In case of cameras as environmental sensor systems, for example visible range cameras or thermal cameras, the training datasets correspond to images. However, also the data generated by other types of environmental sensor systems, for example lidar sensor systems or ultrasonic sensor systems, may be provided in or transformed into the format of two-dimensional arrangement of pixels, which can be treated as images. It is noted that a training dataset, in particular an image, may also comprise two or more such two- dimensional arrangements of pixels, for example according to different color channels. The first training dataset and the second training dataset, in particular, represent or depict the same scene or partially the same scene.

[0019] In known approaches for knowledge distillation, the student network is forced to learn to predict the same prediction data as the teacher network. In the computer-implemented training method according to the present invention, however, the student ANN is forced to learn the relevant feature representations in latent space from the teacher ANN. This helps the unimodal ANN to create a latent feature space, which is close to the latent space a multimodal network.

[0020] By means of the invention, it is achieved to overcome the drawbacks of a unimodal ANN at least partially without using a multimodal ANN once the training phase is completed. In particular, the performance of the unimodal ANN trained according to the invention may almost reach the performance of a multimodal network. Consequently, the reliability and robustness of functions, in particular for guiding a motor vehicle at least in part automatically, using the trained student ANN is improved under various environmental conditions, for example during day and nighttime. The trained student ANN can be deployed on target devices with low computational resources without any extra overhead as the proposed knowledge distillation in the latent space does not increase memory requirements. The idea can also be extended to teacher ANNs using more than two different sensor modalities. According to several implementations, a set of mixed features is computed by multiplying the set of fused features with the third set of features and the arguments of the at least one loss function include the set of mixed features.

[0021] According to several implementations, the at least one loss function comprises a first loss function, whose arguments include the set of mixed features and the third set of features.

[0022] For example, the first loss function is a cross-entropy loss function given by

[0023] Li = C.E(fm,g3) , wherein fmdenotes the set of mixed features, g3denotes the third set of features and C. E ,.) denotes a cross entropy function.

[0024] In this way, it is achieved that the feature extraction module of the student ANN learns to generate the third set of features taking into account also the information according to the second type of environmental sensor system.

[0025] According to several implementations, the at least one loss function comprises a second loss function, which is a correlation function, whose arguments include the set of fused features and the third set of features.

[0026] In this way, it is achieved that the feature extraction module of the student ANN learns to generate the third set of features taking into account also the information according to the second type of environmental sensor system.

[0027] In such implementations, in particular implementations using the first loss function and the second loss function, a mix and segregation strategy may be implemented to generate a mixture distribution for efficient learning.

[0028] According to several implementations, the at least one loss function comprises a third loss function, which is a similarity loss function depending on a similarity measure between the set of fused features and the third set of features.

[0029] In this way, the minimization process of the mix and segregation approach can be regulated, such that it can resolve problems of the rigidness for the optimization process in a way that the coefficients can be estimated smoothly for the hidden representation of the correlation for both distributions, that is multimodal fused distribution and the unimodal distribution.

[0030] According to several implementations, the similarity measure is a cosine similarity.

[0031] According to several implementations, the at least one loss function comprises at least one further loss function, whose arguments include the prediction data and the ground truth data for the first training dataset.

[0032] The at least one further loss function may for example be selected depending on the specific computer vision task. In particular, loss functions, which are known for training an ANN to carry out the computer vision task may be used. For example, the same or similar loss functions as used for training in the teacher ANN can be used.

[0033] According to several implementations, the computer vision task comprises an object detection task and the prediction data comprises a predicted bounding box for an object depicted in the first training dataset and the ground truth data comprises a ground truth bounding box for the object.

[0034] Consequently, the at least one further loss function may take the bounding box and the ground truth bounding box as arguments in order to achieve a training specifically adapted for object detection.

[0035] According to several implementations, the at least one further loss function comprises a fourth loss function, which depends on a Jaccard index of the predicted bounding box and the ground truth bounding box.

[0036] The Jaccard index is also denoted as intersection over union, loll. The fourth loss function may for example be given by a logarithm of the Jaccard index. In this way, the prediction of bounding boxes with less overlap with the ground truth bounding box may be penalized.

[0037] According to several implementations, the prediction data comprises a predicted score map for the bounding box and the ground truth data comprises a ground truth score map for the ground truth bounding box. The score map can be considered as a segmentation map of the respective bounding box or ground truth bounding box estimated at pixel level. In this way, the bounding boxes may be estimated more precisely.

[0038] According to several implementations, the at least one further loss function comprises a fifth loss function, which depends on or is given by a binary cross entropy of the predicted score map and the ground truth score map.

[0039] According to several implementations, the evaluation of the at least one loss function comprises evaluating a weighted sum of all loss functions of the at least one loss function, in particular a weighted sum of the first loss function, the second loss function, the third loss function, the fourth loss function and the fifth loss function.

[0040] In this way, the task specific learning for the computer vision task and the knowledge distillation may be commonly achieved in an end-to-end fashion.

[0041] According to several implementations, the first training dataset is a visible range camera image and the second training dataset is a thermal camera image, also denoted as thermal image.

[0042] In other words, the first type of environmental sensor system is a visible range camera, also denoted as RGB camera, and the second type of environmental sensor system is a thermal camera.

[0043] The thermal camera may also be denoted as thermographic camera or infrared camera. In particular, the thermal camera contains an infrared detector or infrared-sensitive imager, which is sensitive to infrared radiation, which may also be denoted as infrared light. For example, the infrared-sensitive imager maybe sensitive to wavelengths in the range of 750 nm to 15 pm or in a subrange of this range.

[0044] Consequently, after the training, the student ANN can carry out the computer vision task based on visible range camera images only, but still may achieve a performance similar to a multimodal ANN, which uses thermal camera images as well as visible range camera images as input data.

[0045] According to several implementations, the first training dataset is a thermal camera image and the second training dataset is a visible range camera image. In other words, the second type of environmental sensor system is a visible range camera and the first type of environmental sensor system is a thermal camera.

[0046] Consequently, after the training, the student ANN can carry out the computer vision task based on thermal camera images only, but still may achieve a performance similar to a multimodal ANN, which uses thermal camera images as well as visible range camera images as input data.

[0047] According to a further aspect of the invention, a computer-implemented method for carrying out a computer vision task is provided. Therein, a student ANN is trained by using a computer-implemented training method according to the invention. An input dataset is received from an environmental sensor system of the first type of environmental sensor system, in particular of a motor vehicle, and the computer vision task is carried out by applying the trained student ANN to the input dataset.

[0048] Unless stated otherwise, all steps of the computer-implemented method apart from the steps to carry out the computer-implemented training method may be performed by a further data processing apparatus, which comprises at least one further computing unit, in particular a further data processing apparatus of the motor vehicle. In particular, the at least one further computing unit is configured or adapted to perform the additional steps of the computer-implemented method. For this purpose, the at least one further computing unit may for example store the trained student ANN.

[0049] All further computing units of the at least one further computing unit may be comprised by the motor vehicle. However, it is also possible that all further computing units of the at least one further computing unit are part of an external computing system external to the motor vehicle, for example a backend server or a cloud computing system. It is also possible that the at least one further computing unit comprises at least one vehicle computing unit of the motor vehicle as well as at least one external computing unit comprised by the external computing system. The at least one vehicle computing unit may for example be comprised by one or more electronic control units, ECUs, and / or one or more zone control units, ZCUs, and / or one or more domain control units, DCUs, of the motor vehicle and / or by the environmental sensor system of the motor vehcile.

[0050] According to a further aspect of the invention, a method for guiding a motor vehicle at least in part automatically is provided. Therein, a computer-implemented method according to the invention is carried out. At least one control signal for guiding the motor vehicle at least in part automatically is generated depending on a result of the computer vision task and / or assistance information for assisting a driver of the motor vehicle at guiding the motor vehicle is generated depending on the result of the computer vision task.

[0051] The at least one control signal may for example be provided to one or more actuators of the motor vehicle, including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the motor vehicle. The one or more actuators may affect a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least in part automatically.

[0052] The assistance information may be output by means of an output device of the motor vehicle, for example a display and / or an audio output system and / or a haptic output system.

[0053] According to a further aspect of the invention, a data processing apparatus comprising at least one computing unit is provided. The at least one computing unit is adapted to carry out a computer-implemented training method according to the invention.

[0054] According to a further aspect of the invention, a further data processing apparatus comprising at least one further computing unit is provided. The at least one further computing unit is adapted to carry out a computer-implemented method according to the invention.

[0055] A computing unit may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0056] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units. In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0057] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a nonvolatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0058] According to a further aspect of the invention, an electronic vehicle guidance system for a motor vehicle is provided. The electronic vehicle guidance system comprises a data storage device, which stores a student ANN trained by using a computer-implemented training method according to the invention. The electronic vehicle guidance system comprises at least one computing unit, which is configured to receive an input dataset from an environmental sensor system of the first type of environmental sensor system, in particular of the motor vehicle, and to carry out the computer vision task by applying the trained student ANN to the input dataset. The at least one computing unit is configured to generate at least one control signal for guiding the motor vehicle at least in part automatically depending on a result of the computer vision task and / or to generate assistance information for assisting a driver of the motor vehicle at guiding the motor vehicle depending on the result of the computer vision task.

[0059] An electronic vehicle guidance system may be understood as an electronic system, configured to guide a vehicle in a fully automated or a fully autonomous manner and, in particular, without a manual intervention or control by a driver or user of the vehicle being necessary. The vehicle carries out all required functions, such as steering maneuvers, deceleration maneuvers and / or acceleration maneuvers as well as monitoring and recording the road traffic and corresponding reactions automatically. In particular, the electronic vehicle guidance system may implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system may also be implemented as an advanced driver assistance system, ADAS, assisting a driver for partially automatic or partially autonomous driving. In particular, the electronic vehicle guidance system may implement a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated April 2021 .

[0060] Guiding the vehicle at least in part automatically may therefore comprise guiding the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. Guiding the vehicle at least in part automatically may also comprise guiding the vehicle according to a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

[0061] According to several implementations, the electronic vehicle guidance system comprises the environmental sensor system.

[0062] According to several implementations, the environmental sensor system is a visible range camera or a thermal camera.

[0063] According to a further aspect of the invention, a computer program comprising instructions is provided. When the instructions are executed by a data processing apparatus, in particular by a data processing apparatus according to the invention, the instructions cause the data processing apparatus to carry out a computer-implemented training method according to the invention.

[0064] The instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python.

[0065] According to a further aspect of the invention, a further computer program comprising further instructions is provided. When the further instructions are executed by a further data processing apparatus, in particular by a further data processing apparatus according to the invention, the further instructions cause the further data processing apparatus to carry out a computer-implemented method according to the invention or a method according to the invention apart from the steps for carrying out the computer-implemented training method.

[0066] The further instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python. According to a further aspect of the invention, a computer-readable storage medium storing a computer program and / or a further computer program according to the invention is provided.

[0067] The computer program, the further computer program and the computer-readable storage medium are respective computer program products with the instructions and / or the further instructions.

[0068] Further features of the invention are apparent from the claims, the figures and the figure description. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of figures and / or shown in the figures may be comprised by the invention not only in the respective combination stated, but also in other combinations. In particular, embodiments and combinations of features, which do not have all the features of an originally formulated claim, may also be comprised by the invention. Moreover, embodiments and combinations of features, which go beyond or deviate from the combinations of features set forth in the recitations of the claims may be comprised by the invention.

[0069] In the following, the invention will be explained in detail with reference to specific exemplary implementations and respective schematic drawings. In the drawings, identical or functionally identical elements may be denoted by the same reference signs. The description of identical or functionally identical elements is not necessarily repeated with respect to different figures.

[0070] In the Figures,

[0071] Fig. 1 shows schematically a motor vehicle with an exemplary implementation of an electronic vehicle guidance system according to the invention;

[0072] Fig. 2 shows schematically a student ANN and a teacher ANN according to an exemplary implementation of a computer-implemented training method according to the invention; and

[0073] Fig. 3 shows schematically a student ANN and a teacher ANN according to a further exemplary implementation of a computer-implemented training method according to the invention. Fig. 1 shows schematically a motor vehicle 1 with an exemplary implementation of an electronic vehicle guidance system 2 according to the invention. The electronic vehicle guidance system 2 comprises at least one computing unit 4. The electronic vehicle guidance system 2, for example the at least one computing unit 4, comprises a storage device, which stores a student ANN, which has been trained by using a computer- implemented training method according to the invention.

[0074] The motor vehicle 1 , for example the electronic vehicle guidance system 2, comprises an environmental sensor system 3 of a first type, for example a visual range camera or a thermal camera. The environmental sensor system 3 is configured to generate an input dataset, for example a visible range camera image or a thermal camera image, respectively, and to provide it to the at least one computing unit 4.

[0075] The at least one computing unit 4 carries out a computer vision task, for example an object detection task, by applying the trained student ANN 5 to the input dataset. The at least one computing unit 4 generates at least one control signal for guiding the motor vehicle 1 at least in part automatically depending on a result of the computer vision task and / or generates assistance information for assisting a driver of the motor vehicle 1 at guiding the motor vehicle 1 depending on the result of the computer vision task.

[0076] Fig. 2 shows schematically the student ANN 5 and a teacher ANN 6 according to an exemplary implementation of a computer-implemented training method according to the invention. The teacher ANN 6 is trained to carry out the computer vision task based on multimodal input data. For training the student ANN 5, a plurality of training cycles, also denoted as training epochs, are carried out. For example, the student ANN 5 may be trained over -10.000 training cycles. For example, a momentum optimizer with a momentum of 0.9 may be used. The learning rate is for example 0.01 and may be decreased by a factor of 10 over each 3000 epochs.

[0077] For each training cycle, a first set of features 18 is generated by applying a first feature extraction module 9 of the teacher ANN 6 to a first training dataset 7, which corresponds to the first type of environmental sensor system 3. For example, if the environmental sensor system 3 of the motor vehicle 1 is a visible range camera, then the first training dataset 7 is a visible range camera image and if the environmental sensor system 3 of the motor vehicle 1 is a thermal camera, then the first training dataset 7 is a thermal image. A second set of features 19 is generated by applying a second feature extraction module 10 of the teacher ANN 6 to a second training dataset 8, which corresponds to a second type of environmental sensor system. For example, if the environmental sensor system 3 of the motor vehicle 1 is a visible range camera, then the second training dataset 8 may be a thermal image, and if the environmental sensor system 3 of the motor vehicle 1 is a thermal camera, then the second training dataset 8 may be a visible range camera image.

[0078] A set of fused features 20 is generated by fusing the first set of features 18 with the second set of features 19, in particular by a fusion module 11 of the teacher ANN 6. A third set of features 21 is generated by applying a feature extraction module 14 of the student ANN 5 to the first training dataset 7. Prediction data 16 according to the computer vision task is generated by applying a decoder module 15 of the student ANN 5 to the third set of features 21 . Network parameters of the student ANN 5 are updated depending on a result of an evaluation of at least one loss function 17. Arguments of the at least one loss function 17 include the prediction data 16, ground truth data 22 for the first training dataset 7, the set of fused features 20 and the third set of features 21 .

[0079] The fusion module 11 may for example implement a fusion technique to reduce the domain gap between the different sensor modalities. Inside the fusion module 11 , the first set of features 18 and the second set of features 19 may first be passed through to a subtraction operation and a normalization process. The result may then be passed through a global average pooling layer followed by a sigmoidal function to construct a modality agnostic feature vector. To accomplish a blending operation of the multimodal features efficiently, each of the elements of the modality agnostic feature vector may be multiplied with the channel of the both modality dependent features.

[0080] As shown in Fig. 2, the teacher ANN 6 comprises one or more further modules 12, which can be applied to the set of fused features 20 to carry out the computer vision task and to generate respective further prediction data 13. However, once the training of the teacher ANN 6 is completed, the one or more further modules 12 or the further prediction data 13 are not required for training the student ANN 5.

[0081] Fig. 2 may be considered as an end-to-end pipeline to train the unimodal student ANN 5 with the guidance of the multimodal teacher ANN 6. Further details are explained in the following for the exemplary computer vision task object detection. Furthermore, it is assumed, without loss of generality, that the environmental sensor system 3 of the first type is a visual range camera and the environmental sensor system of the second type is a thermal camera.

[0082] In particular, for training the student ANN 5, two strategies may be employed, namely a mix and segregation strategy and a correlation learning strategy. By means of mix and segregation, a mixture distribution is created. First, the correlation between the set of fused features 20 and the third set of features 21 is computed to estimate the gap between the distributions, in particular to estimate whether features are positively correlated or negatively correlated. To this end, a cross-correlation function may be used, whose output is a cross correlation matrix. A tangent function may be used for normalization. The result is a second loss function of the at least one loss function 17

[0083] ^2= / cC / fi's) > wherein f12denotes the fused set of features 20 and g3denotes the third set of features 21.

[0084] Furthermore, a set of mixed features 23 is computed by multiplying the set of fused features 20 with the third set of features 21 . For example, if the set of fused features 20 or the third set of features 21 are given by matrices or higher dimensional tensors, they may be flattened to vectors before multiplying them.

[0085] Then, a first loss function of the at least one loss function 17 may be computed, which is for example a cross-entropy loss function given by

[0086] Li = C.E(fm,g3), wherein fmdenotes the set of mixed features 23. The first loss function helps to learn the correlation of both modalities while only unimodal data fed to the student ANN 5. A correlation matrix may regulates the sensitivity of the entropy function by multiplying it with the cross entropy function. The feature extraction module 14 of the student ANN 5 identifies the identity of a thermal distribution based on the provided visible range camera image.

[0087] As a third loss function of the at least one loss function 17, a similarity loss function may be used, which comntains for example a cosine similarity with a temperature parameter T: z=_1qexp(stm(zfc,z;) / T) exp(sjm(zk,zm) / T) ’ wherein sim(.) denotes the cosine similarity, zkare patches of the set of fused features 20 and zi and zmare patches of the third set of features 21 . In order to obtain said patches, a window is moved over the corresponding feature matrix, for example with a window size of 5x5. Each window represents a patch. P is the number of patches.

[0088] The prediction data 16 comprises for example a bounding box for one or more objects in the first training dataset 7 and for example a score map for each of the bounding boxes. Then, a fourth loss function of the at least one loss function 17 may be given by wherein Badenotes the respective predicted bounding box, Bathe corresponding ground truth bounding box and loll() denotes the intersection over union. Furthermore, a fifth loss function of the at least one loss function 17 may be given by the cross entropy loss ) log(l - Sa) , wherein Sadenotes the respective predicted score map and Sathe corresponding ground truth score map and p is a class balancing weight factor between positive and negative input samples.

[0089] The final loss function for training the student ANN 5 is then for example

[0090] L = A-^L^ + A2L2+ A2L2+ 4L4 + A5L5, wherein A1, A2, A3, A4, and A5are predefined or learned hyperparameters for weighting the individual contributions.

[0091] It is noted that the fourth loss function L4and the fifth loss function L5may also be used for training the teacher ANN 6. As indicated in Fig. 3, the one or more further modules 12 of the teacher ANN 6 may for example comprise a spatio-contextual learning block 12a, which receives the fused set of features 20, and a detection head 12b, which receives the output of the spatio-contextual learning block 12a and outputs the further prediction data 13. The spatio-contextual learning block 12a may extract spatial and contextual features from the fused set of features 20 by using for example spatial attention and a 4DIRNN operation. The spatial and contextual features may be concatenated before fed to the detection head 12b.

[0092] Regarding details of the spatio-contextual learning block 12a, it is referred to the publication K. Dasgupta et al.: "Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving" (arXiv:2105.12713v3), in particular section C, and regarding the 4DIRNN it is referred to section 3.2 of the same publication.

[0093] As described, in particular with respect to the figures, the invention provides a possibility to overcome the limitations of a unimodal ANN due to the specific characteristics of the corresponding sensor modality.

[0094] In particular, a pipeline for correlation learning in latent space may be employed, by redesigning the strategy of knowledge distillation using a procedure, which makes the student ANN learn the correlation between the unimodal distribution and the fused distribution from the multimodal teacher ANN.

Claims

Claims1 . Computer-implemented training method for training a student artificial neural network, ANN, (5) to carry out a computer vision task, wherein a trained teacher ANN (6) is provided, which is trained to carry out the computer vision task based on multi-modal input data, wherein, for a training cycle of the student ANN (5) a first set of features (18) is generated by applying a first feature extraction module (9) of the teacher ANN (6) to a first training dataset (7), which corresponds to a first type of environmental sensor system (3); a second set of features (19) is generated by applying a second feature extraction module (10) of the teacher ANN (6) to a second training dataset (8), which corresponds to a second type of environmental sensor system; a set of fused features (20) is generated by fusing the first set of features (18) with the second set of features (19); a third set of features (21 ) is generated by applying a feature extraction module (14) of the student ANN (5) to the first training dataset (7); prediction data (16) according to the computer vision task is generated by applying a decoder module (15) of the student ANN (5) to the third set of features (21); network parameters of the student ANN (5) are updated depending on a result of an evaluation of at least one loss function (17); and arguments of the at least one loss function (17) include the prediction data (16), ground truth data (22) for the first training dataset (7), the set of fused features (20) and the third set of features (21).

2. Computer-implemented training method according to claim 2, wherein a set of mixed features (23) is computed by multiplying the set of fused features (20) with the third set of features (21) and the arguments of the at least one loss function (17) include the set of mixed features (23).

3. Computer-implemented training method according to claim 2, wherein the at least one loss function (17) comprises a first loss function, whose arguments include the set of mixed features (23) and the third set of features (21).

4. Computer-implemented training method according to claim 3, wherein the first loss function is a cross-entropy loss function given byLi = C.E(fm,g3), wherein fmdenotes the set of mixed features (23) and g3denotes the third set of features (21).

5. Computer-implemented training method according to one of the preceding claims, wherein the at least one loss function (17) comprises a second loss function, which is a correlation function, whose arguments include the set of fused features (20) and the third set of features (21).

6. Computer-implemented training method according to one of the preceding claims, wherein the at least one loss function (17) comprises a third loss function, which is a similarity loss function depending on a similarity measure between the set of fused features (20) and the third set of features (21).

7. Computer-implemented training method according to claim 6, wherein the similarity measure is a cosine similarity.

8. Computer-implemented method according to one of the preceding claims, wherein the at least one loss function (17) comprises at least one further loss function, whose arguments include the prediction data (16) and the ground truth data (22) for the first training dataset (7).

9. Computer-implemented training method according to one of the preceding claims, wherein the computer vision task comprises an object detection task and the prediction data (16) comprises a predicted bounding box for an object depicted in the first training dataset (7) and the ground truth data (22) comprises a ground truth bounding box for the object.

10. Computer-implemented training method according to claim 8 and claim 9, wherein the at least one further loss function comprises a fourth loss function, which depends on a Jaccard index of the predicted bounding box and the ground truth bounding box.11 . Computer-implemented training method according to one of claims 9 or 10, wherein the prediction data (16) comprises a predicted score map for the bounding box and the ground truth data (22) comprises a ground truth score map for the ground truth bounding box.

12. Computer-implemented training method according to claim 8 and claim 11 , wherein the at least one further loss function comprises a fifth loss function, which depends on a binary cross entropy of the predicted score map and the ground truth score map.

13. Computer-implemented training method according to one of the preceding claims, wherein the evaluation of the at least one loss function (17) comprises evaluating a weighted sum of all loss functions of the at least one loss function (17).

14. Computer-implemented training method according to one of the preceding claims, wherein the first training dataset (7) is a visible range camera image and the second training dataset (8) is a thermal camera image; or the first training dataset (7) is a thermal camera image and the second training dataset (8) is a visible range camera image.

15. Computer-implemented method for carrying out a computer vision task, wherein a student ANN (5) is trained by using a computer-implemented training method according to one of the preceding claims; and an input dataset is received from an environmental sensor system (3) of the first type and the computer vision task is carried out by applying the trained student ANN (5) to the input dataset.

16. Method for guiding a motor vehicle (1) at least in part automatically, wherein a computer-implemented method according to claim 15 is carried out and at least one control signal for guiding the motor vehicle (1) at least in part automatically is generated depending on a result of the computer vision task; and / or assistance information for assisting a driver of the motor vehicle (1 ) at guiding the motor vehicle (1 ) is generated depending on the result of the computer vision task.

17. Data processing apparatus comprising at least one computing unit, which is adapted to carry out a computer implemented training method according to one of claims 1 to 14 or a computer-implemented method according to claim 15.

18. Electronic vehicle guidance system (2) for a motor vehicle (1 ) comprising a data storage device, which stores a student ANN (5) trained by using a computer- implemented training method according to one of claims 1 to 14, and comprising at least one computing unit (4), which is configured to receive an input dataset from an environmental sensor system (3) of the first type, and to carry out the computer vision task by applying the trained student ANN (5) to the input dataset and to generate at least one control signal for guiding the motor vehicle (1) at least in part automatically depending on a result of the computer vision task; and / or assistance information for assisting a driver of the motor vehicle (1 ) at guiding the motor vehicle (1 ) depending on the result of the computer vision task.

19. Electronic vehicle guidance system (2) according to claim 18, which comprises the environmental sensor system (3).

20. Electronic vehicle guidance system (2) according to claim 19, wherein the environmental sensor system (3) is a visible range camera or a thermal camera.21 . Computer program product comprising instructions, which, when carried out by a data processing apparatus, cause the data processing apparatus to carry out a computer implemented training method according to one of claims 1 to 14 or a computer implemented method according to claim 15 or a method according to claim 16.

Images