Tracking an emergency vehicle

Abstract

For tracking an emergency vehicle, a thermal image (8) depicting a target vehicle is received from a thermal camera (3) for each frame of a sequence of frames. An object detection algorithm (15) is carried out based on the thermal images (8), wherein an output comprises vehicle bounding boxes for the target vehicle and respective light bounding boxes for an active light source. If it is determined that the light source is alternatingly active and inactive over the frames, the target vehicle is classified as an emergency vehicle and a sequence of refined bounding boxes (10) for the emergency vehicle is determined by tracking the emergency vehicle, wherein the tracking comprises applying a first ANN (32) to first input data (20) depending on the bounding boxes.

Description

[0001] Tracking an emergency vehicle

[0002] The present invention is directed to a computer implemented method for tracking an emergency vehicle in an environment of an ego-vehicle. The invention is further directed to a method for guiding an ego-vehicle at least in part automatically, wherein such a computer implemented method is carried out, to a corresponding data processing apparatus, to an electronic vehicle guidance system comprising such a data processing apparatus, and to a computer program product.

[0003] In the context of driver assistance functions or other functions for guiding an ego-vehicle at least in part automatically, the reliable automatic tracking of emergency vehicles, for example police vehicles or ambulance vehicles, may be important for safety reasons and also for unblocking a path for the emergency vehicle.

[0004] Document US 2021 / 003414914 A1 describes a method for detecting an emergency vehicle, wherein a plurality of images taken from a perspective of an autonomous vehicle is received. One or more gates are generated in the images, each gate corresponding to a region of interest at a respective distance from the vehicle. Then, lights within the gates are detected and candidates for an emergency vehicle are identified based on the detected lights.

[0005] YoloV5 (doi:10.5281 / zenodo.7347926) is a widely used computer-vision algorithm for object detection based on convolutional neural networks, CNNs.

[0006] Known object tracking algorithms may have poor reliability in scenarios with poor lighting, in particular at night time, or with adverse weather conditions.

[0007] It is an objective of the present invention to detect emergency vehicles with increased reliability, in particular for poor lighting conditions.

[0008] This objective is achieved by the subject matter of the independent claim. Further implementations and preferred embodiments are subject matter of the dependent claims. The invention is based on the idea to capture respective thermal images for a sequence of frames and to use an object detection algorithm to identify a bounding box of a target vehicle in the thermal images as well as a bounding box for an active light source of the target vehicle. The target vehicle is classified as an emergency vehicle, if the light source is found to be alternatingly active and inactive. An artificial neural network is used to track the emergency vehicle and to generate a sequence of refined bounding boxes for the emergency vehicle.

[0009] According to an aspect of the invention, a computer implemented method for tracking an emergency vehicle in an environment of an ego-vehicle is provided. Therein, for each frame of a sequence of frames, a respective thermal image depicting a target vehicle is received from a thermal camera mounted to the ego-vehicle. An object detection algorithm is carried out based on the thermal images received for the sequence of frames, wherein an output of the object detection algorithm comprises respective vehicle bounding boxes for the target vehicle and respective light bounding boxes for an active light source of the target vehicle depicted by the thermal images. It is determined, based on the light bounding boxes, whether the light source is alternatingly active and inactive over the sequence of frames. If it is determined that the light source is alternatingly active and inactive over the sequence of frames, the target vehicle is classified as an emergency vehicle and a sequence of refined bounding boxes for the emergency vehicle is determined by tracking the emergency vehicle, wherein the tracking comprises applying a trained first artificial neural network, ANN, to first input data. The first input data depends on, in particular comprises, the vehicle bounding boxes and the light bounding boxes.

[0010] Unless stated otherwise, all steps of the computer implemented method may be performed by a data processing apparatus, which comprises at least one computing unit, in particular a data processing apparatus of the ego-vehicle. In particular, the at least one computing unit is configured or adapted to perform the steps of the computer implemented method. For this purpose, the at least one computing unit may for example store a computer program comprising instructions which, when executed by the at least one computing unit, cause the at least one computing unit to execute the computer implemented method.

[0011] All computing units of the at least one computing unit may be comprised by the egovehicle. However, it is also possible that all computing units of the at least one computing unit are part of an external computing system external to the ego-vehicle, for example a backend server or a cloud computing system. It is also possible that the at least one computing unit comprises at least one vehicle computing unit of the ego-vehicle as well as at least one external computing unit comprised by the external computing system. The at least one vehicle computing unit may for example be comprised by one or more electronic control units, ECUs, and / or one or more zone control units, ZCUs, and / or one or more domain control units, DCUs, of the vehicle and / or by the thermal camera.

[0012] For any implementation of the computer implemented method, corresponding implementations of a method, which is not purely computer implemented, are obtained by including method steps of generating the thermal images by the thermal camera.

[0013] The vehicle bounding boxes may for example be temporally ordered according to the sequence of frames and the light bounding boxes may also be temporally ordered according to the sequence of frames. In particular, it is determined based on the temporally ordered light bounding boxes, whether the light source is alternating ly active and inactive over the sequence of frames and the first input data comprises the temporally ordered vehicle bounding boxes and the temporally ordered light bounding boxes.

[0014] The thermal camera may also be denoted as thermal imaging camera or thermographic camera or infrared camera. In particular, the thermal camera contains an infrared detector or infrared-sensitive imager, which is sensitive to infrared radiation, which may also be denoted as infrared light. For example, the infrared-sensitive imager maybe sensitive to wavelengths in the range of 750 nm to 15 pm or in a subrange of this range.

[0015] The object detection algorithm may be understood as a computer algorithm, which is able to identify and localize one or more objects within each of the thermal images by specifying respective bounding boxes and, in particular, assigning a respective object class to each of the bounding boxes, wherein the object classes may be selected from a predefined set of object classes. Therein, assigning an object class to a bounding box may be understood such that a corresponding confidence value for the object identified within the bounding box being of the corresponding object class is provided. In the present example, the object classes comprise at least a class for vehicles, such as the target vehicle, and a class for active light sources, such as the active light source of the target vehicle. Since the light source of the target vehicle may be alternating ly active and inactive, respectively, the active light source of the target vehicle may be detected in some of the frames and may not be detected in other frames. In other words, the output of the object detection algorithm comprises, for example, a corresponding vehicle bounding box for the target vehicle for each frame and a corresponding light bounding box for the active light source of the target vehicle for a subset of the frames.

[0016] In the output of the object detection algorithm, each vehicle bounding box and each light bounding box may for example be defined by a corresponding position of the respective bounding box and a size and / or shape of the respective bounding box.

[0017] In addition, the output of the object detection algorithm may comprise bounding boxes for other objects including other vehicles and / or other active light sources. In the context of the presented computer implemented method, bounding boxes for objects that are neither vehicles nor active light sources are neglected and may for example be filtered out. For temporally ordering the vehicle bounding boxes, all bounding boxes corresponding to the same target vehicle are ordered such that a sequence results, which corresponds to the sequence of frames. Analogously, also the light bounding boxes may be ordered temporally.

[0018] The temporally ordered light bounding boxes can be analyzed to find out whether the active light source of the target vehicle does periodically or alternatively disappear and reappear over the sequence of frames in order to determine that the target vehicle is an emergency vehicle. In other words, it is assumed that a vehicle with a light source that is alternating ly active and inactive over the sequence of frames is an emergency vehicle.

[0019] By tracking the emergency vehicle over the sequence of frames, for example using a recurrent neural network, RNN, in particular a long short-term memory, LSTM, as the first ANN, the sequence of refined bounding boxes is determined. In other words, the sequence of refined bounding boxes may be considered as a result of the computer implemented method for tracking the emergency vehicle.

[0020] By means of the computer implemented method according to the invention, a reliable detection and tracking of the emergency vehicle is possible. By using the thermal images as a basis for the tracking, the method is robust and reliable also for adverse weather conditions and / or low lighting scenarios. Compared with approaches based on visible light imaging, imaging in the infrared or thermal regime, respectively, may achieve an improved performance and accuracy. In contrast to camera images captured by means of light in the visible range, for example RGB images, emergency vehicles cannot be detected as such based on the color of the emergency lights when using thermal images. This is presently overcome by identifying whether the light source of the target vehicle is alternating ly active and inactive.

[0021] According to several implementations, respective audio datasets for the sequence of frames are received from at least two microphones, which are mounted at different positions on the ego-vehicle. A source region in the environment of the ego-vehicle, in which an audio source emitting a siren sound is located, is determined by applying a trained second ANN to second input data depending on the audio datasets. The first input data comprises the source region.

[0022] The two or more microphones are, in particular, mounted at the ego-vehicle to detect sounds from the outer environment of the ego-vehicle.

[0023] By taking into account the audio datasets in addition to the thermal images a multimodal, in particular bi-modal, approach for the tracking of the emergency vehicle is implemented, which further increases the reliability and robustness of the method.

[0024] For any implementation of the computer implemented method, corresponding implementations of a method, which is not purely computer implemented, are obtained by including method steps of generating the audio datasets by the at least two microphones.

[0025] According to several implementations, a plurality of Mel-frequency cepstral coefficients, MFCC, is computed for each of the audio datasets. The audio datasets are filtered depending on the respective Mel-frequency cepstral coefficients to separate the audio source emitting the siren sound from one or more further audio sources and / or background noise. In this way, the reliability and robustness of the method is further increased.

[0026] The audio datasets can be understood as the respective amplitude or intensity of sound waves captured by the respective microphone as a function of time. In particular, the audio datasets may be given by respective time series of the respective amplitude or intensity. According to several implementations, a zero-crossing rate, ZCR, is computed for each of the audio datasets and the filtering is carried out depending on the respective zerocrossing rate. In this way, the reliability and robustness of the method is further increased.

[0027] According to several implementations, a respective spectrogram is generated based on each of the audio datasets, in particular depending on the filtered audio datasets. A sound event is detected by using a classification algorithm and a direction of arrival, DOA, of the sound event is determined using a regression algorithm depending on the audio datasets.

[0028] Each of the spectrograms is obtained from one of the audio datasets. The spectrogram represents the frequency spectrum of the respective audio dataset as a function of time. It is therefore a two-dimensional array, wherein different lines of the array correspond to different frequencies and different columns of the array correspond to different time instances or vice versa. The values of the array are the corresponding amplitudes in the frequency domain. In particular, the spectrogram is generated by applying a short-time Fourier transform, STFT, to the time series of the respective audio dataset or to a pre- processed version of the time series of the respective audio dataset. The pre-processing may for example include noise filtering. The result of the STFT may in some implementations also be post-processed to generate the spectrogram. The postprocessing may for example include scaling operations, for example to convert the frequencies into the Mel scale. The spectrograms are then Mel spectrograms accordingly.

[0029] Due to the 2D format of the spectrograms, they can be processed by means of a perception algorithm in a similar manner as a camera image, for example by applying convolutional layers and so forth for feature extraction, regression, classification et cetera.

[0030] According to several implementations, the second ANN is trained by minimizing a loss function, which is given by wherein ytdenotes the location of the respective ground truth bounding boxes and ytis the location of the source region predicted by the second ANN. According to at least one implementation, the output of the object detection algorithm comprises a respective detection confidence value for each of the light bounding boxes. The tracking of the emergency vehicle comprises determining a tracking status of the light bounding boxes for each of the sequence of frames depending on the detection confidence value of the light bounding box for the respective frame and a Jaccard index, also denoted as intersection over union, loll, of the respective light bounding box with the light bounding box of the respective subsequent frame of the sequence of frames.

[0031] For example, the tracking status is determined as a disappeared status if the detection confidence value of the respective light bounding box is smaller than a predefined minimum confidence value. Each time the tracking status is determined as the disappeared status, a counter is incremented and a tracking of the light bounding box is cancelled only if, in particular if and only if, a value of the counter is greater than a predefined limit value.

[0032] In this way, it is avoided that the tracking of the light bounding box is cancelled if the disappearing complies with the expected flashing nature of an emergency vehicle light.

[0033] In particular, the limit value is determined depending on a predefined range for an emergency flash rate of the emergency vehicle.

[0034] According to several implementations, the first ANN is an RNN, in particular an LSTM network. In this way, a reliable and robust tracking of the bounding boxes may be achieved.

[0035] According to several implementations, the thermal images are preprocessed, wherein the preprocessing comprises applying at least one spatial filter for image sharpening to the respective thermal image and the object detection algorithm is carried out based on the preprocessed thermal images.

[0036] A detection target in a thermal image may have a low contrast and / or a low signal-to-noise ratio. The target may occupy for example only a few tens of pixels, in particular if it is far away from the thermal camera. Thus, the preprocessing improves the detection performance in complex backgrounds, in particular by increasing the degree of separation between a small target object and the background and enhancing the characteristics of the target object.

[0037] According to several implementations, the preprocessing comprises upsampling the respective thermal image after applying the at least one spatial filter for image sharpening.

[0038] Therefore, the number of pixels representing the target object is increased. In particular, interpolation may be used to increase the target pixels.

[0039] According to several implementations, carrying out the object detection algorithm comprises applying a trained third ANN, which comprises a convolutional neural network, CNN, to third input data depending on the thermal images, in particular depending on, for example comprising, the preprocessed thermal images.

[0040] According to several implementations, the output of the object detection algorithm comprises respective vehicle bounding boxes for at least one further target vehicle depicted by the thermal images. For each frame of the sequence of frames, a priority value is assigned to each of the vehicle bounding boxes for the respective frame depending on a position of the respective vehicle bounding box within the thermal image of the respective frame. The first input data comprises the priority values of the vehicle bounding boxes.

[0041] According to several implementations, the output of the object detection algorithm comprises respective light bounding boxes for a respective active light source of the at least one further target vehicle depicted by the thermal images. For each frame of the sequence of frames, a priority value is assigned to each of the light bounding boxes for the respective frame depending on a position of the respective light bounding box within the thermal image of the respective frame. The first input data comprises the priority values of the light bounding boxes.

[0042] According to several implementations, the first ANN is trained by minimizing at least one loss function comprising a first loss function, which is given by wherein Xtdenotes a bounding box predicted during the training, Xtdenotes a corresponding ground truth for the bounding box, a and ft are predefined weighting factors and loll denotes the intersection over union.

[0043] In other words, the first loss function combines the Jaccard index based loss with a position based mean square error loss to train the first ANN, in particular the LSTM.

[0044] According to several implementations, the at least one loss function comprises a second loss function, which is given by the cross entropy loss function wherein ycis the ground truth and ycthe predicted probability of which light source c the target vehicle is associated.

[0045] Thus, the first ANN is trained to associate the predicted light bounding box with their respective vehicle bounding box using a classification loss function.

[0046] For use cases or use situations which may arise in a method according to the invention and which are not explicitly described herein, it may be provided that, in accordance with the method, an error message and / or a prompt for user feedback is output and / or a default setting and / or a predetermined initial state is set.

[0047] According to a further aspect of the invention, a method for guiding an ego-vehicle at least in part automatically is provided, wherein a computer implemented method according to the invention is carried out. At least one control signal for guiding the ego-vehicle at least in part automatically is generated by at least one computing unit, for example at least one computing unit of the ego-vehicle, depending on the sequence of refined bounding boxes, and / or assistance information for assisting a driver of the ego-vehicle at guiding the egovehicle is generated by the at least one computing unit depending on the sequence of refined bounding boxes. The at least one control signal may for example be provided to one or more actuators of the ego-vehicle, including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the ego-vehicle. The one or more actuators may affect a longitudinal and / or lateral control of the ego-vehicle in order to guide the ego-vehicle at least in part automatically.

[0048] The assistance information may be output by means of an output device of the egovehicle, for example a display and / or an audio output system and / or a haptic output system.

[0049] According to a further aspect of the invention, a data processing apparatus comprising at least one computing unit is provided. The at least one computing unit is adapted to carry out a computer implemented method according to the invention.

[0050] A computing unit may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0051] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0052] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0053] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a nonvolatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0054] According to a further aspect of the invention, an electronic vehicle guidance system for an ego-vehicle is provided. The electronic vehicle guidance system comprises a data processing apparatus according to the invention. The at least one computing unit is configured to generate at least one control signal for guiding the ego-vehicle at least in part automatically depending on the depending on the sequence of refined bounding boxes, and / or to generate assistance information for assisting a driver of the ego-vehicle at guiding the ego-vehicle depending on the sequence of refined bounding boxes.

[0055] An electronic vehicle guidance system may be understood as an electronic system, configured to guide a vehicle in a fully automated or a fully autonomous manner and, in particular, without a manual intervention or control by a driver or user of the vehicle being necessary. The vehicle carries out all required functions, such as steering maneuvers, deceleration maneuvers and / or acceleration maneuvers as well as monitoring and recording the road traffic and corresponding reactions automatically. In particular, the electronic vehicle guidance system may implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system may also be implemented as an advanced driver assistance system, ADAS, assisting a driver for partially automatic or partially autonomous driving. In particular, the electronic vehicle guidance system may implement a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated April 2021 .

[0056] Guiding the vehicle at least in part automatically may therefore comprise guiding the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. Guiding the vehicle at least in part automatically may also comprise guiding the vehicle according to a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

[0057] According to several implementations, the electronic vehicle guidance system comprises the thermal camera for the ego-vehicle and / or the two or more microphones. Further implementations of the electronic vehicle guidance according to the invention follow directly from the various embodiments of the computer implemented method according to the invention or the method for guiding the ego vehicle-according at least in part automatically according to the invention and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various implementations of the computer implemented method according to the invention can be transferred analogously to corresponding implementations of the electronic vehicle guidance according to the invention. In particular, the electronic vehicle guidance according to the invention is designed or programmed to carry out a computer implemented method or a method according to the invention. In particular, the electronic vehicle guidance according to the invention carries out the computer implemented method or the method according to the invention.

[0058] According to a further aspect of the invention, a computer program comprising instructions is provided. When the instructions are executed by a data processing apparatus, in particular by a data processing apparatus according to the invention, the instructions cause the data processing apparatus to carry out a computer implemented method according to the invention.

[0059] The instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python.

[0060] According to a further aspect of the invention, a further computer program comprising further instructions is provided. When the further instructions are executed by an electronic vehicle guidance system according to the invention, in particular by the data processing apparatus of the electronic vehicle guidance system, the further instructions cause the electronic vehicle guidance system to carry out a method for guiding an egovehicle at least in part automatically according to the invention.

[0061] The further instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python. According to a further aspect of the invention, a computer readable storage medium storing a computer program according to the invention and / or a further computer program according to the invention is provided.

[0062] The computer program, the further computer program and the computer-readable storage medium are respective computer program products with the instructions and / or the further instructions.

[0063] Further features of the invention are apparent from the claims, the figures and the figure description. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of figures and / or shown in the figures may be comprised by the invention not only in the respective combination stated, but also in other combinations. In particular, embodiments and combinations of features, which do not have all the features of an originally formulated claim, may also be comprised by the invention. Moreover, embodiments and combinations of features, which go beyond or deviate from the combinations of features set forth in the recitations of the claims may be comprised by the invention.

[0064] In the following, the invention will be explained in detail with reference to specific exemplary implementations and respective schematic drawings. In the drawings, identical or functionally identical elements may be denoted by the same reference signs. The description of identical or functionally identical elements is not necessarily repeated with respect to different figures.

[0065] In the figures,

[0066] Fig. 1 shows schematically an ego-vehicle with an exemplary implementation of an electronic vehicle guidance system according to the invention;

[0067] Fig. 2 shows a schematic block diagram of an exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention; Fig. 3 shows schematically the backbone structure of an object detection algorithm of a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention;

[0068] Fig. 4 shows schematically a CSP module of an object detection algorithm of a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention;

[0069] Fig. 5 shows schematically an attention module of an object detection algorithm of a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention;

[0070] Fig. 6 shows schematically bounding boxes generated by an object detection algorithm of a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention;

[0071] Fig. 7. shows schematically an audio source localization according to a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention;

[0072] Fig. 8 shows schematically an architecture of an audio localization block according to a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention; and

[0073] Fig. 9 shows schematically an LSTM according to a further exemplary implementation of a computer implemented method for tracking an emergency vehicle according to the invention.

[0074] In Fig. 1 an ego-vehicle 1 with an exemplary implementation of an electronic vehicle guidance system 2 according to the invention is shown. The ego-vehicle 1 , for example the electronic vehicle guidance system 2, comprises a thermal camera 3, which is configured to generate a sequence, also denoted as stream or videostream, of thermal images 8 depicting an outer environment of the ego-vehicle 1 corresponding to a sequence of frames. The electronic vehicle guidance system 2 comprises a data processing apparatus 4, which contains at least one computing unit. The at least one computing unit may for example comprise one or more electronic control units, ECUs, of the ego-vehicle 1 and / or one or more computing units of the thermal camera 3. The data processing apparatus 4 is adapted to carry out a computer implemented method for tracking an emergency vehicle according to the invention depending on the thermal images 8. A schematic block diagram of an algorithm 7 implementing an exemplary implementation of such a method is shown in Fig. 2. As a result of the computer implemented method, the data processing apparatus 4 generates a sequence of refined bounding boxes 10 for the emergency vehicle.

[0075] The at least one computing unit may generate at least one control signal for guiding the ego-vehicle 1 at least in part automatically depending on the sequence of refined bounding boxes 10. The at least one control signal may for example be provided to one or more actuators of the ego-vehicle 1 , including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the egovehicle 1 . The one or more actuators may affect a longitudinal and / or lateral control of the ego-vehicle 1 in order to guide the ego-vehicle 1 at least in part automatically.

[0076] Alternatively or in addition, the at least one computing unit may generate assistance information for assisting a driver of the ego-vehicle 1 at guiding the ego-vehicle 1 depending on the sequence of refined bounding boxes 10. The assistance information may be output by means of an output device of the ego-vehicle 1 , for example a display and / or an audio output system and / or a haptic output system.

[0077] For each frame of the sequence of frames, the respective thermal image 8 is received by the at least one computing unit from the thermal camera 3 and the at least one computing unit carries out an object detection algorithm 15 based on the thermal images 8, wherein an output of the object detection algorithm 15 comprises respective vehicle bounding boxes for a target vehicle depicted by the thermal images 8 and respective light bounding boxes for an active light source of the target vehicle. The at least one computing unit may for example order the vehicle bounding boxes and the light bounding boxes temporally according to the sequence of frames.

[0078] The at least one computing unit determines, based on the light bounding boxes, in particular the temporally ordered light bounding boxes, whether the light source is alternating ly active and inactive over the sequence of frames. If this is the case, the at least one computing unit classifies the target vehicle as an emergency vehicle and generates the sequence of refined bounding boxes 10 for the emergency vehicle by using a tracking algorithm. The tracking comprises applying a trained first ANN 32, in particular an LSTM, for example a double LSTM, also denoted as dLSTM, to first input data 20 depending on, in particular comprising, the vehicle bounding boxes and the light bounding boxes, in particular temporally ordered.

[0079] Optionally, the ego-vehicle 1 , for example the electronic vehicle guidance system 2, comprises two microphones 5, 6 mounted at different position on the ego-vehicle 1 , which are configured to generate respective time-dependent audio datasets 9 based on sound reaching the microphones 5, 6 from the outer environment of the ego-vehicle 1 . In such implementations, the at least one computing unit may generate the sequence of refined bounding boxes 10 based on the thermal images 8 and the audio datasets 9. In particular, the at least one computing unit determines a source region in the environment of the egovehicle 1 , in which an audio source emitting a siren sound is located, by applying a trained second ANN 18, 19 to second input data depending on the audio datasets 9, and the first input data 20 comprises the source region.

[0080] The algorithm 7 may be considered as end-to-end architecture for detecting and tracking emergency vehicles in road traffic scenarios. The architecture uses multimodal inputs from the thermal camera 3 and the microphones 5, 6, respectively. The inputs are processed in two separate blocks 11 , 12, namely a thermal detection block 11 and an audio localization block 12. The thermal detection block 11 outputs the bounding box coordinates of predefined target detection classes, which include or consist of a vehicle class and a vehicle light class. The object detection algorithm 15 may be implemented as a third ANN. The third ANN may be based, for example, on the well-known YoloV5 architecture, which may be adapted for thermal detection, as explained in more detail below.

[0081] The thermal detection block 11 may for example comprise an object association block 16, which combines useful information from the outputs of the object detection algorithm 15 and the audio localization block 12. The object association block 16 may implement a priority based association of detections which are then fed to the LSTM network, as explained in more detail below. The audio localization block 12 outputs the source regions of audio sources in the scene. This information may be used to enhance the proposed algorithm 7 to learn scene related information and distinguish the sound of an emergency vehicle from other vehicles and from background noise. The source regions may also be shared with the object association block 16 to enhance the proposed object association. The respective outputs of thermal detection block 11 and the audio localization block 12 are channeled into an object refiner block 13 comprising the first ANN 32. In the object refiner block 13, the multimodal information from the thermal detection block 11 and the audio localization block 12 are fused and a working algorithm to refine the detections and accurately track emergency vehicles across different scenarios is established. The object refiner block 13 may implement an adaptive way to track emergency flashing lights based on learned thresholds. An audio fusion loss, a coupling loss and an loll-location loss may be used to train the algorithm 7, as explained in more detail below. The output of the object refiner block 13 comprises the sequence of refined bounding boxes 10 tracking the emergency vehicle in the scene.

[0082] A detection target in the thermal domain may be, in particular, an object with low contrast and a low signal-to-noise ratio that occupies only a few tens of pixels in the imaging plane, in particular if it is far away from the thermal camera 3. However, compared with target detection under visible light, thermal detection may feature a better anti-interference ability, better night detection ability and higher accuracy. The thermal images 8 may for example be sharpened by a preprocessing block 14 of the thermal detection block 1 1 comprising one or more spatial filters to enhance the image features given in grayscale, to improve the separation degree between the small target and the background and to enhance the characteristics of the target object. The preprocessing block 14 may also apply other spatial filters, for example for edge detection, and / or noise reduction filters.

[0083] The preprocessing block 14 may also upsample the thermal images 8 and use interpolation to increase the target pixels in order to enlarge the target. By enhancing the information on the small targets and increasing the number of pixels of the small targets, the detection in complex backgrounds may be improved.

[0084] Based on the characteristics of thermal images 8, the object detection algorithm 15 may be designed as a modified version of the YoloV5 algorithm. The advantages of YoloV5 include a fast convergence, a high precision, and low hardware computing requirements, meaning that it can be well used on embedded systems.

[0085] The object detection algorithm 15 is for example designed to make use of shallow features. The detection head network may comprise four layers to focus on small objects. The feature extraction network extends and iterates a shallow cross stage partial, CSP, module, which uses an attention module forcing the network to pay more attention to the shallow and detailed features in thermal images and making the model more robust by learning more distinguishable features.

[0086] The idea behind extending the CSP module is that the resolution of the feature map in the shallow layer is high, whereas the semantic feature information extracted by the shallow network is weak. For small and weak objects with a few features in thermal images, deep convolution may cause object features to be difficult to extract. To maximize the extraction of features that are conducive to the detection of weak and small objects in thermal images, it is made full use of the high-level resolution features of the CNN in the shallow layer.

[0087] Fig. 7 shows schematically the backbone structure of the object detection algorithm 15. The input 35 to the object detection algorithm 15 is passed through a 256x256 focus slicing module 36, followed by a 128x128 CSP module 37, followed by a 64x64 CSP module 38 followed by a 32x32 CSP module 39, followed by a 16x16 CSP module 40. The shallow layer is extended to the same number of CSP module feedback iterations as the deep layer.

[0088] The structure of a CSP module is shown schematically in Fig. 4. Each convolution module 41 , 42, 43, 44, 45 represents the three operations of a standard convolution, a normalization, and applying an activation function, while the further convolution module 48 represents a standard convolution only. Through a concatenation operation 47, the feature maps containing the two branches of the convolution and the attention module 46, which is a select kernel, SK, layer, are merged, and the 128x128 features are fully extracted in the shallow layer.

[0089] The idea behind including the attention module 46 in the residual blocks of the shallow feature extraction stage is to effectively select object information, and assign more weights to small and weak objects to improve the feature expression ability of small objects for accurate detection. As shown in Fig. 4, the attention module 46 is directly embedded into the residual blocks. The attention module 46, which is shown schematically in Fig. 5, is mainly divided into three parts implementing a splitting operation, a fusion operation, and a scaling operation.

[0090] The splitting operation separates the input vector by performing convolution operations 49, 50 with two kernel sizes, for example 3x3 and 5x5, to obtain output matrices 51 and 52, and, after an addition operation, and the output matrix 53. For the fusing operation, global average pooling may be used to compress the output matrix 53 to 1x1 xC, wherein C is the number of object classes, and a channel descriptor may be used to represent the information of each channel. The dependency between the channels can be expressed as wherein U denotes the output matrix 53. W and H are the width and height, respectively, and i and j are the i-th row and j-th column of the input. A fully connected layer makes the relationship between the channels flexible and nonlinear and can be expressed as wherein represents a weight, a represents the ReLLI activation function, and B represents the batch normalization operation. The weight values calculated in the fusion stage are multiplied back to the original matrix to obtain the final output 54 of the attention block 46, which can strengthen the useful information for weak and small objects for different channel wise scenarios. The matrices are added again and merged to make full use of the shallow and deep layer information. The output from the fusion operation can be passed to a sigmoid activation, and the weight value is directly multiplied by the output matrix 53 to obtain final output 54.

[0091] The original YoloV5 algorithm uses three types of output features, while for the object detection algorithm 15, a scale feature may be added to focus on smaller objects. The feature maps are for example upscaled to 128x128 as compared to 64x64 in the original YoloV5 algorithm. The idea behind this upscaling is to make full use of the shallow and deep features by fusing the expanded 128x128 feature maps with feature maps of the same size from the second layer in the backbone network. It is further proposed to use twelve anchors with four detection scales instead of nine anchors with three detection scales in the original YoloV5 algorithm. The loss function Lp2for each detection layer is the same and is implemented by calculating the sum of a bounding box regression loss LCIoU, a classification loss Lcts, and a confidence loss Lobj

[0092] Lp2ciou T Lcis + LObj-

[0093] The detection results received from the object detection algorithm 15 is converted into sequences of outputs for each detected object, which can be input to the object refiner block 13. Usually, object detection models generate the detection results according to a confidence priority, which is, however, not a robust index for object association since the confidences may vary from time to time. Therefore, a method is proposed to associate predicted bounding boxes for each detected object across different frames by means of the object association block 16.

[0094] The object association block 16 may additionally receive the output from the audio localization block 12, namely the information about the predicted source locations of audio sources across a frame. A first priority value P1 is assigned to the detected bounding boxes, wherein P1 is higher priority for such bounding boxes, which are closer to the source location. To make the priority assignment more robust, one can also consider a second priority value P2, which is higher priority for such bounding boxes, which have a smaller distance measure to a predefined reference point 55 as depicted exemplarily for several bounding boxes 56 in Fig. 6. The reference point 55 may for example be defined as a point in the thermal images 8 at (W / 2, H), wherein W and H are the width and the height of the thermal image, respectively. The distance measure may be computed as d = (AX2+ (AY)2)1 / 2, wherein AX is the horizontal distance of the respective bounding box from the reference point 55, AY is the vertical distance of the respective bounding box from the reference point 55, and p is a predefined regularization parameter. The overall priority is assigned to each bounding box 56 according to the first priority P1 followed by the second priority P2. It is noted that, while improving the accuracy and robustness of the tracking, the audio datasets 9 are not necessarily required. In this case, only the second priority P2 may be considered.

[0095] By means of the prioritization, it is avoided that the bounding boxes 56 of a given frame are fed randomly to the object refiner block 13. For two consecutive frames, the priorities of bounding boxes 56 belonging to the same targets can be expected to differ only very little by way of construction of the spatial association achieved by the object association block 16.

[0096] The audio localization block 12 may comprise a noise filter module 17 as an initial step in order to capture important sound features among various other sounds in a traffic scene. For each of the audio datasets 9, two features may be extracted by a feature extraction module 18, namely the Mel-frequency cepstral coefficients, MFCCs, and the zero crossing rate, ZCR. For determining the MFCCs, two filters may be applied that recognize frequencies both above and below a threshold, respectively, for example 1 kHz. ZCR indicates the number of times a signal crosses the zero-amplitude line from positive to negative values or vice versa.

[0097] For example, the mean value and standard deviation of thirteen MFCCs and ZCR may be determined based on test audio datasets for an emergency vehicle siren on the one hand and for urban background sounds on the other hand as

[0098] MFCC = 2595 log and

[0099] A comparative analysis of the features between the two types of test audio datasets reveal distinctly different distributions for siren and urban background sounds.

[0100] The audio datasets 9 may be resampled at 48 kHz, converted into a log-spectrogram, for example with a window length of 0.01 s and half-window overlap, and may then be treated as a 257x200 grayscale image. The Mel spectrogram may be used to provide the models with sound information similar to visually perceivable information. The raw audio waveforms may be passed through filter banks to obtain the Mel spectrograms.

[0101] The audio source detection may be treated as a multiclass classification task and the localization or the direction of arrival, DOA, estimation may be treated as a regression task as shown in schematically in Fig. 7 by a detection block 19 of the audio localization block 12. The output format of the detection may be a C-dimensional one-hot label vector 59, where C is the number of classes, for example C=2 in the illustration of Fig. 7. The regression-based DOA output is a real-valued matrix of dimension GxCx3, wherein the 3 origins from the 3-dimensional spatial cartesian coordinates. In Fig. 7, this is illustrated as C=2 matrices 57, 58 of dimensions Gx3. G is the total number of instances detected sound instances of the C classes. In the example of Fig. 7, G=3. For example, the two classes in Fig. 7 may correspond to a police siren and an ambulance siren active at different spatial locations, respectively. This approach is able to provide a high-resolution, continuous, real-valued estimation for the DOAs. The regression-based DOA estimations also allows estimate DOA locations that have not been seen in the training data.

[0102] In Fig. 8 an exemplary architecture of the audio localization block 12 is shown schematically. The phase and magnitude spectrograms generated by the feature extraction module 18 are used as separate features. For example, only the M / 2 positive frequencies without the zeroth bin are used. The output of the feature extraction module 18 is a feature sequence of T frames, with an overall dimension of T x M / 2 x 2C, where the 2C dimension consists of C magnitude components and C phase components and is fed to an ANN 67 with multiple, for example three, convolutional layers. Each convolutional layer has P filters of 3 x 3 x 2C dimensional receptive fields acting along the time-frequency-channel axis with a ReLLI activation. After each convolutional layer, the output activations are normalized using batch normalization, and the dimensionality is reduced using maximum pooling along the frequency axis, thereby keeping the sequence length T unchanged.

[0103] The output of the final convolutional layer with P filters is of dimension T x 2 x P, where the reduced frequency dimension of 2 is a result of the maximum pooling. The output is further reshaped to a T frame sequence of feature vectors with lengths 2P and fed to bidirectional RNN layers which are used to learn the temporal context information from the output activations. Q nodes of gated recurrent units, GRUs, 68 are used in each layer with tanh activations. This is followed by two branches of fully connected layers 69, 70, 71 , 72 in parallel, one for the audio source detection and one for the DOA estimation. The first fully connected layer 69, 71 in both branches contains R nodes each with linear activation, respectively. The last fully connected layer 70 in the in the audio source detection branch consists of N nodes with sigmoid activation, each corresponding to one of the N sound event classes to be detected. The last fully connected layer 72 in the DOA estimation branch consists of 3N nodes with tanh activation, where each of the N sound event classes is represented by 3 nodes corresponding to the sound event location in x, y, and z, respectively.

[0104] The input data 20 to the object refiner block 13 comprises output from the thermal detection block 11 and the audio localization block 12 and uses it to track the vehicles and flashing emergency vehicle lights, followed by coupling the tracked outputs to produce the refined bounding boxes 10. In particular, the input data 20 comprises the priority indices of detected objects P, an input data array {X1 , X2, X3, ...XT} of detected bounding boxes over T frames, the fed back predicted output data, T sets of the top 5 confidence values, and the audio regions from the audio localization block 12.

[0105] For each detected object, the object tracking state is monitored, which can be "newly appeared", "gradually appearing", "stable detection" or "disappeared". The tracking status helps to control the dLSTM refinement correctly. With all the information of the detected objects collected from the thermal detection block 11 and the audio localization block 12, the tracking state can be determined as follows (see also Fig. 2).

[0106] For consecutive pairs of frames t and t+1 , the intersection over union, loll, of the corresponding bounding boxes of a target object is computed in block 21 . Two counters D1 , and D2 are initialized and set to 0. The first counter D1 counts the number of times the tracking status is set to "disappeared" and the second counter D2 counts the number of times the tracking status is set to "gradually appearing". If the confidence value for the bounding box of frame t is at least a predefined minimum confidence and the loU is less than a predefined threshold or if the class index of the consecutive bounding boxes is different, then the tracking status is set to "newly-appeared" in block 22, the first counter D1 is set to 0 and the second counter D2 is incremented by 1 in block 23.

[0107] If the confidence value for the bounding box of frame t is at least the minimum confidence, the second counter D2 is at least 1 and less than a predefined maximum number, the loU is greater than the threshold and the class index is also the same, then the tracking status is set to "gradually appearing" in block 24, the first counter D1 is set to zero and the second counter D2 is incremented by 1 in block 25. If the confidence value for the bounding box of frame t is at least the minimum confidence, the second counter D2 is equal to the maximum number, the loll is greater than the threshold and the class index is also the same, then the tracking status is set to "stable detection" in block 26, the first counter D1 is set to zero and the second counter D2 is set to zero in block 27. If the confidence value for the bounding box of frame t is less than the minimum confidence, then the tracking status is set to "disappeared" in block 29, and the first counter D1 is incremented by 1 in block 30. In block 31 , it is checked whether the first counter D1 is equal to or less than a predefined further maximum number N. If this is the case, the tracking is continued, otherwise the tracking for this bounding box is stopped.

[0108] D1 is a counter designed to count for example the number of times the vehicle lights of vehicles are disappearing from the tracking system. In this way, the blinking nature of emergency vehicle lights is taken into account. The maximum number N for the first counter D1 may be set based on predefined norms or regulations for the flashing rate of an emergency vehicle light. It may for example be specified that the flashing rate is at least 75 times per minute and at most 150 times per minute. Considering for example a flashing rate of 120 per minute, this means two flashes per second. Assuming 20 frames per second, and a duration of the audio datasets 9 of one second, the maximum number N can be set to 10, for example. As long as D1 < N, the tracking algorithm will still treat the object as a tracked object.

[0109] In case the tracking status is set to "stable detection" in block 26, it may be checked in block 28 whether the tracked object is a vehicle bounding box or a light bounding box. In the latter case, its tracking may be terminated, because the chances that the detected light source is not an emergency vehicle light source because it is apparently not flashing are high.

[0110] As shown schematically in Fig. 9, the double-layer dLSTM of the first ANN 32 contains K- element hidden state vectors with T time instants. A fully connected layer 66 takes h2T, the T-th hidden state vector of the second LSTM layer and outputs a 4-point prediction position of the bounding box, X~. c’t and h'tdenote the LSTM cell 60, 61 , 62, 63, 64, 65 and the hidden states of the l-th layer at the t-th time step of the dLSTM model, respectively. To make the model deeper and more accurate, two LSTM layers are stacked to achieve a better time series prediction and to avoid long delays. The hidden state h’t is treated as the decision output and the cell state and c’t receives the updates from the output of the previous step h't-i . Finally, the second LSTM layer with the LSTM cells 63, 64, 65 is followed by the fully connected layer 66, which interprets the K-feature vector to the predicted location X~ learned by the dLSTM module.

[0111] To train dLSTM module, an loll-localization loss is used, which combines the loll and a position mean square error, wherein XLdenotes a bounding box predicted during the training, Xtdenotes a corresponding ground truth for the bounding box, and a and ft are predefined weighting factors.

[0112] A coupling loss may be used to train the association of vehicles with respective light sources. Every emergency vehicle ground truth ycwill have information about the associated emergency lights through meta information in its annotation, so this loss function will associate the predicted vehicle detections and emergency light detections.

[0113] An audio fusion loss may be used to consider the predicted bounding box locations of emergency vehicles and the source regions predicted by the audio localization block 12. This loss is used to minimize the distance between the two proposed regions, thus making the predictions more robust. Therein, ytdenotes the location of ground truth bounding boxes and ytis the location of the source region predicted by the audio localization block 12.

Claims

Claims1 . Computer implemented method for tracking an emergency vehicle in an environment of an ego-vehicle (1), wherein for each frame of a sequence of frames, a respective thermal image (8) depicting a target vehicle is received from a thermal camera (3) mounted to the ego-vehicle (1 ); an object detection algorithm (15) is carried out based on the thermal images (8), wherein an output of the object detection algorithm (15) comprises respective vehicle bounding boxes for the target vehicle and respective light bounding boxes for an active light source of the target vehicle depicted by the thermal images (8); if it is determined, based on the light bounding boxes, that the light source is alternating ly active and inactive over the sequence of frames, the target vehicle is classified as an emergency vehicle and a sequence of refined bounding boxes (10) for the emergency vehicle is determined by tracking the emergency vehicle, wherein the tracking comprises applying a trained first artificial neural network, ANN, (32) to first input data (20) depending on the vehicle bounding boxes and light bounding boxes.

2. Computer implemented method according to claim 1 , wherein respective audio datasets (9) for the sequence of frames are received from at least two microphones (5, 6), which are mounted at different positions on the egovehicle (1); a source region in the environment of the ego-vehicle (1), in which an audio source emitting a siren sound is located, is determined by applying a trained second ANN (18) to second input data depending on the audio datasets (9); and the first input data (20) comprises the source region.

3. Computer implemented method according to claim 2, wherein a plurality of Mel-frequency cepstral coefficients is computed for each of the audio datasets (9); andthe audio datasets (9) are filtered depending on the respective Mel-frequency cepstral coefficients to separate the audio source emitting the siren sound from one or more further audio sources and / or background noise.

4. Computer implemented method according to claim 3, wherein, a zero-crossing rate is computed for each of the audio datasets (9); and the filtering is carried out depending on the respective zero-crossing rate.

5. Computer implemented method according to one of claims 2 to 4, wherein a respective spectrogram is generated based on each of the audio datasets (9); a sound event is detected by using a classification algorithm and a direction of arrival of the sound event is determined using a regression algorithm depending on the spectrograms.

6. Computer implemented method according to one of the preceding claims, wherein the output of the object detection algorithm (15) comprises a respective detection confidence value for each of the light bounding boxes; and the tracking of the emergency vehicle comprises determining a tracking status of the light bounding boxes for each of the sequence of frames depending on the detection confidence value of the light bounding box for the respective frame and an Jaccard index of the respective light bounding box with the light bounding box of the respective subsequent frame of the sequence of frames.

7. Computer implemented method according to claim 6, wherein the tracking status is determined as a disappeared status if the detection confidence value of the respective light bounding box is smaller than a predefined minimum confidence value; and each time the tracking status is determined as the disappeared status, a counter is incremented and a tracking of the light bounding box is cancelled only if a value of the counter is greater than a predefined limit value.

8. Computer implemented method according to claim 7, wherein the limit value is determined depending on a predefined range for an emergency flash rate of the emergency vehicle.

9. Computer implemented method according to one of the preceding claims, wherein the first ANN (32) is a long short-term memory network.

10. Computer implemented method according one of the preceding claims, wherein the thermal images (8) are preprocessed, wherein the preprocessing comprises applying at least one spatial filter for image sharpening to the respective thermal image (8); and the object detection algorithm (15) is carried out based on the preprocessed thermal images (8).11 . Computer implemented method according to claim 10, wherein the preprocessing comprises upsampling the respective thermal image (8) after applying the at least one spatial filter for image sharpening.

12. Computer implemented method according to one of the preceding claims, wherein carrying out the object detection algorithm (15) comprises applying a trained third ANN, which comprises a convolutional neural network, to third input data depending on the thermal images (8).

13. Computer implemented method according to one of the preceding claims, wherein the output of the object detection algorithm (15) comprises respective vehicle bounding boxes for at least one further target vehicle depicted by the thermal images (8); for each frame of the sequence of frames, a priority value is assigned to each of the vehicle bounding boxes for the respective frame depending on a position of the respective vehicle bounding box within the thermal image (8) of the respective frame; and the first input data (20) comprises the priority values of the vehicle bounding boxes.

14. Computer implemented method according to claim 13, wherein the output of the object detection algorithm (15) comprises respective light bounding boxes for a respective active light source of the at least one further target vehicle depicted by the thermal images (8); for each frame of the sequence of frames, a priority value is assigned to each of the light bounding boxes for the respective frame depending on a position of therespective light bounding box within the thermal image (8) of the respective frame; and the first input data (20) comprises the priority values of the light bounding boxes.

15. Computer implemented method according to one of the preceding claims, wherein the first ANN (32) is trained by minimizing at least one loss function comprising a loss function, which is given bywherein Xtdenotes a bounding box predicted during the training, Xtdenotes a corresponding ground truth for the bounding box, a and ft are predefined weighting factors, and loll denotes the Jaccard index.

16. Method for guiding an ego-vehicle (1 ) at least in part automatically, wherein a computer implemented method according to one of the preceding claims is carried out and at least one control signal for guiding the ego-vehicle (1 ) at least in part automatically is generated by at least one computing unit depending on the sequence of refined bounding boxes (10); and / or assistance information for assisting a driver of the ego-vehicle (1 ) at guiding the ego-vehicle (1 ) is generated by the at least one computing unit depending on the sequence of refined bounding boxes (10).

17. Data processing apparatus (4) comprising at least one computing unit, which is adapted to carry out a computer implemented method according to one of claims 1 to 15.

18. Electronic vehicle guidance system (2) for an ego-vehicle (1 ) comprising a data processing apparatus (4) according to claim 17, wherein the at least one computing unit is configured to generate at least one control signal for guiding the ego-vehicle (1 ) at least in part automatically depending on the sequence of refined bounding boxes (10); and / orassistance information for assisting a driver of the ego-vehicle (1 ) at guiding the ego-vehicle (1) depending on the sequence of refined bounding boxes (10).

19. Electronic vehicle guidance system (2) according to claim 17, which comprises the thermal camera (3) for the ego-vehicle (1 ).

20. Computer program product comprising instructions, which, when carried out by a data processing apparatus (4), cause the data processing apparatus (4) to carry out a computer implemented method according to one of claims 1 to 15 or a method according to claim 16.

Images