Locating an emergency vehicle and at least partially automated driving of a vehicle

Abstract

To locate an emergency vehicle, first sensor data (8), which includes camera images (8a, 8b) generated by a camera (5, 6) of an ego vehicle (1) and depicts the environment of the ego vehicle (1), and second sensor data (9), which includes radar sensor data generated by a radar system (4) of the ego vehicle (1) and depicts the environment of the ego vehicle (1), are acquired. By applying a machine learning model (MLM) (10), trained at least for object recognition, to the first sensor data (8), a light pattern of an activated emergency lighting unit is identified. The position of the emergency vehicle is determined by applying at least one trained recurrent neural network (RNN) (11) to the second sensor data (9). A lane in which the emergency vehicle is located is determined based on the determined position.

Description

[0001] The present invention relates to a computer-implemented method for locating an emergency vehicle, wherein initial sensor data, comprising camera images depicting the environment of the ego-vehicle, are obtained. The invention further relates to a method for at least partially automating the control of a vehicle, wherein the aforementioned computer-implemented method is carried out, to a data processing system configured to carry out the aforementioned computer-implemented method, to an electronic vehicle control system incorporating the aforementioned data processing system, and to corresponding computer program products.

[0002] The smooth and unimpeded movement of emergency vehicles, including motorcycles, cars, and trucks, on roads is a crucial consideration for advanced driver assistance systems (ADAS) and other functions that enable at least partially automated vehicle operation. Emergency vehicles, such as ambulances, fire engines, and police cars, should be accurately detected in real time to allow for appropriate safety measures to be taken when necessary. In particular, locating an emergency vehicle is advantageous for determining whether a safety action, such as changing lanes, is required.

[0003] Numerous approaches exist that utilize visible-vision cameras and their corresponding images as the basis for emergency vehicle detection using a trained machine learning model (MLM). A challenge lies in reliably detecting the emergency vehicle in both bright and dark scenarios, for example, due to sensor limitations related to camera contrast. Furthermore, reliability and accuracy are often limited to close-range detection, such as 0-25 meters from the vehicle.

[0004] One objective of the present invention is to provide a means of locating an emergency vehicle which exhibits increased robustness in the face of various environmental conditions.

[0005] This objective is achieved by the subject matter of the independent claim. Further embodiments and preferred embodiments are the subject matter of the dependent claims.

[0006] The invention is based on the idea of ​​using a multimodal approach in which at least one type of camera is used to identify a light pattern of an activated emergency lighting unit of the emergency vehicle and a radar system is used to determine the position of the emergency vehicle.

[0007] According to one aspect of the invention, a computer-implemented method for locating an emergency vehicle, in particular an emergency vehicle in the vicinity of an ego-vehicle, is provided. First sensor data, comprising camera images of the ego-vehicle's environment generated by a camera of the ego-vehicle according to a sequence of camera frames, are obtained, particularly from the camera. Second sensor data, comprising radar sensor data generated by a radar system of the ego-vehicle and also representing the ego-vehicle's environment, are obtained. By applying a machine learning model (MLM), trained at least for object recognition, to the first sensor data, a light pattern of an activated emergency lighting unit of the emergency vehicle is identified.The position of the emergency vehicle is determined by applying at least one trained recurrent neural network (RNN), for example, a long short-term memory (LSTM), particularly a bidirectional LSTM, to the second set of sensor data. A lane in which the emergency vehicle is located is then determined based on this position.

[0008] In other words, determining the lane in which the emergency vehicle is located is equivalent to locating the emergency vehicle.

[0009] Unless otherwise specified, all steps of the computer-implemented procedure can be performed by a data processing system that includes at least one data processing device, in particular a data processing system of the vehicle. Specifically, the at least one data processing system is configured or adapted to perform the steps of the computer-implemented procedure. For this purpose, the at least one data processing device can, for example, store a computer program containing instructions which, when executed by the at least one data processing device, cause the at least one data processing device to perform the computer-implemented procedure. The terms "data processing system" and "at least one data processing device" may be used synonymously.

[0010] All data processing devices of the at least one data processing device can be contained within the vehicle. However, it is also possible that all data processing devices of the at least one data processing device are part of an external computing system located outside the vehicle, for example, a mobile electronic device, a backend server, or a cloud computing system. It is also possible that the at least one data processing device comprises at least one vehicle data processing device as well as at least one external data processing device that is contained within the external computing system.The vehicle data processing device, at least one of which may be, for example, one or more electronic control units (ECUs), and / or one or more zone control units (ZCUs), and / or one or more domain control units (DCUs) of the vehicle.

[0011] If the at least one data processing device comprises two or more data processing devices, certain steps performed by the at least one data processing device can be understood as different data processing devices performing different steps or different parts of a step. In particular, it is not necessary for each data processing device to perform the steps completely. In other words, the execution of the steps can be divided among the two or more data processing devices.

[0012] Each execution of the computer-implemented method yields a corresponding execution of a method for locating an emergency vehicle, which is not purely computer-implemented, by including the respective steps of generating the first sensor data and / or the second sensor data.

[0013] A trained MLM can be understood as an algorithm, particularly a computer-implemented algorithm, that can reproduce functions that are concretely or more broadly possible through human cognitive performance. A trained MLM can also be described, for example, as a "trained function."

[0014] When training a multi-level marketing (MLM), the MLM's parameters are generally adjusted or updated. Training can be supervised, partially supervised, or unsupervised. It can also involve reinforcement learning, representational learning, and / or other known training methods. In particular, the MLM's parameters can be iteratively adjusted over multiple training steps. Specifically, a predefined loss function can be minimized during training. If the MLM is an artificial neural network (ANN), a backpropagation algorithm can be used to adjust the parameters.

[0015] In particular, an MLM can include an ANN, a support vector machine, a k-means clustering algorithm, a decision tree, and so on. Specifically, an ANN can be or include a deep neural network and / or a convolutional neural network, CNN (in particular a deep CNN) and / or a recurrent neural network, RNN (in particular a recurrent CNN), and / or a transformer network and / or a generative adversarial network, GAN.

[0016] The MLM used in the computer-implemented method according to the invention is preferably an ANN, for example a CNN or a transformer network. The MLM and the RNN are trained before the computer-implemented method according to the invention is carried out. The steps for training the MLM and the RNN are generally not part of the computer-implemented method according to the invention.

[0017] Training the MLM and the RNN can be performed using established training methods, such as supervised training. The MLM and the RNN can be trained separately or together in an end-to-end manner. The training data can be real-world camera images or radar data generated in traffic or road scenarios, or it can be simulated accordingly.

[0018] The MLM is trained at least for object recognition. However, it is possible that the MLM may also be trained for other tasks, such as semantic segmentation, and / or for processing the results of object recognition.

[0019] The camera can be a visible-vision camera, for example, an RGB camera, or a thermal imaging camera, also known as an infrared camera or infrared-range camera. Consequently, the camera images can be visible-vision camera images or thermal imaging camera images.

[0020] By identifying the light pattern, it is confirmed that an emergency vehicle is indeed present in the vicinity. Combining the evaluation of the first sensor data, particularly camera images, for detecting the light pattern with determining the position based on the second sensor data, particularly radar data, effectively leverages the respective advantages of both sensor modalities. Specifically, camera images can provide semantic clues and / or thermal signatures to reliably identify the light pattern, while radar data enables position determination largely independent of environmental conditions, especially at close range. Consequently, the robustness of locating the emergency vehicle is increased.

[0021] The information about which lane the emergency vehicle is in can be used by various downstream functions for driver assistance or other functions for at least partially automatic control of the ego vehicle, for example as a basis for the decision to initiate or recommend a lane change or not.

[0022] According to some embodiments, the radar sensor data are Doppler radar sensor data, and the speed of the emergency vehicle is determined by applying at least the RNN to the second sensor data, and the lane on which the emergency vehicle is located is determined depending on the speed of the emergency vehicle.

[0023] In this case, the radar system is specifically a Doppler radar system. The speed is given, for example, by an absolute value and / or a direction of the speed.

[0024] Taking into account the position and speed, the lane on which the emergency vehicle is located can be determined with greater accuracy.

[0025] According to some embodiments, the camera images are generated by a visible-vision camera of the ego-vehicle. The initial sensor data includes thermal images of the ego-vehicle's surroundings, generated by a thermal imaging camera of the ego-vehicle according to a sequence of thermal imaging camera frames.

[0026] Consequently, the light pattern of the activated operational lighting unit is identified based on the visible-vision camera images and the thermal imaging camera images. This further improves the robustness of the method for various environmental conditions. In particular, the visible-vision camera images provide especially useful semantic information about the environment, while the thermal imaging camera images depict the environment as being unaffected or only minimally affected by environmental conditions such as lighting, rain, etc.

[0027] According to some embodiments, predictions for an area of ​​interest (ROI) containing the operational lighting unit are generated by applying a first object recognition module of the MLM to the camera images in the visible range. The light pattern is identified based on the predictions for the ROI.

[0028] According to some embodiments, further predictions for the ROI containing the operational lighting unit are generated by applying a second object recognition module of the MLM to the thermal imaging camera images. The light pattern is identified depending on the further predictions for the ROI, for example, depending on the predictions for the ROI and depending on the further predictions for the ROI.

[0029] A prediction for a ROI can be given, for example, by a two-dimensional or three-dimensional boundary box, as is known in the context of MLM-based object recognition.

[0030] By focusing on the ROI and not the entire camera image in the visible range or thermal imaging camera image, the computational effort for identifying the light pattern can be reduced and the reliability for identifying the light pattern can be increased.

[0031] An object recognition module can, for example, be an ANN, in particular a CNN, trained for object recognition. Specifically, an object recognition module can include an encoder module for encoding features of the input, a decoder module for generating the prediction (in particular, a bounding box), and optionally a class label according to a variety of predefined object classes. The first object recognition module is specifically trained for object recognition in the visible spectrum, particularly based on camera images in the visible spectrum. The second object recognition module is specifically trained for object recognition in the infrared spectrum, particularly based on thermal imaging camera images.

[0032] Specifically, visible-area features can be generated by applying a trained first encoder module of the first object detection module to the visible-area camera images, and ROI predictions can be generated by applying a trained first object detection decoder module of the first object detection module to the visible-area features. For example, thermal features can be generated by applying a trained second encoder module of the second object detection module to the thermal camera images, and further ROI predictions can be generated by applying a trained second object detection decoder module of the second object detection module to the thermal features.

[0033] For example, the ROI predictions include a specific ROI prediction for each camera frame in the sequence of camera frames. Similarly, the further ROI predictions include a specific ROI prediction for each thermal imaging camera frame in the sequence of thermal imaging camera frames.

[0034] According to some embodiments, semantically segmented ROls are generated based on the predictions and further predictions for the ROI, in particular by a trained semantic segmentation module of the MLM, and the light pattern is identified by tracking a state of the deployment lighting unit based on the segmented ROls, for example using a Bayesian model.

[0035] In this way, false positive detections of emergency vehicles can be largely avoided.

[0036] The semantic segmentation module can be, for example, an ANN, particularly a CNN, trained for semantic segmentation. Specifically, the semantic segmentation module can be an encoder module for encoding features of the respective input and a semantic segmentation decoder module for generating the respective output. The output might include, for example, a semantic map for the ROI, showing which pixels of the semantic map correspond to the operational lighting unit and which do not.

[0037] According to some embodiments, the second sensor data includes at least an audio sequence representing ambient noise generated by at least one microphone of the Ego vehicle.

[0038] Consequently, the RNN determines the vehicle's position and / or speed based on radar data and at least one audio sequence. This increases the accuracy and reliability of determining the position and / or speed. Specifically, the speed is determined at least partially based on the at least one audio sequence if that sequence comprises two or more audio sequences generated by two or more microphones of the ego vehicle.

[0039] According to some embodiments, a segmented representation of a road in the environment is generated by applying a trained semantic segmentation decoder module, for example, the MLM, to the features in the visible area. The lane in which the emergency vehicle is located is determined based on the segmented representation of the road.

[0040] The segmented representation of the road corresponds, for example, to a two-dimensional map depicting the surroundings, similar to a camera image. Each pixel of the two-dimensional map is assigned a class label indicating the type of object to which the respective pixel belongs. Specifically, the class labels of the segmented representation of the road indicate which pixels belong to the road and identify the respective lane of the road to which the pixel belongs. In combination with the position and, if applicable, the speed determined based on the second sensor data, the lane in which the emergency vehicle is located can be determined with particular accuracy and reliability.

[0041] According to a further aspect of the invention, a method for at least partially automated driving of a vehicle is provided. This involves carrying out a computer-implemented method according to the invention. At least one control signal for at least partially automated driving of the ego-vehicle is generated depending on the specific lane in which the emergency vehicle is located, and / or assistance information for assisting a driver of the ego-vehicle is generated depending on the specific lane in which the emergency vehicle is located.

[0042] The at least one control signal can be provided, for example, to one or more actuators of the vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the vehicle. The one or more actuators can influence the longitudinal and / or lateral steering of the vehicle in order to steer the vehicle at least partially automatically based on the at least one control signal.

[0043] The assistance information can be output via a vehicle output device, for example a display and / or an audio output system and / or a haptic output system.

[0044] According to some embodiments, a lane change of the ego vehicle is automatically initiated depending on at least one control signal depending on the specific lane on which the emergency vehicle is located, in particular if the specific lane on which the emergency vehicle is located corresponds to a lane on which the ego vehicle is located.

[0045] Consequently, potential collisions between the ego vehicle and the emergency vehicle can be avoided.

[0046] According to some accounts, the assistance information includes a request to initiate a lane change if the specific lane in which the emergency vehicle is located corresponds to a lane in which the ego vehicle is located.

[0047] Consequently, potential collisions between the ego vehicle and the emergency vehicle can be avoided.

[0048] According to another aspect of the invention, a data processing system is provided which is set up to carry out a computer-implemented method according to the invention.

[0049] The terms "data processing system" and "at least one data processing device" can be used synonymously in this disclosure. In this disclosure, a data processing device, also referred to as a computing device, can be understood, for example, as a device with processing circuitry for data processing. A data processing device can therefore perform computational operations to process data. Indexed access to a data structure, such as a lookup table (LUT) or a database, can also be considered a computational operation. Data processing that is partially or completely implemented in hardware can also be considered a computational operation.

[0050] In particular, a data processing device may contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more systems on a chip (SoCs). A data processing device may also contain one or more processors, for example, one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs).The data processing device may also include a physical or virtual network of computers or other units of the aforementioned type.

[0051] A data processing device may also include one or more hardware and / or software interfaces, for example for receiving and / or providing data.

[0052] A data processing device can also include one or more storage devices. A storage device can be volatile data storage, for example, dynamic random access memory (DRAM) or static random access memory (SRAM), or non-volatile data storage, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), or magnetoresistive random access memory.It can be implemented as MRAM (magnetoresistive random access memory) or as phase-change random access memory, PCRAM (phase-change random access memory).

[0053] According to a further aspect of the invention, an electronic vehicle guidance system is provided. The electronic vehicle guidance system comprises a data processing system and a control system according to the invention. The control system is configured to generate at least one control signal for at least partially automatic guidance of the ego vehicle depending on the specific lane in which the emergency vehicle is located, and / or to generate assistance information to assist a driver of the ego vehicle depending on the specific lane in which the emergency vehicle is located.

[0054] The control system can, for example, be part of the data processing system or another data processing system.

[0055] An electronic vehicle control system can be understood as an electronic system designed to control a vehicle fully automatically or autonomously, and in particular, without the need for manual intervention or control by a driver or user of the vehicle. The vehicle automatically performs all necessary functions, such as steering, deceleration, and / or acceleration maneuvers, as well as monitoring and recording road traffic and reacting accordingly. Specifically, the electronic vehicle control system can implement a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. An electronic vehicle control system can also be implemented as an Advanced Driver Assistance System (ADAS), which assists a driver in semi-automated or semi-autonomous driving.In particular, the electronic vehicle guidance system can implement a semi-automatic or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the corresponding standard dated April 2021.

[0056] The at least partially automatic control or operation of the vehicle can therefore include operating the vehicle in a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. The at least partially automatic control or operation of the vehicle can also include operating the vehicle in a semi-automatic or semi-autonomous driving mode according to Levels 1 to 4 of the SAE J3016 classification.

[0057] According to some embodiments, the electronic vehicle guidance system includes the visible-area camera and / or the radar sensor system and / or the thermal imaging camera and / or at least one microphone.

[0058] Further embodiments of the electronic vehicle guidance system according to the invention arise directly from the various embodiments of the computer-implemented method for locating an emergency vehicle and the method for guiding a vehicle according to the invention, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various embodiments of the computer-implemented method for locating an emergency vehicle or the method for guiding a vehicle according to the invention can be transferred accordingly to corresponding embodiments of the electronic vehicle guidance system according to the invention. In particular, the electronic vehicle guidance system according to the invention is designed or programmed to carry out the computer-implemented method or the method according to the invention.In particular, the electronic vehicle guidance system according to the invention carries out the computer-implemented method or the method according to the invention.

[0059] According to another aspect of the invention, a computer program containing instructions is provided. When the instructions are executed by a data processing system, they cause the data processing system to carry out a computer-implemented method according to the invention.

[0060] The commands can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly language, and / or as source code in a programming language such as C, and / or as a program script, such as Python.

[0061] According to a further aspect of the invention, a further computer program containing additional commands is provided. When these additional commands are executed by an electronic vehicle guidance system according to the invention, in particular by the data processing system of the electronic vehicle guidance system, the additional commands cause the electronic vehicle guidance system to carry out a method according to the invention for at least partially automatically guiding a vehicle.

[0062] The additional commands can be provided, for example, as program code. This program code can be provided, for example, as binary code or assembly language, and / or as source code in a programming language such as C, and / or as a program script, such as Python.

[0063] According to another aspect of the invention, a computer-readable storage medium is provided which stores a computer program and / or another computer program according to the invention.

[0064] The computer program, the further computer program, and the computer-readable storage medium are respective computer program products that contain the commands and / or the further commands.

[0065] Further features of the invention will become apparent from the claims, the figures, and the description of the figures. The features and combinations of features mentioned above in the description, as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures, may be encompassed by the invention not only in the combinations specified, but also in other combinations. In particular, the invention may also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention may encompass embodiments and combinations of features that go beyond or deviate from the combinations of features set out in the references to the claims.

[0066] The invention is explained in detail below with reference to specific exemplary embodiments and corresponding schematic drawings. Identical and functionally equivalent elements may be designated by the same reference numerals in the drawings. The description of identical or functionally equivalent elements is not necessarily repeated with reference to other figures.

[0067] This shows: Fig. 1 schematically an ego vehicle with an exemplary embodiment of an electronic vehicle guidance system according to the invention; Fig. 2 a schematic flowchart of an exemplary embodiment of a computer-implemented method according to the invention for locating an emergency vehicle; Fig. 3 a schematic flowchart of a further exemplary embodiment of a computer-implemented method according to the invention for locating an emergency vehicle; Fig. 4 a schematic block diagram of an object recognition module for use in a further exemplary embodiment of a computer-implemented method according to the invention for locating an emergency vehicle; Fig. 5 a schematic block diagram of an RNN for use in a further exemplary embodiment of a computer-implemented method according to the invention for locating an emergency vehicle; and Fig. 6 a schematic flowchart of an exemplary embodiment of a method according to the invention for at least partially automatic driving of a vehicle.

[0068] Fig. Figure 1 schematically shows an ego-vehicle 1 with an exemplary embodiment of an electronic vehicle guidance system 2 according to the invention. The electronic vehicle guidance system 2 includes a data processing system 3 according to the invention, which is configured to carry out a computer-implemented method according to the invention to determine a lane in which an emergency vehicle is located in the vicinity of the ego-vehicle 1.

[0069] The electronic vehicle guidance system 2 further includes a control system which is designed to generate at least a control signal for at least partially automatic guidance of an ego vehicle 1 depending on the specific lane on which the emergency vehicle is located, and / or assistance information to assist a driver of the ego vehicle 1 depending on the specific lane on which the emergency vehicle is located.

[0070] The ego vehicle 1, for example the electronic vehicle guidance system 2, includes a radar system 4, in particular a Doppler radar system, and at least one camera 5, 6, for example a visible-vision camera 5 or a thermal imaging camera 6. In some embodiments, the ego vehicle 1, for example the electronic vehicle guidance system 2, includes the visible-vision camera 5 and the thermal imaging camera 6. In some embodiments, the ego vehicle 1, for example the electronic vehicle guidance system 2, includes at least one microphone 7a, 7b, for example two microphones 7a, 7b, which are mounted at different positions of the ego vehicle 1.

[0071] As mentioned above, the data processing system 3 is configured to carry out a computer-implemented method according to the invention for locating the emergency vehicle. A schematic flowchart of such a computer-implemented method is shown in Fig. 2 shown.

[0072] The first sensor data 8, consisting of camera images of the ego-vehicle 1 generated by a camera 5, 6 according to a sequence of camera frames, is obtained. Second sensor data 9, consisting of radar sensor data generated by the ego-vehicle 1's radar system according to a sequence of radar frames, is also obtained. By applying an MLM 10, trained at least for object recognition, to the first sensor data 8, a light pattern of an activated emergency lighting unit is identified. The position of the emergency vehicle is determined by applying at least one trained RNN 11, for example, a bidirectional LSTM, to the second sensor data 9. The lane on which the emergency vehicle is located is determined by block 12, depending on the determined position.

[0073] Fig. Figure 3 shows a schematic flowchart of another exemplary embodiment of a computer-implemented method according to the invention for locating an emergency vehicle. The embodiment of Fig. 3 is based on the embodiment of Fig. 2. In this embodiment, the first sensor data 8 include the camera images 8a generated by the visible-vision camera 5 and the thermal-vision camera images 8b generated by the thermal-vision camera 6. The second sensor data 9 include the radar sensor data 9a generated by the radar system and at least one audio sequence 9b representing ambient noise generated by the at least one microphone 7a, 7b.

[0074] Predictions 13a for a region of interest (ROI) containing the operational lighting unit are generated by applying a first object detection module 10a of the MLM 10 to the visible-area camera images 8a, and further predictions 13b for the ROI containing the operational lighting unit are generated by applying a second object detection module 10b of the MLM 10 to the thermal imaging camera images 8b. Semantically segmented ROls are generated based on predictions 13a and 13b for the ROI by a semantic segmentation module 10c. The light pattern is identified by tracking the state of the operational lighting unit based on the segmented ROls using a Bayesian model 14.

[0075] A fusion block 15 fuses the tracking results with the predictions of the RNN 11, including the position and speed of the emergency vehicle. A segmented representation of a road in the environment is generated by applying a trained semantic segmentation decoder module 16 to the features in the visible area, and the lane in which the emergency vehicle is located is determined based on the segmented representation of the road.

[0076] During a training phase of the algorithm 46, 14, 15, ground truth data 17 can be provided to evaluate a loss function that depends on a deviation between the fused predictions from the fusion block and the corresponding ground truth 17.

[0077] In some embodiments, the categorical cross-entropy loss, also known as the softmax loss, can be used. The categorical cross-entropy loss is given by L=−∑i=1Nyi log y^i, where N denotes the number of object classes, y i the binary indicator for whether class i is correct or not for a given instance obtained from the ground truth data 17, and ŷ i the predicated class.

[0078] There are well-documented limitations to previous approaches, such as accurately detecting emergency vehicles within a range of 0 to 25 meters, both day and night, determining their speed and trajectory, and so on. This is particularly challenging in nighttime scenarios, as conventional deep learning algorithms often fail to identify the emergency vehicle. The presented multimodal sensor approach can accelerate the detection and localization of the emergency vehicle.

[0079] For example, a four-stage scheme can be used. The first stage corresponds to the detection of the emergency vehicle and segmentation of the ROI. The object detection modules 10a and 10b can, for example, be based on YoloV5m architectures to reliably detect the emergency vehicle using visual and thermal signatures. The noise, especially the siren noise, of the emergency vehicle can also be detected using RNN 11. The speed and position of the emergency vehicle can also be identified using the same RNN 11. A light component segmentation algorithm, represented by the semantic segmentation module 10c, can be used to extract the segmented ROI. In particular, the acoustic-based and radar-based decisions for future use can be recorded.

[0080] Fig. Figure 4 shows a schematic block diagram of the first object recognition module 10a for use in a further exemplary embodiment of a computer-implemented method according to the invention, for example in the one relating to Fig. 3 described computer-implemented methods.

[0081] The first object detection module 10a is based on the YoloV5 architecture, in particular YoloV5I, and includes a backbone block 19, which is designed, for example, according to the CSPDarkNet-53 architecture, followed by a neck block 20, which is designed, for example, according to the PANet architecture, followed by a detection head block 21, which is designed, for example, as a Yolo layer.

[0082] The camera images 8a in the visible range are guided through two successive bottleneck CSP blocks 22, 23 of the backbone block 19 followed by an SPP block 24 of the backbone block 19. SPP block 24 is followed by another bottleneck CSP block 25 of neck block 20. Bottleneck CSP block 25 is followed by a series of: a 1x1 folding block 26, an upsampling block 27, a chaining block 28 that chain the outputs of upsampling block 27 and bottleneck CSP block 23, a bottleneck CSP block 29, a 1x1 folding block 30, an upsampling block 31, and a chaining block 32 that chain the outputs of upsampling block 31 and bottleneck CSP block 22.This is followed by a series of: a bottleneck CSP block 33, a 3x3 S2 folding block 34, a chain block 35 that chain the outputs of the 3x3 S2 folding block 34 and the 1x1 folding block 30, a bottleneck CSP block 36, a 3x3 S2 folding block 37, a chain block 38 and a bottleneck CSP block 39 that also receives the output of the bottleneck CSP block 25.

[0083] Detection head block 21 contains a 1x1 convolution block 40, which receives the output of bottleneck CSP block 33, a 1x1 convolution block 41, which receives the output of bottleneck CSP block 36, and a 1x1 convolution block 42, which receives the output of bottleneck CSP block 39. Detection head block 21 also contains a scaling block 43, which receives the output of 1x1 convolution block 40, a scaling block 44, which receives the output of 1x1 convolution block 41, and a scaling block 45, which receives the output of 1x1 convolution block 42.

[0084] Issue 13a of the first object recognition module 10a includes the 13a for the ROI and optionally respective class labels such as "ambulance", "fire engine", "police car", "rescue motorcycle", "fire engine motorcycle", "police motorcycle" or similar.

[0085] Various modifications can be made to the architecture to adapt it to specific use cases. For example, ResNet50 or ResNeXt can be used for backbone block 19. Anchorbox sizes and ratios can be adjusted for better matching of object sizes and shapes within a given dataset. Hyperparameters such as learning rate, batch size, or epoch count can also be customized.

[0086] The second object recognition module 10b can be designed analogously, except that YoloV5m is used instead of YoloV5I.

[0087] Fig. Figure 5 shows a schematic block diagram of an RNN 11 for use in a further exemplary embodiment of a computer-implemented method according to the invention, for example in the computer-implemented method which refers to Fig. 3 is described. The RNN 11 is designed as a bidirectional LSTM with an input gate 47, a forget gate 48 and an output gate 49.

[0088] The output of the LSTM represents a decision regarding the position and speed of the emergency vehicle based on the audio sequence 9b and the radar data 9a. The forward output corresponds to a sequence of hidden states produced by the LSTM as it processes the input data 9a, 9b from the beginning to the end of the sequence. The reverse output corresponds to a sequence of hidden states produced by the LSTM as it processes the input data 9a, 9b in reverse, from the end to the beginning of the sequence. This bidirectional architecture is particularly useful when analyzing time-series data, such as acoustic signals or radar feedback. The final output is, for example, a combination—such as a concatenation, sum, or average—of the forward and reverse outputs for each time step.

[0089] The LSTM is designed to process the audio sequence 9b to extract features such as frequency patterns, amplitude, and temporal dynamics characteristic of emergency vehicle sirens. The LSTM is also designed to process the radar data 9a to extract features such as Doppler shifts indicating velocity and area information indicating position.

[0090] The LSTM architecture can be adapted in various ways to further improve its performance. For example, separate input layers for acoustic and radar features can be used, possibly followed by initial processing layers, such as convolution layers, to extract higher-level features. A fusion layer that combines acoustic and radar features, for example by chaining them together or using an attention mechanism to focus on the most relevant features, can be implemented. Different LSTM units can be used for different sets of features, with their outputs combined later. For example, the final output layer could be designed to predict the speed and position of the responding vehicle, optionally using separate branches for each task.

[0091] The output from the bidirectional LSTM layers can be processed through one or more dense layers to predict the speed of the emergency vehicle. These dense layers can, for example, apply a regression function to estimate the speed. Similarly, the position of the emergency vehicle can be predicted using a separate branch that processes the output from the bidirectional LSTM layers focused on radar features. Optionally, the speed and position predictions can be integrated or optimized together to ensure consistency between the predicted values. Other known loss functions can be used to train the speed and position prediction, training the model to simultaneously minimize these losses.The model can also include a mechanism for dynamically weighting the importance of acoustic features compared to radar features, for example depending on the context; for example, if the radar signal is weak, the model could rely more on acoustic features.

[0092] In the second stage of the four-stage approach, the light pattern, specifically the on / off pattern of the emergency lighting unit, is tracked based on the segmented ROI. The Bayesian probability model 14 ?Verb - is configured to track the on / off light pattern from the segmented ROI. Deviating rules and regulations can be introduced during the frame rate adjustment based on local light frequency standards for emergency vehicles. The tracking results are communicated to the third stage for further action.

[0093] One purpose of ROI segmentation is to segment light sources from the rest of the image or scene. This helps distinguish relevant patterns from other objects or background noise. A semantic segmentation algorithm is applied to the input to identify and label the areas corresponding to the light sources. This might involve, for example, using a deep learning model trained to recognize different types of light in varying proportions. Once the light sources are segmented, features such as shape, intensity, and movement patterns are extracted. These features help characterize and differentiate the light sources, enabling the identification of the emergency vehicle lighting unit among other light sources.The segmented light sources can be compared, for example, with known patterns, such as the typical arrangement of headlights on a vehicle. This comparison helps in identifying the type of light source and its relevance to the tracking task.

[0094] Bayesian Model 14 is used to estimate the state of the emergency vehicle lighting unit among other light sources. The segmented light sources can be compared over a period of time, incorporating both observed data (i.e., the segmented ROI) and prior knowledge (e.g., expected movement patterns, typical light source behavior, etc.). Bayesian Model 14 tracks the position, velocity, and optionally the intensity of each light source by continuously updating its estimates as new observations are received. Bayesian Model 14 combines the segmentation data with previous probabilities to make informed predictions about the current state and future trajectory of the light source.

[0095] Segmentation reduces the amount of irrelevant data that Bayesian Model 14 has to process. By focusing only on the segmented light sources, Bayesian Model 14 can make more accurate and reliable predictions. Segmentation also helps resolve ambiguities in tracking, such as distinguishing between multiple light sources that are close together or may overlap in the field of view. Furthermore, segmentation helps filter out noise or other irrelevant parts of the scene, which is beneficial for Bayesian Model 14 to maintain accurate tracking, especially in challenging environments such as low-light or heavy traffic scenarios.

[0096] The third stage corresponds to a decision fusion for emergency vehicle detection. The detection results from the first stage, generated by the RNN 11 based on the audio sequences 9b and the radar sensor data 9a, are fused with the decisions from the tracking in the second stage. The fused result can be considered a final decision for emergency vehicle detection.

[0097] The fourth stage corresponds to the automatic or semi-automatic control of Ego Vehicle 1. The lane segmentation decoder module 16 is used to identify the lane in which the emergency vehicle is located, and in particular the lane in which Ego Vehicle 1 is located. If both are in the same lane, then Ego Vehicle 1 can be controlled to perform a lane change and / or the driver can be instructed accordingly.

[0098] By taking the model outputs from the first object detection module 10a, the second object detection module 10b, and the RNN 11, the lane segmentation decoder module 16 can detect lane boundaries and identify the lanes occupied by the ego vehicle 1 and the emergency vehicle, respectively. By analyzing the positions and trajectories, the lane segmentation decoder module 16 can determine whether the ego vehicle 1 needs to change lanes to avoid a collision or to yield to the emergency vehicle.

[0099] Fig. Figure 6 shows a schematic flowchart of another exemplary embodiment of a computer-implemented method according to the invention for locating an emergency vehicle.

[0100] The process begins with step 500. In step 505, the first sensor data (8) and the second sensor data (6) are obtained. In step 510, the first object recognition module (10a) and the second object recognition module (10b) are applied to the first sensor data (8), and in step 515, the RNN (Rapid Network Nebula) (11) is applied to the second sensor data (9). In step 520, it can be checked whether the ROls (Reference Objects) were successfully predicted. If not, the process starts again with step 505.

[0101] Otherwise, the segmented ROI is calculated in step 525. In step 530, the Bayesian model 14 is calculated and the segmented ROI is tracked. In step 530, the tracking results and the RNN 11 are fused to detect the emergency vehicle and determine, for example, its type, such as whether it is a car, motorcycle, or truck, and / or whether it is a fire engine, police car, or ambulance.

[0102] In step 540, the segmented representation of the road is calculated. In step 545, it is determined whether the ego vehicle 1 and the emergency vehicle are in the same lane. If not, the procedure can end with step 555. Otherwise, the ego vehicle 1 can be controlled to initiate a lane change and / or the driver can be warned accordingly in step 550.

[0103] The invention can improve the detection, tracking and localization of the emergency vehicle under various conditions, including different scenarios such as daytime or nighttime or other lighting conditions, weather conditions such as rain, fog, snow, etc., traffic density, speed of movement of the emergency vehicle and the ego vehicle 1, different road types such as highways, city streets, country roads and so on.

Claims

[1] Computer-implemented method for locating an emergency vehicle, wherein - first sensor data (8) which include camera images (8a, 8b) generated by a camera (5, 6) of the ego vehicle (1) according to a sequence of camera frames and depicting an environment of the ego vehicle (1); - second sensor data (9) which includes radar sensor data (9a) generated by a radar system (4) of the ego vehicle (1) according to a sequence of radar frames and representing the environment of the ego vehicle (1); - by applying a machine learning model (MLM) trained at least for object recognition to the initial sensor data (8), a light pattern of an activated emergency lighting unit of the emergency vehicle is identified; - a position of the emergency vehicle is determined by applying at least one trained recurrent neural network, RNN, (11) to the second sensor data (9); - a lane in which the emergency vehicle is located, depending on its specific position. [2] Computer-implemented method according to claim 1, wherein the radar sensor data (9a) are Doppler radar sensor data and a speed of the emergency vehicle is determined by at least applying the RNN (11) to the second sensor data (9) and determining the lane on which the vehicle is located depending on the speed of the emergency vehicle. [3] Computer-implemented method according to any one of the preceding claims, wherein - the camera images (8a, 8b) produced by a camera (5) of the Ego vehicle (1) operating in the visible range are camera images (8a) in the visible range; and - the first sensor data (8) from a thermal imaging camera (6) of the ego vehicle (1) include thermal imaging camera images (8b) generated according to a sequence of frames of the thermal imaging camera and depicting the environment of the ego vehicle (1). [4] Computer-implemented method according to claim 3, wherein - Predictions (13a) for an area of ​​interest, ROI, containing the operational lighting unit, are generated by applying a first object recognition module (10a) of the MLM (10) to the camera images (8a) in the visible range; - further predictions (13b) for the ROI containing the operational lighting unit are generated by applying a second object recognition module (10b) of the MLM (10) to the thermal imaging camera images (8b); and - the light pattern is identified depending on the predictions (13a) and the further predictions (13b) for the ROI. [5] Computer-implemented method according to claim 4, wherein - Features in the visible range are generated by applying a trained first encoder module (19, 20) of the first object recognition module (10a) to the camera images (8a) in the visible range; and - the predictions (13a) for the ROI are generated by applying a trained first object recognition decoder module (21) of the first object recognition module (10a) to the features in the visible area. [6] Computer-implemented method according to claim 5, wherein - a segmented representation of a street in the surroundings is generated by applying a trained semantic segmentation decoder module (16) to the features in the visible area; and - the lane in which the emergency vehicle is located is determined depending on the segmented representation of the road. [7] Computer-implemented method according to any one of claims 4 to 6, wherein - thermal features are generated by applying a trained second encoder module of the second object recognition module (10b) to the thermal imaging camera images (8b); and - the further predictions (13b) for the ROI are generated by applying a trained second object recognition decoder module of the second object recognition module (10b) to the thermal features. [8] Computer-implemented method according to any one of claims 4 to 7, wherein semantically segmented ROls are generated based on the predictions (13a) and the further predictions (13b) for the ROI and the light pattern is identified by tracking a state of the deployment lighting unit based on the segmented ROls. [9] Computer-implemented method according to claim 8, wherein a Bayesian model (14) is used for tracking. [10] Computer-implemented method according to one of the preceding claims, wherein the second sensor data (9) includes at least one audio sequence (9b) representing ambient sounds generated by at least one microphone (7a, 7b) of the Ego vehicle (1). [11] Computer-implemented method according to any of the preceding claims, wherein the RNN (11) is a bidirectional long short-term memory network, LSTM network. [12] Method for at least partially automating the driving of a vehicle, wherein a computer-implemented method according to one of the preceding claims is carried out and - at least one control signal is generated for at least partially automatic control of the ego vehicle (1) depending on the specific lane in which the emergency vehicle is located; and / or - Assistance information to assist a driver of the ego vehicle (1) is generated depending on the specific lane in which the emergency vehicle is located. [13] Method according to claim 12, wherein, depending on the specific lane on which the emergency vehicle is located, a lane change of the Ego vehicle (1) is automatically initiated depending on the at least one control signal. [14] Method according to claim 12, wherein the assistance information includes a request to initiate a lane change when the specified lane in which the emergency vehicle is located corresponds to a lane in which the ego vehicle (1) is located. [15] Data processing system (3) configured to carry out a computer-implemented method according to any one of claims 1 to 11. [16] Electronic vehicle guidance system (2) comprising a data processing system (3) according to claim 15 and a control system configured to - to generate at least a control signal for at least partially automatic control of the ego vehicle (1) depending on the specific lane in which the emergency vehicle is located; and / or - To generate assistance information to assist a driver of the Ego vehicle (1) depending on the specific lane in which the emergency vehicle is located. [17] Electronic vehicle guidance system (2) according to claim 16, comprising the camera (5, 6) and / or the radar sensor system. [18] computer program product - Commands which, when executed by a data processing system (3), cause the data processing system (3) to perform a computer-implemented method according to any one of claims 1 to 11; and / or - further commands which, when executed by the electronic vehicle guidance system (2) according to one of claims 16 or 17, cause the electronic vehicle guidance system (2) to perform a method according to one of claims 12 to 14.

Images