Automatic visual perception of vehicle using camera and ultrasound sensor system

Abstract

According to a method for automatic visual perception, a first feature map (17) is generated from a camera image (7) by a first encoder module (11) of a neural network (6) and converted into a top view. An ultrasonic pulse is transmitted into the environment, and an ultrasonic sensor signal (8) is generated as a function of a reflected portion of the transmitted ultrasonic pulse. A spatial ultrasound map (9) is generated from the ultrasound sensor signal (8) and a second feature map (22) is generated from the ultrasound map (9) by a second encoder module (12) of the neural network (6). The transformed first feature map (20) and second feature map (22) are fused and a visual perception task is performed by a decoder module (15a, 15b, 15c) of the neural network (6) according to the fused feature map.

Description

Technical Field

[0001] The invention relates to a method for automatic visual perception of a vehicle, an electronic vehicle guidance system for a vehicle, a vehicle having such an electronic vehicle guidance system and a computer program product. Background Art

[0002] For partially or fully automated driving functions, reliable automatic detection of objects in the environment of the autonomous vehicle is very important. Algorithms for automatic visual perception (also denoted as computer vision algorithms) are used to detect, localize and / or characterize objects in the environment of the autonomous vehicle, given camera images of the camera of the autonomous vehicle. Typical visual perception tasks include object detection tasks, bounding box detection of objects, semantic segmentation tasks, size regression of objects, height regression of objects, etc.

[0003] However, for certain environmental conditions, certain object types or object positions in the environment of an autonomous vehicle, reliable perception and characterization of objects is a difficult task for known algorithms for automatic visual perception, which leads to a reduced reliability and / or accuracy of the corresponding output. For example, such environmental conditions include low-light scenes, such as twilight or night scenes, in which the information contained in the camera image is very limited. Furthermore, the detection and characterization of objects in the environment of the vehicle is particularly demanding, said objects having a particularly low height from the ground, such as curbs, low walls, poles or unclassified or unknown objects. This is even more true in the case where such objects should be characterized and detected in low-light scenes.

[0004] Algorithms for automatic visual perception can be based, for example, on trained artificial neural networks, in particular convolutional neural networks (CNNs). In the paper T. Roddick and R. Cipolla "Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks", 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020 or the corresponding preprint arXiv:2003.13402v1 (hereinafter "Roddick and Cipolla"), the authors describe a method for estimating a bird's-eye view map of the environment of an autonomous vehicle directly from a monocular image using a single end-to-end deep learning architecture. For example, a backbone network based on ResNet-50 extracts image features at multiple resolutions from the input image, and a feature pyramid network enhances high-resolution features using spatial context from lower pyramid layers. A stack of dense transformation layers of the bird's-eye view transformation network maps image-based features into a bird's-eye view, and a top-down network processes the bird's-eye view features and predicts the final semantic occupancy probability.

[0005] The Feature Pyramid Network is based on the publication T. Lin et al.: “Feature Pyramid Networks for Object Detection”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 or the corresponding preprint arXiv:1612.03144v2 (hereinafter “Lin et al.”). Summary of the invention

[0006] It is an object of the present invention to enhance the reliability and / or accuracy of a vehicle's automated visual perception.

[0007] This object is achieved by the respective subject matter of the independent claims. Further embodiments and preferred embodiments are subject matter of the dependent claims.

[0008] The invention is based on the idea of ​​extracting features from a camera image by means of a trained artificial neural network, transforming the features into a top-down perspective, and fusing the transformed features with features extracted by the neural network from an ultrasonic map of the environment obtained from ultrasonic sensor signals of an ultrasonic sensor system of the vehicle. A visual perception task is then performed based on the fused features.

[0009] According to one aspect of the invention, a method for automatic visual perception with a vehicle, in particular a motor vehicle, is provided. According to the method, a camera image representing an environment of the vehicle is generated by a camera of the vehicle. At least one first feature map is generated, in particular by at least one computing unit of the vehicle, by applying a first encoder module of a trained artificial neural network to the camera image. A top view transformation module of the neural network is applied to the at least one first feature map, in particular by at least one computing unit, to transform the at least one first feature map from a camera image plane perspective of the camera to a top view perspective, also denoted as a bird's eye view or bird's eye view perspective.

[0010] Ultrasonic pulses are emitted by an ultrasonic sensor system of the vehicle into the environment, and at least one ultrasonic sensor signal is generated by the ultrasonic sensor system of the vehicle depending on the reflected part of the emitted ultrasonic pulses. A spatial ultrasound map, in particular a two-dimensional spatial ultrasound map, is generated in a top-down perspective, in particular by at least one computing unit based on the at least one ultrasonic sensor signal. At least one second feature map is generated by applying a second encoder module of the neural network to the ultrasound map. A fused feature atlas is generated by fusing at least one first feature map after transformation and at least one second feature map, in particular by at least one computing unit, for example by applying a fusion module of the neural network to at least one first feature map after transformation and at least one second feature map. The first visual perception task is performed by a first decoder module of the neural network based on the fused feature atlas, in particular by at least one computing unit using the first decoder module of the neural network based on the fused feature atlas.

[0011] The method for automatic visual perception according to the present invention can also be considered as a method comprising the steps described below: generating a camera image and at least one ultrasonic sensor signal, and applying an algorithm for automatic visual perception to the camera image and at least one ultrasonic sensor signal. The algorithm for automatic visual perception (which can also be expressed as a computer vision algorithm or a machine vision algorithm) includes a trained artificial neural network.

[0012] Algorithms for automatic visual perception can be considered as computer-implemented algorithms for automatically performing one or more visual perception tasks. Visual perception tasks (also denoted computer vision tasks) can be understood, for example, as tasks for extracting visually perceptible information from image data. In particular, visual perception tasks can in principle be performed by a person who is able to visually perceive an image corresponding to the image data. However, in the present context, the visual perception tasks are performed automatically without the support of a person.

[0013] In the context of the present invention, the method and algorithm for automatic visual perception are not performed solely on the basis of camera images as sensor input, but also on the basis of at least one ultrasonic sensor signal or sonogram, respectively. Ultrasonic waves or pulses are invisible to humans. However, the present algorithm used is denoted as an algorithm for automatic visual perception and, therefore, the method is denoted as a method for automatic visual perception.

[0014] Since the involved sensor inputs (particularly a camera image and at least one ultrasonic sensor signal) are generated by a sensor system (particularly a camera and an ultrasonic sensor system) of the vehicle, the method may be considered as a method for automated visual perception of the vehicle.

[0015] For example, the trained artificial neural network can be provided in a computer-readable form, for example stored on a storage medium of the vehicle, in particular on a storage medium of at least one computing unit.

[0016] The neural network comprises several modules, including a first encoder module, a top view transformation module and a first decoder module. A module may be understood as a software module or a part of a neural network. A software module may be understood as a software code that is functionally connected and combined into units. A software module may include or implement several processing steps and / or data structures.

[0017] In particular, a module may represent a neural network or a subnetwork itself. If not stated otherwise, a module of a neural network may be understood as a trainable module of the neural network, in particular a trained module. For example, before executing the method for automatic visual perception, the neural network and therefore all its trainable modules may be trained in an end-to-end manner. However, in other embodiments, different modules may be trained or pre-trained individually. In other words, the method according to the invention corresponds to a deployment phase of a neural network.

[0018] The training itself can be performed using conventional methods for training neural networks, in particular convolutional neural networks (CNNs), such as supervised training methods.

[0019] In order to generate a sonogram from the at least one ultrasound sensor signal, a further software module may be applied to the at least one ultrasound sensor signal, wherein the further software module is not necessarily part of the neural network. In particular, the further software module is not necessarily a trained or trainable module. However, in other embodiments, the further software module may also be a trainable or training module of the neural network.

[0020] For example, the first encoder module and the second encoder module may include one or more convolutional layers, while the first decoder module includes one or more deconvolutional layers.

[0021] The encoder module may comprise one or more convolutional layers which extract features of at least one first feature map from the camera image. The spatial dimensions of the at least one first feature map may generally be different from the spatial dimensions of the camera image, depending on the architecture of the encoder, in particular the convolutional layers, e.g. the convolution kernels involved or used. In general, the camera image may be considered as a three-dimensional tensor of size H×W×C. Therein, H×W denotes the spatial size of the camera image, i.e. the height H and width W in pixels of the camera image. C is the channel size and may, for example, correspond to different color channels in the case of camera images. However, for camera images, C may also be equal to 1. Each feature map i in the at least one first feature map is also represented by a corresponding spatial size H i ×W i and the number of channels C i to characterize, however, it may be different from the size and number of channels of the camera image. The exact height Hi and width Wi and the number of channels Ci of one of the first feature maps depends on the number of convolutional layers used by the encoder module and the size and number of convolutional kernels used. However, the features of at least one feature map can still be considered to be given in the same camera image plane perspective as the camera image. In other words, the directions corresponding to the height and width of the feature map of at least one first feature map correspond to the height and width directions in the original camera image, even though there may not be a one-to-one pixel correspondence between the features and the pixels of the camera image.

[0022] Algorithms for converting a camera image from a camera image plane perspective to a top view perspective are known per se. The camera image plane is for example perpendicular to a predefined longitudinal direction of a camera coordinate system, which may for example be parallel to the optical axis of the camera. The top view perspective corresponds to a perspective according to a top view plane, which is perpendicular to a predefined height axis. The height axis may for example be perpendicular to a road surface on which the vehicle is positioned, or in other words, parallel to a vehicle height axis of the vehicle. In some cases, this may be parallel to another coordinate axis of the camera coordinate system. Typically, the camera coordinate system may also be tilted or rotated.

[0023] However, according to the invention, instead of the camera image being transformed from the camera image plane perspective into the top view perspective, each first feature map of the at least one first feature map is transformed into the top view perspective. In this way, it is achieved that the at least one first feature map is directly comparable and fusible with the at least one second feature map derived from the ultrasound map, which also represents the environment in the top view perspective.

[0024] The top view transform module may be designed, for example, as described in the publications by Roddick and Cipolla regarding multi-scale dense transformers or stacks of dense transformer layers, respectively.

[0025] The ultrasonic sensor system comprises an ultrasonic transmitter configured to transmit ultrasonic pulses in an environment. Furthermore, the ultrasonic sensor system comprises at least one ultrasonic detector, each of which is configured to generate one of at least one ultrasonic sensor signal. In addition, the ultrasonic sensor transmitter and / or one or more other ultrasonic transmitters of the ultrasonic sensor system may transmit one or more other ultrasonic pulses, and the at least one ultrasonic sensor signal may also depend on reflected portions of those other transmitted pulses.

[0026] In general, the transmitter and the detector of the ultrasonic sensor system can be combined and designed as an ultrasonic transducer or can be implemented separately from each other. In particular, the reflected portion of the ultrasonic pulse emitted by the ultrasonic sensor can be detected by the same ultrasonic sensor (which is represented as a direct signal path) or by another ultrasonic sensor of the ultrasonic sensor system (which is represented as an indirect signal path).

[0027] Each ultrasonic sensor signal can, for example, be considered to represent the envelope of an ultrasonic wave corresponding to the reflected portion of the transmitted pulse. Each ultrasonic sensor signal is then given by the amplitude of the corresponding sensor signal as a function of time. Since the speed of sound is known, in particular for a given air temperature in the environment, the amplitude as a function of time can be directly converted into an amplitude as a function of radial distance, where appropriate interpolation or averaging can be taken into account in the case of an indirect signal path. In principle, different ultrasonic detectors can be used at different locations and at least one ultrasonic sensor signal can be evaluated to obtain not only radial distance information, but also angular information about the reflection source in the environment relative to the ultrasonic transmitter and / or detector. In this way, a two-dimensional top-down ultrasonic map representing the environment can be generated based on at least one ultrasonic sensor signal.

[0028] However, this is possible even without extracting angular information from the combination of ultrasound sensor signals. For example, in a simple approach, each angular position or each angular position within a predefined angular field of view may be processed equivalently, so that the contribution of each ultrasound sensor signal to the sonogram is essentially circular. However, heuristics or empirical information may also be used to improve the accuracy of the sonogram. For example, the contribution of each ultrasound sensor signal may be limited to the angular field of view of the corresponding emitter and / or detector of the ultrasound sensor system. Furthermore, a radial cutoff may be used.

[0029] However, alternatively or additionally, a more accurate angular distribution may be employed to take into account the angular emission and / or detection characteristics of the ultrasound transmitter or the corresponding ultrasound detector, respectively. For example, the intensity of the emitted ultrasound pulse may be maximum along the longitudinal direction of the transmitter and / or the detector sensitivity of the corresponding ultrasound detector may be maximum along the longitudinal direction of the corresponding detector. An empirically or heuristically determined distribution (e.g., a beta distribution) may be used to approximate the ultrasound characteristics. In combination with radial information obtained directly from the time-correlated at least one ultrasound sensor signal, the ultrasound map may be obtained as a two-dimensional spatial map, or in other words, an ultrasound image in a top-view perspective.

[0030] This has the advantage that the ultrasound image can be processed by the neural network in a similar way to the camera images, for example by passing the ultrasound image through one or more convolutional layers of the second encoder module to extract corresponding features. Even though the content of the ultrasound image may not be immediately interpretable by a human, the trained neural network is able to automatically interpret the encoded information, similar to camera images. By fusing the features obtained from the ultrasound image with the features obtained from the camera image, the reliability of the first visual perception task and its output can be significantly improved, in particular for low-light scenes and / or objects at a relatively low distance to the ground. In particular, the ultrasonic sensor signal represents reflective objects in the vehicle environment regardless of the lighting conditions. Since the ultrasonic sensor system is usually mounted on the vehicle at a relatively low position above the ground, for example on the rear bumper, low objects can be reliably detected and characterized.

[0031] Fusing the at least one transformed first feature map and the at least one second feature map may include applying a known mechanism for fusing feature maps, such as a cascade of corresponding feature maps. For fusion, in some embodiments, one or more of the at least one transformed first feature map and / or the at least one second feature map may be upsampled or downsampled to achieve a desired spatial dimension for further processing. For example, the fusion module is not necessarily trainable or trained.

[0032] To perform the first visual perception task, the first decoder module can be applied to the fused feature map set. In other embodiments, another network module (e.g., a top-down network module) of the neural network can be applied to the fused feature map set, and then the first decoder module is applied to the corresponding output of the other network module.

[0033] In particular, the field of view of the camera may partially overlap with the field of view of the ultrasonic sensor system (in particular relating to the joint field of view of all ultrasonic sensors of the ultrasonic sensor system generating at least one ultrasonic sensor signal). Thus, the camera image and the at least one ultrasonic sensor signal at least partially represent the same spatial area of ​​the environment. For example, the camera may be a rear-facing or side-facing camera of the vehicle, and the ultrasonic sensor system may be mounted on the side and / or rear end of the vehicle accordingly.

[0034] According to several embodiments of the method for automatic visual perception, an intermediate feature atlas is generated by applying a top-down network module of a neural network to the fused feature atlas. A first visual perception task is performed by applying a first decoder module to the intermediate feature atlas.

[0035] For example, the architecture of the top-down network described in the publication of Roddick and Cipolla can be used. However, according to the present invention, the top-down network module does not fully perform the first visual perception task. Instead, the top-down network can be considered as a task-independent processing block or a task-independent decoder block, which can process the fused feature atlas to form the basis of the first visual perception task by providing an intermediate feature atlas, and in a corresponding embodiment, form the basis of further visual perception tasks. In other words, although the top-down network represents a task-independent processing block, the first decoder module is a task-specific decoder module that is specifically trained to perform the first visual perception task.

[0036] For example, the second visual perception task is performed by the second decoder module of the neural network according to the fused feature atlas, and / or the third visual perception task can be performed by the third decoder module of the neural network according to the fused feature atlas. In particular, the first, second and third visual perception tasks are different from each other.

[0037] For example, a second visual perception task may be performed by applying a second decoder module to the intermediate feature atlas and / or a third visual perception task may be performed by applying a third decoder module to the intermediate feature atlas.

[0038] As mentioned for the first decoder module, the top-down network may also include one or more deconvolution layers. In a corresponding implementation, the same is true for the second decoder module and / or the third decoder module.

[0039] Wherein, as described for the first decoder module, the second decoder module and the third decoder module are task-specific decoder modules specially trained for the second visual perception task and the third visual perception task, respectively. On the other hand, the output of the top-down network, i.e., the intermediate feature atlas, can be used for all the first, second, and third visual perception tasks. It is conceivable to include more than three visual perception tasks in the same way.

[0040] The first visual perception task may be one of an object height regression task, a semantic segmentation task, and a bounding box detection task. Similarly, the second visual perception task may be one of an object height regression task, a semantic segmentation task, and a bounding box detection task. Similarly, the third visual perception task may be one of an object height regression task, a semantic segmentation task, and a bounding box detection task.

[0041] Preferably, the first visual perception task and the second visual perception task are different from each other. In a corresponding embodiment, the third visual perception task is preferably different from the first visual perception task and different from the second visual perception task. For example, the first visual perception task is an object height regression task, the second visual perception task is a semantic segmentation task, and the third visual perception task is a bounding box detection task.

[0042] The present invention can be used in a particularly efficient manner using two or three or more task-specific decoder modules to perform a corresponding number of visual perception tasks, since all decoder modules share the same fused feature atlas or intermediate feature atlas to perform different visual perception tasks, respectively.

[0043] The output of the corresponding decoder module according to the object height regression task includes a height map in a top-down view perspective. For example, in the case where the first visual perception task is an object height regression task, the output of the first decoder module includes a height map in a top-down view perspective. The height map in the top-down view perspective contains the predicted object heights of one or more objects in the environment.

[0044] For example, the height map may correspond to a grid, wherein each grid cell of the grid is associated with or located at a corresponding two-dimensional spatial position in the top-down view perspective, and the height map includes a predicted or estimated height value for each grid cell. For example, the grid cells may be rectangular or quadratic cells having respective side lengths of approximately one centimeter or several centimeters.

[0045] By fusing the information obtained from the at least one ultrasonic sensor signal with the information obtained from the camera image, a height map can be calculated in a particularly reliable and accurate manner. The height map can be used as a valuable input for driver assistance functions or other automated or semi-automated driving functions of the vehicle.

[0046] The output of the corresponding decoder module according to the semantic segmentation task includes the semantic segmentation image in the top view perspective. For example, when the second visual perception task is a semantic segmentation task, the output of the second decoder module includes the semantic segmentation image in the top view perspective.

[0047] The semantically segmented image can be understood as a rectangular array of pixels, where an object class from a number of predefined object classes is assigned to each pixel. There can be a one-to-one correspondence between the pixels of the semantically segmented image and a version of the camera image that is projected or transformed to a top-down perspective. However, this is not necessarily the case, as the resolution or size of the semantically segmented image may also be different from the corresponding transformed camera image.

[0048] By fusing the information obtained from the at least one ultrasonic sensor signal with the information obtained from the camera image, a semantically segmented image can be calculated in a particularly reliable and accurate manner. The semantically segmented image can be used as a valuable input for driver assistance functions or other automated or semi-automated driving functions of the vehicle.

[0049] The output of the corresponding decoder module according to the bounding box detection task includes the corresponding position and size of at least one bounding box in the top-down perspective for at least one object in the environment. For example, if the third visual perception task corresponds to a bounding box detection task, the output of the third decoder module includes the corresponding position and size of the at least one bounding box.

[0050] The bounding box may be, for example, a rectangular bounding box. However, other geometric figures may also be used. For example, in the case of a rectangular bounding box, its position may be given by the position of the center of the rectangle or the position of a corner of the rectangle or another defined position of the rectangle. In this case, the size of the bounding box may be given by the width and / or height of the rectangle or by an equivalent amount.

[0051] By fusing the information obtained from the at least one ultrasonic sensor signal with the information obtained from the camera image, the position and size of the at least one bounding box can be calculated in a particularly reliable and accurate manner. The position and size of the at least one bounding box can be used as valuable input for driver assistance functions or other automated or semi-automated driving functions of the vehicle.

[0052] According to some embodiments, the first encoder module comprises at least two encoder branches. By applying the first encoder module to the camera image, each of the at least two encoder branches generates a corresponding first feature map of at least one first feature map, the size of which is reduced relative to the size of the camera image according to a predefined scaling factor of the corresponding encoder branch.

[0053] Therein, the size of the camera image and the corresponding first feature map corresponds to the spatial size according to the camera image plane viewing angle, as described above. In particular, the scaling factor of at least two encoder branches can be implemented by using correspondingly adapted convolution kernels.

[0054] For example, the first encoder module includes a feature extraction block and a feature enhancement block after the feature extraction block. Each of the at least two encoder branches includes a corresponding portion of the feature extraction block and an associated portion of the feature enhancement block. Although the feature extraction block extracts image features according to different scaling factors, the feature enhancement block can enhance spatial features by utilizing spatial contacts from other scales (e.g., from lower scales).

[0055] The first encoder module can be designed, for example, as described in the publication of Roddick and Cipolla. Using the wording of the publication, the feature extraction block corresponds to the backbone network and the feature enhancement block corresponds to the feature pyramid or feature pyramid network.

[0056] The use of a first encoder module having at least two encoder branches as described in conjunction with a transformation of at least one first feature map into a top view perspective view is particularly beneficial.

[0057] According to several embodiments, the one or more first feature maps include at least two first feature maps, the sizes of which are reduced relative to the size of the camera image according to different predefined scaling factors.

[0058] According to some embodiments, fusing the at least one first feature map and the at least one second feature map after transformation includes concatenating the at least one first feature map and the at least one second feature map after transformation.

[0059] In this way, a particularly efficient fusion can be achieved which does not require a lot of computing time.

[0060] According to several embodiments, the top view transformation module includes a transformer pyramid network.

[0061] In particular, the top view transform module comprises a respective transformer layer, in particular a dense transformer layer, for each of the at least two encoder branches. In this way, each of the at least one first feature map can be transformed into a top view perspective according to its spatial dimension or scale, respectively.

[0062] According to several embodiments, for each of the at least one ultrasonic sensor signal, the amplitude of the respective ultrasonic sensor signal as a function of time is converted into an amplitude as a function of radial distance from the ultrasonic sensor system. For each of the at least one ultrasonic sensor signal, a distribution amplitude is calculated as the product of the amplitude as a function of radial distance and the respective predetermined angular distribution. Generating the sonogram comprises or consists of summing the distribution amplitudes, in particular for all signal summing of the at least one ultrasonic sensor signal.

[0063] The angular distribution describes, for example, how the amplitude as a function of the radial distance generally differs for different angles, in particular polar angles in a top view perspective. In addition to the angular distribution, a radial cutoff or a maximum radial distance or another radial cutoff, i.e. a minimum radial distance, can also be defined, wherein, for example, for radial distances below the minimum radial distance or above the maximum radial distance, the value of the sonogram is zero.

[0064] Empirical, experimental or heuristic knowledge can be used to define the corresponding angular distribution. In this way, two-dimensional information can be obtained from at least one ultrasonic sensor signal in a particularly simple manner. In general, the angular distribution can be different for different ultrasonic sensor signals. However, the angular distribution can also be identical and only differ in position offset and / or rotation, which reflects the different positions of the ultrasonic transmitter and / or detector.

[0065] The angular distribution can be considered as a representation of the emission and detection characteristics of the ultrasonic sensor system. In other words, the angular distribution can be considered as describing how relatively large the amplitude of the corresponding ultrasonic sensor signal will be if a given reflecting object in the vehicle's environment will be positioned at different angles. For very large angles, the amplitude may eventually drop below the noise level, which is accounted for by using the corresponding angular distribution.

[0066] For example, the angular distribution may be given by at least one beta distribution. The angular distribution may be given by a beta distribution or by a weighted sum or an average or a minimum of two or more beta distributions. In particular, in the case where the ultrasonic transmitter and the corresponding ultrasonic detector generating the ultrasonic sensor signal are not located at the same position, the angular distribution may be considered for the two positions, for example, the beta distribution may be considered for the two positions, and the actual angular distribution may be given by the minimum or average of the respective angular distributions for the different positions, etc.

[0067] The beta distribution can be given, for example, by:

[0068] f p,q (x)~x p-1 (1-x) q-1 ,

[0069] In particular, where p = q = 2, so that

[0070] f 2,2 (x)~x(1-x),

[0071] with appropriate normalization factors. Wherein x denotes a quantity that depends on (e.g. is proportional to) the polar angle in the top-down viewing angle, in particular relative to the longitudinal axis of the corresponding ultrasonic sensor of the ultrasonic sensor system. It has been found that in this way the actual behavior of the ultrasonic sensor system can be well modeled.

[0072] According to another aspect of the invention, a method for at least partially automatically guiding a vehicle, in particular a motor vehicle, is provided. The method comprises performing the method for automatic visual perception according to the invention. The method further comprises generating at least one control signal for at least partially automatically guiding the vehicle based on the result of a first visual perception task.

[0073] At least one control signal may, for example, be provided to one or more actuators of the vehicle, which automatically or partially automatically influence or perform lateral output longitudinal control of the vehicle.

[0074] For use cases or usage scenarios that may occur in the method and are not explicitly described here, it can be provided that, according to the method, error messages and / or prompts for user feedback are output and / or default settings and / or predetermined initial states are set.

[0075] According to another aspect of the invention, an electronic vehicle navigation system for a vehicle is provided. The electronic vehicle navigation system comprises a camera for the vehicle, a storage device for the vehicle storing a trained artificial neural network, at least one computing unit for the vehicle and an ultrasonic sensor system for the vehicle. The camera is configured to generate a camera image representing an environment of the vehicle, in particular when mounted accordingly to the vehicle. The at least one computing unit is configured to generate at least one first feature map by applying a first encoder module of the neural network to the camera image. The at least one computing unit is configured to transform the at least one first feature map from a camera image plane perspective to a top view perspective by applying a top view transformation module of the neural network to the at least one first feature map.

[0076] The ultrasonic sensor system is configured to transmit ultrasonic pulses and to generate at least one ultrasonic sensor signal depending on the reflected part of the transmitted ultrasonic pulses, in particular when mounted accordingly to the vehicle. At least one computing unit is configured to generate a spatial ultrasound map in a top-down perspective based on the at least one ultrasonic sensor signal, and to generate at least one second feature map by applying a second encoder module of the neural network to the ultrasound map. At least one computing unit is configured to generate a fused feature map set by fusing the transformed at least one first feature map and at least one second feature map, and to perform a first visual perception task based on the fused feature map set using a first decoder module of the neural network. The at least one computing unit is configured to generate at least one control signal for at least partially automatically guiding the vehicle based on the result of the first visual perception task.

[0077] An electronic vehicle navigation system may be understood as an electronic system that is configured to guide the vehicle in a fully automated or fully autonomous manner and, in particular, does not require manual intervention or control by the driver or user of the vehicle. The vehicle performs all required functions, such as steering maneuvers, deceleration maneuvers and / or acceleration maneuvers and automatically monitors and records road traffic and corresponding reactions. In particular, the electronic vehicle navigation system may implement a fully automated or fully automated driving mode according to level 5 of the SAE J3016 classification. The electronic vehicle navigation system may also be implemented as an advanced driver assistance system ADAS that assists the driver in partially automated or partially autonomous driving. In particular, the electronic vehicle navigation system may implement a partially automated or partially autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and hereinafter, SAE J3016 refers to the corresponding standard dated June 2018.

[0078] Thus, at least partially automatically guiding the vehicle may include guiding the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. At least partially automatically guiding the vehicle may also include guiding the vehicle according to a partially automatic or partially autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

[0079] A computing unit may be understood in particular as a data processing device, which comprises a processing circuit. Thus, a computing unit may in particular process data to perform a computing operation. This may also include operations for performing index access to a data structure (e.g. a lookup table LUT).

[0080] In particular, the computing unit may include one or more computers, one or more microcontrollers and / or one or more integrated circuits, for example, one or more application specific integrated circuits ASIC, one or more field programmable gate arrays FPGA and / or one or more systems on chip SoC. The computing unit may also include one or more processors, for example, one or more microprocessors, one or more central processing units CPU, one or more graphics processing units GPU and / or one or more signal processors, in particular one or more digital signal processors DSP. The computing unit may also include a physical or virtual cluster of computers or other such units.

[0081] In various embodiments, a computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0082] The memory unit can be implemented as a volatile data memory, such as a dynamic random access memory DRAM or a static random access memory SRAM, or as a non-volatile data memory, such as a read-only memory ROM, a programmable read-only memory PROM, an erasable programmable read-only memory EPROM, an electrically erasable programmable read-only memory EEPROM, a flash memory or a flash EEPROM, a ferroelectric random access memory FRAM, a magnetoresistive random access memory MRAM, or a phase change random access memory PCRAM.

[0083] If it is mentioned in the present disclosure that a component of the electronic vehicle navigation system according to the present invention, in particular at least one computing unit of the electronic vehicle navigation system, is adapted, configured or designed, etc. to perform or realize a specific function, to achieve a specific effect or to serve a specific purpose, then this can be understood as that in addition to being usable or suitable for the function, effect or purpose in principle or theory, the component is also specifically and actually capable of performing or realizing the function, achieving the effect or serving the purpose through corresponding adaptation, programming, physical design, etc.

[0084] Further implementations of the electronic vehicle navigation system according to the present invention directly follow the various embodiments of the method according to the present invention, and vice versa. In particular, the various features and corresponding explanations and advantages related to the various embodiments of the method according to the present invention can be similarly transferred to the corresponding embodiments of the electronic vehicle guidance system according to the present invention. In particular, the electronic vehicle navigation system according to the present invention is designed or programmed to perform the method according to the present invention. In particular, the electronic vehicle navigation system according to the present invention performs the method according to the present invention.

[0085] According to another aspect of the invention, a vehicle, in particular a motor vehicle, is provided comprising an electronic vehicle navigation system according to the invention, wherein a camera and an ultrasonic sensor system are mounted to the vehicle.

[0086] According to another aspect of the invention, a computer program product is provided comprising instructions which, when executed by an electronic vehicle navigation system according to the invention, in particular by the at least one computing unit of the electronic vehicle navigation system, cause the electronic vehicle navigation system to perform the method according to the invention for automatic visual perception or the method according to the invention for at least partially automatically guiding a vehicle.

[0087] According to another aspect of the present invention, there is provided a computer readable storage medium storing a computer program according to the present invention.

[0088] The computer program and the computer-readable storage medium may be represented as a corresponding computer program product comprising instructions.

[0089] Other features of the invention are apparent from the claims, the drawings and the description of the drawings. The features and feature combinations mentioned above in the specification and the features and feature combinations mentioned below in the description of the drawings and / or shown in the drawings may be included by the invention not only in the respective combinations described, but also in other combinations. In particular, embodiments and feature combinations that do not have all the features of the original claims may also be included in the invention. In addition, the invention may include embodiments and feature combinations that exceed or deviate from the combinations of features set forth in the claims.

[0090] In the following, the present invention will be explained in detail with reference to specific exemplary embodiments and corresponding schematic diagrams. In the accompanying drawings, identical or functionally identical elements may be represented by the same reference numerals. The description of identical or functionally identical elements is not necessarily repeated with respect to different drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 Schematically illustrates a motor vehicle according to an exemplary embodiment of the invention;

[0092] Figure 2 schematically shows a block diagram representing an exemplary embodiment of a method for automatic visual perception according to the present invention;

[0093] Figure 3 A block diagram showing a first encoder module of a neural network used in another exemplary embodiment of the method for automatic visual perception according to the present invention;

[0094] Figure 4 schematically illustrates the transformation of a feature map from a camera image plane to a top-down view perspective; and

[0095] Figure 5 A part of a method for automatic visual perception according to another exemplary embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0096] Figure 1 An example of a vehicle 1 according to the invention is shown. The vehicle 1 comprises an exemplary embodiment of an electronic vehicle navigation system 2 according to the invention.

[0097] The electronic vehicle navigation system 2 comprises a camera 4, e.g. a rear-facing camera, mounted to the vehicle 1. The electronic vehicle navigation system 2 further comprises an ultrasonic sensor system comprising one or more ultrasonic sensors 5a, 5b, 5c, 5d, 5e. Each ultrasonic sensor 5a, 5b, 5c, 5d, 5e can operate as an ultrasonic transmitter and an ultrasonic detector. However, different embodiments are also conceivable. The ultrasonic sensors 5a, 5b, 5c, 5d, 5e are mounted to the vehicle 1, e.g. at the rear end, e.g. at or in the rear bumper of the vehicle 1. In particular, the overall field of view of the ultrasonic sensors 5a, 5b, 5c, 5d, 5e at least partially overlaps the field of view of the camera 4.

[0098] The electronic vehicle navigation system 2 also includes a storage device (not shown) that stores the trained artificial neural network 6 (see, for example, Figure 2 The electronic vehicle navigation system 2 further comprises a computing unit 3 which can apply an artificial neural network 6 to input data, in particular camera images 7 (see e.g. Figure 2 ) and ultrasound Figure 9 (see e.g. Figure 2 and Figure 5 ) to perform one or more predefined visual perception tasks. The computing unit 3 can then generate at least one control signal for one or more actuators (not shown) of the vehicle 1 to at least partially automatically guide the vehicle 1 based on the corresponding results or outputs of the one or more visual perception tasks.

[0099] In particular, an electronic vehicle navigation system may perform an exemplary embodiment of the method for automatic visual perception according to the invention.

[0100] Figure 2 A schematic block diagram representing such a method and a trained artificial neural network 6 is shown.

[0101] According to the method, the camera 4 generates a camera image 7 representing the environment of the vehicle 1, and an ultrasonic sensor system (e.g., ultrasonic sensors 5a, 5b, 5c, 5d, 5e) transmits one or more ultrasonic pulses into the environment and detects reflected portions of the transmitted ultrasonic pulses, and generates at least one ultrasonic sensor signal 8 based on the detected reflected portions.

[0102] The computing unit 3 generates a spatial ultrasound map 9 in a top-down perspective from the at least one ultrasound sensor signal, wherein the ultrasound map 9 comprises a two-dimensional amplitude distribution in a top-down perspective. To this end, the computing unit 3 may apply a software module 10 (which may also be stored on a storage device) to the at least one ultrasound sensor signal 8. For further details, reference is made further below to Figure 5 explanation.

[0103] The computation unit 3 applies the neural network 6 to input data comprising the ultrasound image 9 and the camera image 7. In particular, at least one first feature map 17 is generated by applying the first encoder module 11 of the neural network 6 to the camera image 7. The top view transformation module 13 of the neural network 6 is applied to the at least one first feature map 17 to transform the at least one first feature map 17 from a camera image plane perspective to a top view perspective. For further details, refer to the following description of Figure 3 and Figure 4 explanation.

[0104] At least one second feature map 22 is generated by applying the second encoder module 12 of the neural network 6 to the ultrasound map 9. By fusing the transformed at least one first feature map 20 (see e.g. Figure 4 ) and at least one second feature map 22, in particular, by cascading at least one first feature map 20 and at least one second feature map 22 after transformation to generate a fused feature map set. One or more visual perception tasks (in Figure 2 In this example, three visual perception tasks) are performed by one or more corresponding decoder modules 15a, 15b, 15c of the neural network 6 (in Figure 2 In this example, the first decoder module 15a, the second decoder module 15b and the third decoder module 15c) are executed according to the fused feature atlas.

[0105] For example, an intermediate feature atlas 23 can be generated by applying the top-down network module 14 of the neural network 6 to the fused feature atlas. Visual perception tasks can then be performed by applying corresponding decoder modules 15a, 15b, 15c to the intermediate feature atlas 23. In particular, a first visual perception task (e.g., an object height regression task) can be performed by applying the first decoder module 15a to the intermediate feature atlas 23, a second visual perception task (e.g., a semantic segmentation task) can be performed by applying the second decoder module 15b to the intermediate feature atlas 23, and a third visual perception task (e.g., a bounding box detection task) can be performed by applying the third decoder module 15c to the intermediate feature atlas 23.

[0106] Typically, the encoder module of a neural network may include a series of convolutional layers for deep feature extraction. Roughly speaking, the deeper the encoder, the better or more detailed the features. However, the type of encoder may be chosen based on the constraints of the embedded platform. For example, a standard encoder family such as ResNet, VGG, Inception may be employed.

[0107] The ultrasonic map 9 is, for example, a single-channel top view of the surroundings in the vicinity of the vehicle 1. For example, it can be given on a grid with a size of the meter level (e.g. 6m×12m) and a cell side length of the centimeter level (e.g. 1cm). In this way, the maximum detection range of the ultrasonic sensors 5a, 5b, 5c, 5d, 5e can also be covered by taking into account the position of the ultrasonic sensors 5a, 5b, 5c, 5d, 5e relative to the camera 4, which is, for example, about 5m, and the position defines the center of the coordinate system by projection on the ground. Therefore, the ultrasonic map 9 can be quite large and relatively sparse, that is, only small areas have high amplitudes. Therefore, the second encoder module 12 is used to input the ultrasonic map 9 into a feature space, in which the ultrasonic map 9 can be matched with at least one first feature map 20 after transformation obtained from the camera image 7.

[0108] In particular, first, the ultrasound image 9 is converted into at least one second feature map by passing through the convolution layer of the second encoder module 12, so that the ultrasound image 9 outputs features of the same size (i.e., height and width) as the output of the top view transform module 13. Then, the feature maps are stacked along the depth or channel direction and pass through the convolution layer of the top-down network module 14. In particular, the second encoder module 12 transforms the sparse ultrasound information contained in the ultrasound image 9 into dense information in the top view perspective, i.e., at least the second feature map, which is then processed by the top-down network module 14, for example.

[0109] As in Figure 3 As shown in more detail in the exemplary embodiment of , the first encoder module 11 may include a sequence of convolutional layers 16a, 16b, 16c, 16d, 16e with different output scales, and may involve regularization techniques. It may be designed as or include a feature pyramid network. The first encoder module 11 may take the camera image 7 as input and generate corresponding first feature maps 17a, 17b, 17c, 17d, 17e at different scales. For example, in Figure 3 , the first encoder module 11 generates five first feature maps 17a, 17b, 17c, 17d, 17e having scaling factors of 1 / 8, 1 / 16, 1 / 32, 1 / 64 and 1 / 128 respectively.

[0110] The top view transformation module 13 may be implemented, for example, as a dense transformation pyramid, such as Figure 4 It converts the corresponding first feature map 17 into a top view perspective, or in other words, into a bird's eye view feature map. The input and output coordinate systems can be completely different from each other.

[0111] like Figure 4As shown, the transformation layer transforms the first feature map 17 of size H×W with C channels into a transformed first feature map 20 in a top view plane with size Z×x and C channels. In the transformation layer, a relationship between the complete horizontal and specific vertical lengths of the camera image 7 and the top view map is first created. Then, in an intermediate stage 18, the vertical dimension H and channel dimension C of the first feature map 20 are collapsed to a bottleneck of size B, while the vertical dimension W remains unchanged. Then, in another intermediate stage 19, a 1D convolution is applied along the horizontal axis to reshape the resulting feature map to a size Z×X with C channels. This process produces a trapezoidal shape in the orthogonal top view space due to perspective differences. Therefore, it is converted to a Cartesian coordinate system by resampling using the known camera focal length and known horizontal offset of the camera 4.

[0112] The top-down network module 14 processes the transformed at least one first feature map and at least one second feature map. The output of the top-down network module 14 is processed by a plurality of decoder modules 15a, 15b, 15c for their respective purposes, such as for height regression, semantic segmentation, and generating bounding boxes, respectively.

[0113] The at least one ultrasound sensor signal 8 may be generated from a time series of measurements and thus represent ultrasound echo amplitudes recorded over a fixed duration.

[0114] Typically, a peak in at least one ultrasonic sensor signal 8 results from an object in the environment of the vehicle 1 reflecting an ultrasonic pulse emitted from one ultrasonic sensor 5a, 5b, 5c, 5d, 5e back to another or the same ultrasonic sensor 5a, 5b, 5c, 5d, 5e. Thus, the distance the ultrasonic pulse had traveled before it was reflected can be calculated, wherein, for example, the ambient temperature can be taken into account to determine the exact speed of sound.

[0115] In order to transform the 1D amplitude data as a function of time into the spatial domain, the distance of the echo can be calculated, taking into account in addition the unknown of the angular position of the object reflecting the echo. It can be located on the longitudinal sensor axis of the ultrasonic sensors 5a, 5b, 5c, 5d, 5e, but due to their larger size, it can also be offset by a large angle from the longitudinal sensor axis due to their larger field of view. As long as the object is positioned so that it reflects back to the sensor before the echo amplitude drops to a level that is no longer distinguishable from random noise, up to 70° may be reasonable.

[0116] like Figure 5, the software module 10 can calculate an ultrasound map 9 which represents the environment of the vehicle 1 in a top-down perspective, in particular in a vehicle coordinate system, wherein the center of the rear axle of the vehicle 1 is located at the origin of the coordinate system. The respective positions and orientations of the ultrasound sensors 5a, 5b, 5c, 5d, 5e are predetermined and known.

[0117] Then, a grid may be generated, the size of which is approximately equal to the field of view of the ultrasonic sensor 5a, 5b, 5c, 5d, 5e, and the grid cell size is small enough to provide a sufficiently high resolution and still be comparable to the distance resolution based on the at least one ultrasonic sensor signal 8. For example, a quadratic grid cell with a side length of 1 cm may be used. The distance of each grid cell to the transmitting ultrasonic sensor 5a, 5b, 5c, 5d, 5e and the receiving ultrasonic sensor 5a, 5b, 5c, 5d, 5e may be calculated for each of the at least one ultrasonic sensor signal 8 and for each time step.

[0118] The angular distribution 21 represents the object likelihood based on the angle of the object with the longitudinal sensor axis, which is multiplied by the corresponding amplitude as a function of radial distance. The angular distribution 21 can be considered as a scaling factor of the amplitude. For example, the angular distribution 21 can be used as Figure 5 . In the case where the transmitting and receiving ultrasonic sensors 5a, 5b, 5c, 5d, 5e are not identical, which is also represented as an indirect signal path, the minimum of the two calculated scaling factors can be used. However, it is also generally conceivable to use a multiplication of two factors or a square root of two factors. The result is an intermediate sonogram 9' for each of the at least one ultrasonic sensor signal 8. These can be summed to obtain the sonogram 9.

[0119] As described, particularly with respect to the accompanying drawings, in several embodiments, the present invention allows for CNN-based low-level object detection, semantic segmentation, and height regression in the bird's-eye view domain using camera images (particularly fisheye camera images) and ultrasound data.

Claims

1. A method for automatic visual perception of a vehicle (1), wherein: - generating a camera image (7) representing an environment of the vehicle (1) by a camera (4) of the vehicle (1), and generating at least one first feature map (17, 17a, 17b, 17c, 17d, 17e) by applying a first encoder module (11) of a trained artificial neural network (6) to the camera image (7); - applying a top view transformation module (13) of the neural network (6) to the at least one first feature map (17, 17a, 17b, 17c, 17d, 17e) to transform the at least one first feature map (17, 17a, 17b, 17c, 17d, 17e) from a camera image plane perspective to a top view perspective; - an ultrasonic sensor system (5a, 5b, 5c, 5d, 5e) of the vehicle (1) emits ultrasonic pulses into the environment and the ultrasonic sensor system (5a, 5b, 5c, 5d, 5e) generates at least one ultrasonic sensor signal (8) based on a reflected portion of the emitted ultrasonic pulses; - generating a spatial ultrasound map (9) in the top-down perspective based on the at least one ultrasound sensor signal (8); - generating at least one second feature map by applying a second encoder module (12) of the neural network (6) to the sonogram (9); - generating a fused feature atlas by fusing the transformed at least one first feature map (20) and the at least one second feature map (22); and -Performing a first visual perception task by a first decoder module (15a, 15b, 15c) of a neural network (6) based on the fused feature atlas.

2. The method according to claim 1, It is characterized in that - generating an intermediate feature atlas by applying a top-down network module (14) of the neural network (6) to the fused feature atlas; and -Performing a first visual perception task by applying a first decoder module (15a, 15b, 15c) to the set of intermediate feature maps.

3. The method according to claim 2, It is characterized in that A second visual perception task is performed by a second decoder module (15a, 15b, 15c) of the neural network (6) based on the fused feature atlas.

4. The method according to claim 3, It is characterized in that A second visual perception task is performed by applying a second decoder module (15a, 15b, 15c) to the set of intermediate feature maps.

5. The method according to claim 3 or 4, It is characterized in that - the first visual perception task is an object height regression task, and the output of the first decoder module (15a, 15b, 15c) comprises a height map in a top-down view perspective, the height map containing predicted object heights of one or more objects in the environment; and / or - the second visual perception task is a semantic segmentation task, and the output of the second decoder module (15a, 15b, 15c) comprises a semantic segmentation image in a top-down perspective; and / or - performing a third visual perception task based on the fused feature atlas by a third decoder module (15a, 15b, 15c) of the neural network (6), wherein the third visual perception task is a bounding box detection task, and the output of the third decoder module (15a, 15b, 15c) includes the corresponding position and size of at least one bounding box of one or more objects in the environment in the top view perspective.

6. The method according to one of claims 1 to 4, It is characterized in that - the first visual perception task is an object height regression task, and the output of the first decoder module (15a, 15b, 15c) comprises a height map in a top-down view perspective, the height map containing predicted object heights of one or more objects in the environment; or - the first visual perception task is a semantic segmentation task, and the output of the first decoder module (15a, 15b, 15c) comprises a semantically segmented image in a top-down perspective; or - The first visual perception task is a bounding box detection task, and the output of the first decoder module (15a, 15b, 15c) comprises the corresponding position and size of at least one bounding box in a top-down perspective of at least one object in the environment.

7. The method according to one of the preceding claims, It is characterized in that - the first encoder module (11) comprises at least two encoder branches (16a, 16b, 16c, 16d, 16e); -By applying the first encoder module (11) to the camera image (7), each of the at least two encoder branches (16a, 16b, 16c, 16d, 16e) generates a corresponding first feature map of the at least one first feature map (17, 17a, 17b, 17c, 17d, 17e), the size of which is reduced relative to the size of the camera image (7) according to a predefined scaling factor of the corresponding encoder branch (16a, 16b, 16c, 16d, 16e).

8. The method according to one of claims 1 to 6, It is characterized in that The one or more first feature maps (17, 17a, 17b, 17c, 17d, 17e) include at least two first feature maps (17, 17a, 17b, 17c, 17d, 17e), and the sizes of the at least two first feature maps are reduced relative to the size of the camera image (7) according to different predefined scaling factors.

9. The method according to one of the preceding claims, It is characterized in that Fusing the transformed at least one first feature map (20) and the at least one second feature map comprises cascading the transformed at least one first feature map (20) and the at least one second feature map.

10. The method according to one of the preceding claims, It is characterized in that The top view transformation module (13) comprises a transformer pyramid network.

11. The method according to one of the preceding claims, It is characterized in that - for each of the at least one ultrasonic sensor signal (8), converting the amplitude of the respective ultrasonic sensor signal (8) as a function of time into an amplitude as a function of radial distance from the ultrasonic sensor system (5a, 5b, 5c, 5d, 5e); - for each of said at least one ultrasonic sensor signal (8), a distribution amplitude is calculated as the product of said amplitude as a function of said radial distance and a corresponding predetermined angular distribution (21); and - Generating said ultrasound map (9) comprises summing said distribution amplitudes.

12. The method according to claim 11, It is characterized in that The angular distribution is given by at least one beta distribution.

13. An electronic vehicle navigation system (2) for a vehicle (1), comprising a camera (4), a storage device storing a trained artificial neural network (6), at least one computing unit (3) and an ultrasonic sensor system (5a, 5b, 5c, 5d, 5e), wherein: - the camera (4) is configured to generate a camera image (7) representing the environment of the vehicle (1); - the at least one computing unit (3) is configured to generate at least one first feature map (17, 17a, 17b, 17c, 17d, 17e) by applying a first encoder module (11) of the neural network (6) to the camera image (7); - the at least one computational unit (3) is configured to transform the at least one first feature map (17, 17a, 17b, 17c, 17d, 17e) from a camera image plane perspective to a top view perspective by applying a top view transformation module (13) of the neural network (6) to the at least one first feature map (17, 17a, 17b, 17c, 17d, 17e); - the ultrasonic sensor system (5a, 5b, 5c, 5d, 5e) is configured to transmit ultrasonic pulses and to generate at least one ultrasonic sensor signal (8) based on reflected parts of the transmitted ultrasonic pulses; - the at least one computing unit (3) is configured to generate a spatial ultrasound map (9) in the top-down perspective from the at least one ultrasound sensor signal (8), and to generate at least one second feature map by applying a second encoder module (12) of the neural network (6) to the ultrasound map (9); - the at least one computing unit (3) is configured to generate a fused feature atlas by fusing the transformed at least one first feature map (20) and the at least one second feature map, and to perform a first visual perception task based on the fused feature atlas by using a first decoder module (15a, 15b, 15c) of the neural network (6); and The at least one computing unit (3) is configured to generate at least one control signal for at least partially automatically guiding the vehicle (1) as a function of the result of the first visual perception task.

14. A vehicle (1) comprising an electronic vehicle navigation system (2) according to claim 13, wherein: The camera (4) and the ultrasonic sensor system (5a, 5b, 5c, 5d, 5e) are installed at the vehicle (1).

15. A computer program product comprising instructions which, when executed by an electronic vehicle navigation system (2) according to claim 13, cause the electronic vehicle navigation system (2) to perform a method according to any one of claims 1 to 12.

Images