Parking slot detection

Abstract

According to a method for parking slot detection, camera image (8) depicting an environment of a motor vehicle (1) is received from a camera (4) mounted to a motor vehicle (1), at least one ultrasonic sensor signal (9) is received from at least one ultrasonic detector (5, 6) mounted to the motor vehicle (1), and a two-dimensional ultrasonic map (29) in a top view perspective is generated depending on the at least one ultrasonic sensor signal (9). Parking slot proposal data comprising a position of a proposed parking slot (18) in the environment, which is equipped with an infrastructure device (22) of a predefined device type, is generated by applying a trained artificial neural network, ANN, (7) to input data, which depends on the camera image (8) and the ultrasonic map (29).

Description

[0001] Parking slot detection

[0002] The present invention is directed to a method for parking slot detection, wherein a camera image depicting an environment of a motor vehicle is received from a camera mounted to a motor vehicle. The invention is further directed to a method for parking a motor vehicle, wherein such a method for parking slot detection is carried out. The invention is also directed to a data processing apparatus for carrying out such a method for parking slot detection, to an electronic vehicle guidance system and to a computer program product.

[0003] In a parking lot, the task of finding an available parking slot, which is equipped with a specific infrastructure device, may require drivers of motor vehicles to engage in a laborious manual search. Such an infrastructure device may for example be a charging station for electric vehicles, a roof of the parking slot, a camera surveillance system of the parking slot, et cetera.

[0004] Document US 2023 / 0005368 A1 describes a method for automatically learning parking preferences for a driver, including for example parking close to a charging station, and generating parking recommendations. Training data based on trajectory data indicating a trajectory of a vehicle during a plurality of parking events, each in which the driver selects a parking candidate from among a plurality of parking candidates, and attribute data indicating attributes of each of the plurality of parking candidates, are generated. Then, a decision model is trained based the training data.

[0005] It is known to use object detection algorithms based on artificial neural networks for detecting, localizing and / or characterizing objects in the environment of a motor vehicle using camera images from the motor vehicle’s cameras. However, for certain environmental conditions, the reliable detection and characterization of objects is a difficult task for known algorithms, which results in a reduced reliability and / or accuracy of the corresponding output. For example, such environmental conditions comprise low light scenarios such as twilight or nighttime scenarios or, in particular, scenarios in indoor parking lots, where the information contained in the camera images is very limited. Furthermore, the detection and characterization of objects in the environment of the motor vehicle, which have a particular low height from the ground, such as curbs, low walls, or poles may be particularly demanding. This holds even more in case such objects should be characterized and detected at low light scenarios.

[0006] In the publication T. Roddick and R. Cipolla “Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks", 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020 or the corresponding preprint arXiv:2003.13402v1 ("Roddick and Cipolla" in the following) the authors describe an approach for estimating bird’s eye view maps of the environment of autonomous vehicles directly from monocular images using a single end-to-end deep learning architecture. The architecture consists of a backbone network, a feature pyramid network, a bird’s eye view transformation network and a top-down network. The backbone network, which may, for example be based on ResNet-50, extracts image features at multiple resolutions from the input image and the feature pyramid network augments the high resolution features with spatial context from lower pyramid layers. A stack of dense transform layers of the bird’s eye view transformation network maps the image-based features into the bird’s eye view and the top-down network processes the bird’s eye view features and predicts the final semantic occupancy probabilities.

[0007] It is an objective of the present invention to provide a possibility for automatic detection of a parking slot, which is equipped with a predefined infrastructure device, in particular with an increased reliability and / or accuracy.

[0008] This objective is achieved by the subject matter of the independent claim. Further implementations and preferred embodiments are subject matter of the dependent claims.

[0009] The invention is based on the idea to use a trained artificial neural network to propose a parking slot, which is equipped with an infrastructure device of a predefined type, based on a camera image and a two-dimensional ultrasonic map in a top view perspective, which is generated based on respective ultrasonic sensor signals of ultrasonic detectors of the motor vehicle.

[0010] According to an aspect of the invention, a method for parking slot detection is provided. Therein, a camera image depicting an environment of a motor vehicle is received from a camera, in particular a visible range camera, mounted to a motor vehicle. At least one ultrasonic sensor signal is received from at least one ultrasonic detector mounted to the motor vehicle. A two-dimensional ultrasonic map in a top view perspective is generated depending on the at least one ultrasonic sensor signal. Parking slot proposal data comprising a position of a proposed parking slot in the environment, which is equipped with an infrastructure device of a predefined device type, is generated by applying a trained artificial neural network, ANN, to input data, which depends on the camera image and the ultrasonic map. In particular, the input data may comprise the camera image and the ultrasonic map or consist of the camera image and the ultrasonic map.

[0011] The method according to the invention may be purely computer implemented. Unless stated otherwise, all steps of such a computer-implemented method may be performed by a data processing apparatus, which comprises at least one computing unit, in particular a data processing apparatus of the motor vehicle. In particular, the at least one computing unit is configured or adapted to perform the steps of the computer-implemented method. For this purpose, the at least one computing unit may for example store a computer program comprising instructions which, when executed by the at least one computing unit, cause the at least one computing unit to execute the computer-implemented method.

[0012] All computing units of the at least one computing unit may be comprised by the motor vehicle. However, it is also possible that all computing units of the at least one computing unit are part of an external computing system external to the motor vehicle, for example a backend server or a cloud computing system. It is also possible that the at least one computing unit comprises at least one vehicle computing unit of the motor vehicle as well as at least one external computing unit comprised by the external computing system. The at least one vehicle computing unit may for example be comprised by one or more electronic control units, ECUs, and / or one or more zone control units, ZCUs, and / or one or more domain control units, DCUs, of the vehicle and / or the camera.

[0013] For each implementation of a method for parking slot detection according to the invention, which is purely computer implemented, further implementations of the method, which are not purely computer implemented, are obtained by including respective method steps for generating the camera image by means of the camera and / or for generating the at least one ultrasonic sensor signal by means of the at least one ultrasonic detector.

[0014] The position of the proposed parking slot may be provided in different forms. For example, the position may be given in terms of coordinates of one or more characteristic points of the proposed parking slot, for example a center point or corner points, in a predefined reference coordinate system, for example in an image coordinate system according to the camera image or the camera, respectively. The position may also be given in terms of a geometric figure, for example a polygon or a polyline, approximating the proposed parking slot, in particular in the predefined reference coordinate system.

[0015] The infrastructure device of the predefined device type may vary for different implementations of the method. For example, the infrastructure device is located on the proposed parking slot or at a boundary of the proposed parking slot or above the proposed parking slot or in an otherwise predefined vicinity of the proposed parking slot. For example, the predefined device type may correspond to a roof of the respective parking slot or a camera surveillance system of the respective parking slot. In preferred embodiments, the predefined device type corresponds to a charging station for electric vehicles.

[0016] For example, the trained artificial neural network, ANN, may be provided in a computer readable way, for example stored on the at least one computing unit. The ANN may for example be designed as a convolutional neural network, CNN, or comprise one or more CNN modules. The training of the ANN may be carried out using conventional methods for training neural networks, in particular CNNs, such as supervised training approaches. The respective training data comprise, in particular training images depicting one or more parking slots, wherein one or more of them are equipped with an infrastructure device of the predefined device type as well as respective annotations for the positions of those parking slots. The training data may also comprise respective ultrasonic sensor signals from ultrasonic detectors. The training data does not necessarily contain annotations for the ultrasonic sensor signals.

[0017] For generating the ultrasonic map depending on the at least one ultrasonic sensor signal, a conversion module, which is a software module, may be applied to the at least one ultrasonic sensor signal, wherein the conversion module is not necessarily a part of the ANN. In particular, the conversion module is not necessarily a trained or trainable software module. However, in other implementations, the conversion module may also be a trainable or trained module of the ANN.

[0018] The camera image is generated and provided in a camera image plane. The camera image plane is, for example, perpendicular to a predefined longitudinal direction of a camera coordinate system, which may for example be parallel to an optical axis of the camera. The top view perspective corresponds to a perspective according to a top view plane, which is perpendicular to a predefined height axis. The height axis may for example be perpendicular to a road surface of a road on which the motor vehicle is positioned or, in other words, parallel to a vehicle height axis of the motor vehicle. This may in some cases be parallel to a further coordinate axis of the camera coordinate system. In general, however, the camera coordinate system may also be tilted or rotated.

[0019] For example, the at least one ultrasonic sensor signal may be generated by the at least one ultrasonic detector based on reflected portions of ultrasonic pulses emitted into the environment by at least one ultrasonic emitter of the motor vehicle.

[0020] All of the ultrasonic pulses may for example be emitted by the at least one ultrasonic emitter according to a predefined wavelength spectrum of the involved ultrasonic waves or in other words, according to a predefined emission band. In particular, all ultrasonic emitters of the at least one ultrasonic emitter operate with the same emission band. Analogously, all detectors of the at least one ultrasonic detector may be adapted to detect ultrasonic waves according to the same predefined detection band, wherein the detection band matches the emission band at least partly. In other words, all of the ultrasonic detectors are in principle able to detect ultrasonic waves generated by any of the ultrasonic emitters.

[0021] This does, however, not exclude that the motor vehicle comprises further ultrasonic emitters operating at different emission bands and corresponding further ultrasonic detectors. In this case, further ultrasonic sensor signals may be generated by the further ultrasonic detectors based on reflected portions of ultrasonic pulses emitted by the further ultrasonic emitters. The ultrasonic map may in this case be generated depending also on the further ultrasonic sensor signals. In the following, as long as not stated otherwise, the discussion is limited to the at least one ultrasonic emitter and the at least one ultrasonic detector operating in said matching emission band and detection band, respectively. However, all of the explanations may be carried over to the further ultrasonic emitters and the further ultrasonic detectors accordingly.

[0022] It is further noted that each of the at least one ultrasonic emitter may emit respective ultrasonic pulses repeatedly, in particular in a periodic manner, and the at least one ultrasonic sensor signal may be considered as at least one time series of measurements. For example, each ultrasonic sensor signal may be considered to represent an envelope of an ultrasonic wave corresponding to the reflected portions of the corresponding emitted ultrasonic pulse. Each ultrasonic sensor signal is then given by an amplitude of the respective envelope as a function of time. Since the speed of sound is known or can be estimated, in particular for a given air temperature or an estimated air temperature in the environment, the amplitude as a function of time may be directly converted into an amplitude as a function of a travel distance, which corresponds to a sum of distances from the respective ultrasonic emitter to a reflecting object in the environment and back to the corresponding ultrasonic detector.

[0023] The at least one ultrasonic emitter and the at least one ultrasonic detector may be combined as ultrasonic transceivers, or they may be implemented separately from each other. In particular, reflected portions of an ultrasonic pulse emitted by an ultrasonic transceiver may be detected by the same ultrasonic transceiver, which is denoted as a direct signal path, or by another ultrasonic transceiver, which is denoted as an indirect signal path.

[0024] In particular, a field of view of the camera may partially overlap with a field of view of the at least one ultrasonic detector and the at least one ultrasonic emitter, respectively. Consequently, the camera image and the at least one ultrasonic sensor signal represent at least in part the same spatial region in the environment of the motor vehicle.

[0025] The ultrasonic map may be understood as an ultrasonic image, for example. In particular, the ultrasonic map may be given by a plurality of grid values, each grid value corresponding to a respective grid cell of a predefined spatial grid in the top view perspective.

[0026] Each grid cell corresponds then to a respective pixel in the ultrasonic map, wherein the grid value may be considered as the respective pixel value. Consequently, the ultrasonic map may be treated analogously as known for camera images by the ANN, for example by passing it through one or more convolutional layers. Even though the content of the ultrasonic map may not be immediately interpretable for a human, the trained ANN is able to automatically interpret the encoded information similar as for camera images.

[0027] Since the parking slot proposal data is generated based on both, the ultrasonic map as well as the camera image, the reliability of detecting the parking slot and its associated infrastructure device of the predefined device type may be significantly improved, in particular for low light scenarios and / or adverse weather conditions. In particular, the ultrasonic sensor signals represent reflecting objects in the environment of the motor vehicle widely independently of the lighting conditions and / or weather conditions. Thus, the combined use of the camera image, which contains semantic information regarding the depicted objects, and the ultrasonic sensor signals achieves a particularly reliable and accurate automatic detection of a parking slot, which is equipped with the infrastructure device.

[0028] According to several implementations, a respective parking slot feature map is generated for at least one parking slot in the environment by applying a parking slot detection module of the ANN to first input data, which is for example a first part of the input data, which depends on the camera image and is, in particular, independent of the at least one ultrasonic sensor signal. The parking slot feature map comprises a position of the respective parking slot. The parking slot proposal data is generated by applying a decoder module of the ANN to intermediate data, which comprises the generated parking slot feature maps for each of the at least one parking slot, wherein the proposed parking slot is one of the at least one parking slot.

[0029] The parking slot detection module may comprise, for example, one or more convolutional layers, for example 2D-convolution layers. The parking slot detection module may comprise one or more further layers, for example one or more activation layers implementing an activation function, for example a ReLU activation function. For example, the parking slot detection module may comprise at least two convolution layers, wherein one of the at least one activation layers is arranged between each pair of subsequent convolution layers. The parking slot detection module may for example be based on an architecture of the YOLO decoders, for example YOLOv3.

[0030] The first input data may correspond to the camera image. The first input data may also be generated based on the camera image. For example, at least one image feature map may be generated by applying an encoder module of the ANN to the camera image. The first input data may correspond to the at least one image feature map.

[0031] The encoder module comprises, for example, a series of convolutional layers for image feature extraction. The architecture of the encoder module can be based, for example, on standard encoder families, such as ResNet, VGG, Inception and so forth.

[0032] The decoder module may comprise, for example, one or more deconvolutional layers and may be based on known architectures, such as the YOLO decoders, for example YOLOv3. The parking slot proposal data comprises the position of the proposed parking slot and, for example, at least one respective confidence value. In case more than one of the at least one parking slots are found to be equipped with an infrastructure device of the predefined device type, the proposed parking slot may be selected according to a predefined selection rule. The selection rule may include a random selection in some implementations. The selection rule may also take into account a distance of the position of the respective parking slot from the motor vehicle, the least one confidence value, a size of the respective parking slot, and so forth.

[0033] Automatically locating a parking slot, in particular an unoccupied parking slot, may be challenging, in particular in real-time scenarios and / or in crowded parking lots. The parking slot detection module may leverage encoded features extracted from the camera image to effectively identify available parking slots by regressing, in particular, their positions.

[0034] According to several implementations, the parking slot feature map comprises an occupancy status of the respective parking slot for each of the at least one parking slot. The occupancy status of the proposed parking slot indicates that the proposed parking slot is unoccupied.

[0035] According to several implementations, the parking slot feature map comprises a position of a center point, also denoted as center position, of the respective parking slot and / or a respective position of one or more boundary points on a boundary of the respective parking slot.

[0036] The positions of the center point and / or the one or more boundary points are for example given in image coordinates of the camera image.

[0037] For example, the parking slot detection module may regress the center position and one or more, for example four, corner points on the boundary of each parking slot. The boundary may for example be defined by painted markings on the road surface, curbs, barriers, et cetera. Each parking slot may for example be assumed to be defined by a rectangular boundary or an approximately rectangular boundary on the road surface. In the image coordinates, the rectangle is, in general, distorted. Nevertheless, the center point and / or the corner points of the respective rectangle or distorted rectangle, respectively, may be regressed and represent the position of the respective parking slot. According to several implementations a respective infrastructure device feature map is generated for at least one infrastructure device of the predefined device type in the environment by applying an infrastructure device detection module of the ANN to the first input data. The infrastructure device feature map comprises a position of the respective infrastructure device. The intermediate data comprises the generated infrastructure device feature maps for each of the at least one infrastructure device.

[0038] The infrastructure device detection module may comprise, for example, one or more convolutional layers, for example 2D-convolution layers. The infrastructure device detection module may comprise one or more further layers, for example one or more activation layers implementing an activation function, for example a ReLU activation function. For example, the infrastructure device detection module may comprise at least two convolution layers, wherein one of the at least one activation layers is arranged between each pair of subsequent convolution layers. The infrastructure device detection module may for example be based on an architecture of the YOLO decoders, for example YOLOv3.

[0039] The infrastructure device detection module may leverage encoded features extracted from the camera image to effectively identify infrastructure devices of the predefined type by regressing, in particular, their positions. The infrastructure device detection module may also classify the infrastructure devices to distinguish infrastructure devices of the predefined devcie type from others.

[0040] According to several implementations, the infrastructure device feature map comprises a position of a center point of the respective infrastructure device and / or a respective position of one or more boundary points on a boundary of the respective infrastructure device.

[0041] The positions of the center point and / or the one or more boundary points are for example given in image coordinates of the camera image.

[0042] For example, the infrastructure device detection module may regress the center position and one or more, for example four, corner points on the boundary of each infrastructure device. Each infrastructure device may for example be assumed to be defined or confined by a rectangular boundary or an approximately rectangular boundary, in particular in a plane perpendicular to the road surface. In the image coordinates, the rectangle is, in general, distorted. Nevertheless, the center point and / or the corner points of the respective rectangle or distorted rectangle, respectively, may be regressed and represent the position of the respective infrastructure device.

[0043] According to several implementations, at least one ultrasonic feature map is generated by applying an ultrasonic feature encoder module of the ANN to second input data, which depends on the ultrasonic map or corresponds to the ultrasonic map and is, in particular, independent of the camera image. The intermediate data depends on the at least one ultrasonic feature map or the intermediate data comprises the at least one ultrasonic feature map.

[0044] The ultrasonic feature encoder module comprises, for example, a series of convolutional layers for image feature extraction. The architecture of the ultrasonic feature encoder module can be based, for example, on standard encoder families, such as ResNet, VGG, Inception and so forth.

[0045] According to several implementations, an ultrasonic map transformation module is applied to the ultrasonic map to transform the ultrasonic map from the top view perspective into a camera image plane perspective of the camera. The second input data comprises the transformed feature map.

[0046] The ultrasonic map transformation module may be a non-trainable pre-processing module and for example not be a part of the ANN.

[0047] In order to transform the ultrasonic map from the top view perspective into the camera image plane perspective, a predefined camera model for the camera may be used. Corresponding functions are known in image processing. For example, corresponding functions of the open CV library, like for example the Fisheye function may be used. Corresponding models for various types of fisheye cameras or pinhole cameras or other types of cameras are available.

[0048] According to several implementations, a feature transformation module, in particular a feature transformation module of the ANN, is applied to the at least one ultrasonic feature map to transform the at least one ultrasonic feature map from the top view perspective into a camera image plane perspective of the camera. The intermediate data comprises the transformed at least one ultrasonic feature map. In this way, it is achieved that the transformed at least one ultrasonic feature map and the features extracted from the camera image, for example the parking slot feature map and / or the infrastructure device feature map, are given in the same perspective, which may improve the performance or training efficiency of the ANN.

[0049] The feature transformation module may for example be designed as described in the publication of Roddick and Cipolla with respect to the multi-scale dense transformers or stack of dense transformer layers, respectively.

[0050] According to several implementations, the decoder module is applied to the intermediate data, which comprises the at least one ultrasonic feature map, the infrastructure device feature maps, and the parking slot feature maps.

[0051] For example, fused features may be generated by fusing, for example concatenating, the at least one ultrasonic feature map, the infrastructure device feature maps, and the parking slot feature maps. The decoder module is then for example applied to the fused features.

[0052] According to several implementations, the ultrasonic map is given by the plurality of grid values, each grid value corresponding to a respective grid cell of a predefined spatial grid in the top view perspective, wherein for each of the grid cells and for each emitter-detector pair of the at least one ultrasonic emitter and the at least one ultrasonic detector, a corresponding travel distance is computed and a signal value is computed depending on the travel distance. The grid value of the respective grid cell is computed depending on the signal value.

[0053] Therein, the travel distance is, in particular, a travel distance from a position of the ultrasonic emitter of the emitter-detector pair via a position of the grid cell to a position of the ultrasonic detector of the emitter-detector pair. The signal value is a signal value of the ultrasonic sensor signal, which is generated by the ultrasonic detector of the emitterdetector pair.

[0054] The emitter-detector pairs can be understood such that each ultrasonic detector of the at least one ultrasonic detector forms an emitter-detector pair with any of the at least one ultrasonic emitter, irrespective of whether they form a common ultrasonic transceiver. For n ultrasonic emitters and m ultrasonic detectors, n*m emitter-detector pairs result. The grid is, in particular, a two-dimensional grid. For example, the grid cells may be arranged in an array of rows and columns and, consequently, may be considered as a Cartesian or rectangular grid. However, also other implementations are possible, for example using a polar grid, wherein each grid cell is defined by an interval of a radial distance and an angular interval.

[0055] In particular, as described above, each value of the ultrasonic sensor signals corresponds to a certain travel time and, consequently, to a certain travel distance. Thus, the ultrasonic sensor signal can be evaluated at the computed travel distance from the position of the ultrasonic emitter to the grid cell back to the ultrasonic detector. Since the resolution of the ultrasonic sensor signals is finite, an interpolation of the respective values may be performed to evaluate the ultrasonic sensor signal at the computed travel distance.

[0056] It is noted that the spatial grid in the top view perspective is defined in a real-world coordinate system, for example a coordinate system of the camera or the motor vehicle. Therefore, the signal value computed depending on the travel distance can be understood to indicate the presence or absence of an object at a position in the environment of the vehicle given by the grid cell.

[0057] The described steps being carried out for each grid cell and each emitter-detector pair may be understood such that a particular grid cell is selected and a particular ultrasonic detector of the at least one ultrasonic detector is selected. The travel distance is then computed for the selected grid cell and the selected ultrasonic detector for each of the at least one ultrasonic emitter and then the respective signal value is computed. These steps are then repeated for the same grid cell and all other ultrasonic detectors of the at least one ultrasonic detector. These steps are then again repeated for all other grid cells.

[0058] In case further ultrasonic emitters and further ultrasonic detectors operating at further emission and detection bands, respectively, are involved, the same steps may be carried out for them as well. However, ultrasonic detectors and ultrasonic emitters with nonmatching emission and detection bands, respectively, are not paired.

[0059] According to several implementations, a respective angular weighting function is provided for each emitter-detector pair. For each grid cell, each signal value is multiplied with the respective angular weighting function evaluated at the position of the grid cell to obtain a respective weighted signal value. For each grid cell, the grid value is computed depending on a sum of the weighted signal values obtained for the respective grid cell. For example, if, apart from the at least one ultrasonic emitter and the at least one ultrasonic detector with matching emission and detection bands, respectively, no further ultrasonic emitters and ultrasonic detectors are involved, the grid value for a given grid cell may be given by the sum of the weighted signal values as described above. On the other hand, if further ultrasonic emitters and ultrasonic detectors with different emission and detection bands, respectively, are involved, corresponding further weighted signal values may be computed analogously for each set of further ultrasonic emitters and further ultrasonic detectors with matching emission and detection bands, respectively. The grid value is then given by a sum of all of the weighted signal values and all of the further weighted signal values computed for this grid cell.

[0060] The angular weighting function describes, for example, how the amplitude of the at least one ultrasonic sensor signal typically differs for different angles, in particular polar angles in the top view perspective. Empirical, experimental or heuristic knowledge may be used to define the respective angular weighting functions. In this way, the two-dimensional information of the ultrasonic map may be obtained from the at least one ultrasonic sensor signal with improved accuracy. In general, the angular weighting functions may be different for different ultrasonic sensor signals and thus for different ultrasonic detectors. For example, the angular weighting function may depend on at least one beta-distribution.

[0061] For example, for a direct signal path, the corresponding angular weighting function may be given by a single beta-distribution centered at the corresponding ultrasonic transceiver. In case of an indirect signal path, two such beta-distributions centered at different positions according to the two different involved ultrasonic transceivers may be combined with each other to obtain the angular weighting function. For example, the minimum of both beta-distributions at the respective position may be used or an average value and so forth.

[0062] The beta-distribution may for example be given by in particular with p = q = 2 such that f2,2M ~ (l - x), with an appropriate normalization factor. Therein, x denotes a quantity, which depends on, for example is proportional to, the polar angle in the top view perspective, in particular with respect to a longitudinal axis of the corresponding ultrasonic transceiver. It has been found that in this way, the actual characteristics of ultrasonic transceivers may be modelled well.

[0063] According to several implementations, the predefined device type corresponds to a charging station for an electric vehicle and the motor vehicle is an electric vehicle.

[0064] An electric vehicle can be understood as a vehicle, which comprises a battery for being charged by means of an external device, in particular a charging station. An electric propulsion motor of the electric vehicle is supplied by the battery. Consequently, the electric vehicle may, for example, be a battery electric vehicle, BEV, or a plug-in hybrid electric vehicle, PHEV.

[0065] According to a further aspect of the invention, a method for parking a motor vehicle, in particular for parking a motor vehicle at least in part automatically, is provided. Therein, a method for parking slot detection according to the invention is carried out. At least one control signal for guiding the motor vehicle at least in part automatically to the proposed parking slot is generated, in particular by at least one computing unit of the motor vehicle, and / or visual driver assistance information indicating the proposed parking slot to a driver of the motor vehicle is displayed on a display device of the motor vehicle, in particular generated by at least one computing unit of the motor vehicle and displayed on the display device.

[0066] The at least one control signal may for example be provided to one or more actuators of the motor vehicle, including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the motor vehicle. The one or more actuators may affect a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least in part automatically to the proposed parking slot.

[0067] According to a further aspect of the invention, a data processing apparatus comprising at least one computing unit is provided. The at least one computing unit is adapted to carry out a method for parking slot detection, in particular a computer implemented method for parking slot detection, according to the invention. In the present disclosure, a computing unit may for example be understood as a data processing device with processing circuitry. A computing unit can therefore perform computing operations in order to process data. The computing operations may also include indexed accesses to a data structure, for example a look-up table, LUT.

[0068] In particular, a computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0069] A computing unit may also comprise one or more hardware and / or software interfaces and / or one or more memory units. Therein, a memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a readonly memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable readonly memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0070] According to a further aspect of the invention, an electronic vehicle guidance system comprising a data processing apparatus according to the invention is provided. The at least one computing unit is configured to generate at least one control signal for guiding the motor vehicle at least in part automatically to the proposed parking slot and / or to control a display device of the motor vehicle to display visual driver assistance information indicating the proposed parking slot to a driver of the motor vehicle.

[0071] An electronic vehicle guidance system may be understood as an electronic system, configured to guide a vehicle in a fully automated or a fully autonomous manner and, in particular, without a manual intervention or control by a driver or user of the vehicle being necessary. The vehicle carries out all required functions, such as steering maneuvers, deceleration maneuvers and / or acceleration maneuvers as well as monitoring and recording the road traffic and corresponding reactions automatically. In particular, the electronic vehicle guidance system may implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system may also be implemented as an advanced driver assistance system, ADAS, assisting a driver for partially automatic or partially autonomous driving. In particular, the electronic vehicle guidance system may implement a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated April 2021 .

[0072] Guiding the vehicle at least in part automatically may therefore comprise guiding the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. Guiding the vehicle at least in part automatically may also comprise guiding the vehicle according to a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

[0073] According to several implementations, the electronic vehicle guidance system comprises the camera and / or the at least one ultrasonic detector and / or the at least one ultrasonic emitter.

[0074] Further implementations of the electronic vehicle guidance system according to the invention follow directly from the various embodiments of the method according to the invention and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various implementations of the method according to the invention can be transferred analogously to corresponding implementations of the electronic vehicle guidance system according to the invention. In particular, the electronic vehicle guidance system according to the invention is designed or programmed to carry out a method according to the invention. In particular, the electronic vehicle guidance system according to the invention carries out a method according to the invention.

[0075] According to a further aspect of the invention, a computer program comprising instructions is provided. When the instructions are executed by a data processing apparatus, in particular by a data processing apparatus according to the invention, the instructions cause the data processing apparatus to carry out a method for parking slot detection according to the invention or a method for parking a motor vehicle according to the invention.

[0076] The instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python.

[0077] According to a further aspect of the invention, a computer readable storage medium is provided, which stores a computer program according to the invention.

[0078] The computer program and the computer readable storage medium are respective computer program products comprising the instructions.

[0079] Further features of the invention are apparent from the claims, the figures and the figure description. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of figures and / or shown in the figures may be comprised by the invention not only in the respective combination stated, but also in other combinations. In particular, embodiments and combinations of features, which do not have all the features of an originally formulated claim, may also be comprised by the invention. Moreover, embodiments and combinations of features, which go beyond or deviate from the combinations of features set forth in the recitations of the claims may be comprised by the invention.

[0080] In the following, the invention will be explained in detail with reference to specific exemplary implementations and respective schematic drawings. In the drawings, identical or functionally identical elements may be denoted by the same reference signs. The description of identical or functionally identical elements is not necessarily repeated with respect to different figures.

[0081] In the figures,

[0082] Fig. 1 shows schematically a motor vehicle with an exemplary implementation of an electronic vehicle guidance system according to the invention; Fig. 2 shows a schematic representation of a trained ANN for use in an exemplary implementation of a method for parking slot detection according to the invention;

[0083] Fig. 3 shows a schematic top view representation of parking slots in part being equipped with charging stations for electric vehicles;

[0084] Fig. 4 shows a schematic representation of a charging station for electric vehicles;

[0085] Fig. 5 shows a schematic illustration of the generation of an ultrasonic map according to a further exemplary implementation of a method for parking slot detection according to the invention;

[0086] Fig. 6 shows an illustrative example of a structure in a top view perspective; and

[0087] Fig. 7 shows an illustration of the structure of Fig. 6 transformed into a camera image plane perspective.

[0088] Fig. 1 shows an exemplary implementation of a motor vehicle 1 comprising an exemplary implementation of an electronic vehicle guidance system 2 according to the invention.

[0089] The motor vehicle 1 , for example the electronic vehicle guidance system 2, comprises a camera 4 mounted to the motor vehicle 1 , for example a front camera, a rear-facing camera or a side camera, and an ultrasonic sensor system, which contains one or more ultrasonic transceivers 5, 6. Each ultrasonic transceiver 5, 6 may comprise an ultrasonic emitter and an ultrasonic detector. However, also different implementations are conceivable. The ultrasonic transceivers 5, 6 are, for example, mounted to the motor vehicle 1 at a rear end, for example at or in a rear bumper of the motor vehicle 1 , and / or at a front end, for example at or in a front bumper of the motor vehicle 1 . In particular, an overall field of view of the ultrasonic transceivers 5, 6 overlaps at least partially with the field of view of the camera 4. The electronic vehicle guidance system 2 further comprises a storage device (not shown) storing a computer algorithm including a trained ANN 7 for carrying out a method for parking slot detection according to the invention. An exemplary block diagram of the ANN 7 is depicted in Fig. 2. The electronic vehicle guidance system 2 also comprises a data processing apparatus with at least one computing unit 3, which may store the algorithm including the ANN 7.

[0090] The camera 4 is configured to generate a camera image 8. Each of the at least one ultrasonic transceiver 5, 6 comprises an ultrasonic emitter, which is configured to emit ultrasonic pulses into the environment, and an ultrasonic detector, which is configured to generate one at least one ultrasonic sensor signal based on reflected portions of the emitted ultrasonic pulses.

[0091] A two-dimensional ultrasonic map 29 in a top view perspective is generated by a conversion module 10 of the algorithm depending on the at least one ultrasonic sensor signal 9. Parking slot proposal data comprising a position of a proposed parking slot 18 in the environment, which is equipped with an infrastructure device 22 of a predefined device type, in particular a charging station for electric vehicles, is generated by applying the ANN 7 to input data, which depends on the camera image 8 and the ultrasonic map 29.

[0092] For at least one parking slot 17, 18, 19, 20, shown schematically in the top view perspective in Fig. 3, a respective parking slot feature map may be generated, which comprises a position of the respective parking slot, by applying a parking slot detection module 13 of the ANN 7 to first input data, which depends on the camera image 8. For example, the parking slot detection module 13 is applied to the camera image 8.

[0093] The parking slot feature map may for example comprise an occupancy status of the respective parking slot 17, 18, 19, 20, and / or a position of a center point 25 of the respective parking slot 17, 18, 19, 20 and / or a respective position of one or more boundary points 24, for example corner points, on a boundary of the respective parking slot 17, 18, 19, 20.

[0094] The parking slot proposal data is for example generated by applying a decoder module 15 of the ANN 7 to intermediate data, which comprises the parking slot feature maps, wherein the proposed parking slot 18 one of the at least one parking slot 17, 18, 19, 20, whose occupancy status indicates that the proposed parking slot 18 is unoccupied. For example, some of the parking slots 17, 18, 20 are equipped with an infrastructure device 21 , 22, 23 of the predefined device type, in particular a charging station, while some of the parking slots 19 are not.

[0095] For example, a respective infrastructure device feature map is generated for each of the at least one infrastructure device 21 , 22, 23, which comprises a position of the respective infrastructure device 21 , 22, 23, by applying an infrastructure device detection module 14 of the ANN 7 to the first input data. The intermediate data, which is provided to the decoder module 15, comprises the infrastructure device feature maps.

[0096] The infrastructure device feature map comprises a position of a center point 27 of the respective infrastructure device 21 , 22, 23 and / or a respective position of one or more boundary points 26, for example corner points, on a boundary of the respective infrastructure device 21 , 22, 23, as shown schematically in Fig. 4.

[0097] For example, at least one ultrasonic feature map is generated by applying an ultrasonic feature encoder module 1 1 of the ANN 7 to second input data, which depends on the ultrasonic map 29. The intermediate data, which is provided to the decoder module 15, depends on the at least one ultrasonic feature map.

[0098] At least one control signal for guiding the motor vehicle 1 at least in part automatically to the proposed parking slot 18 may be generated by a controller module 16 of the motor vehicle 1 , in particular of the at least one computing unit 3, and / or visual driver assistance information indicating the proposed parking slot 18 to a driver of the motor vehicle 1 is displayed on a display device of the motor vehicle 1 .

[0099] The at least one computing unit 3 may for example generate the ultrasonic map 29 as a single channel top view map of the close surroundings of the motor vehicle 1 . For example, it may be given on a grid with a size in the order of meters, for example 6 m x 12 m, and a cell side length in the order of centimeters, for example 1 cm. In this way, the maximum detection range of the ultrasonic transceivers 5, 6, which is for example approximately 5 m, may be covered also taking into account their position relative to the camera 4, which defines the center of the coordinate system via projection on the ground surface. The ultrasonic map 29 may therefore be rather large and relatively sparse, that is only a small area has a high amplitude. Thus ultrasonic feature encoder module 1 1 is used to bring the ultrasonic map 29 into the feature space where it can be combined with the infrastructure device feature maps and the parking slot feature maps.

[0100] The at least one ultrasonic sensor signal 9 may result from time-series measurements and thus represent the ultrasonic echo amplitude recorded over a fixed time duration.

[0101] Usually, peaks in the at least one ultrasonic sensor signal 9 stem from an object in the environment of the motor vehicle 1 reflecting the ultrasonic pulse emitted from one ultrasonic transceiver 5, 6 back to the same or another one of the ultrasonic transceivers 5, 6. Consequently, the total travel distance of the ultrasonic pulse can be computed, wherein for example the ambient temperature may be taken into account to determine the accurate speed of sound.

[0102] In order to transform the 1 D amplitude data as a function of time into the spatial domain, one can calculate the travel distance of the echo and in addition consider the ignorance about the angular position of the object reflecting the echo. It could be located on the longitudinal sensor axis of the ultrasonic transceiver 5, 6, but due to its large field of view, it could also be off of the longitudinal sensor axis by a large angle. Up to 70° may be reasonable as long as the object is positioned to reflect back to the sensor before the echo amplitude drops to be no longer distinguishable from random noise.

[0103] Fig. 5 depicts schematically, how the ultrasonic map 29 representing the environment of the motor vehicle 1 is computed in the top view perspective, in particular in a vehicle coordinate system, where the center of the rear axle of the motor vehicle 1 is in the origin of the coordinate system. The respective positions and orientations of the ultrasonic transceivers 5, 6 are predetermined and known.

[0104] A grid may be generated with approximately the size of the field of view of the ultrasonic transceivers 5, 6 and with a grid cell size that is small enough to offer sufficiently high resolution and still comparable to the distance resolution according to the at least one ultrasonic sensor signal 9. For example, quadratic grid cells with a side length of 1 cm may be used. For each of the grid cells and for each emitter-detector pair of the at least one ultrasonic transceiver 5, 6, a corresponding travel distance from a position of respective ultrasonic emitter via a position of the grid cell to a position of the respective ultrasonic detector is computed. A signal value of the ultrasonic sensor signal 9, which is generated by the respective ultrasonic detector, is computed depending on the travel distance. For each emitter-detector pair, a respective angular weighting function 28 is provided. For each grid cell, each signal value is multiplied with the respective angular weighting function 28 evaluated at the position of the grid cell to obtain a respective weighted signal value. For each grid cell, a grid value is computed as a sum of the weighted signal values obtained for the respective grid cell. The grid values of all grid cells yield the ultrasonic map 29.

[0105] In a simplified instructive example, one may assume that there are only two ultrasonic transceivers. One has a first ultrasonic transceiver (E1 , D1 ) with a first ultrasonic emitter E1 and a first ultrasonic detector D1 as well as a second ultrasonic transceiver (E2, D2) with a second ultrasonic emitter E2 and a second ultrasonic detector D2. Then D1 generates a first ultrasonic signal S1 and D2 generates a second ultrasonic signal S2. Considering a grid cell G, one has in principle four travel distances, namely r11 from E1 to G to D1 , r12 from E1 to G to D2, r21 from E2 to G to D1 and r22 from E2 to G to D2.

[0106] Then, S1 is evaluated at r11 and at r21 , yielding respective signal values S1 (r11 ), S1 (r21 ), wherein the available values of S1 may be interpolated accordingly. Analogously, S2 is evaluated at r12 and at r22, yielding respective signal values S2(r12), S2(r22), wherein the available values of S2 may be interpolated accordingly. Furthermore, a first angular weighting function associated to the first ultrasonic transceiver (E1 , D1 ) at the position of G may be given by F1 and a second angular weighting function associated to the second ultrasonic transceiver (E2, D2) at the position of G may be given by F2.

[0107] The grid value at G may then be computed for example as

[0108] S1 (r11 )*F1 + S1 (r21 )*min(F1 ,F2) + S2(r22)*F2 + S2(r12)*min(F1 ,F2), wherein "min" denotes the minimum value of both angular weighting functions. Alternatively, one may combine the angular weighting functions in a different way, for example resulting in the grid value at G

[0109] S1 (r1 1 )*F1 + S1 (r21 )*F11 / 2* F21 / 2+ S2(r22)*F2 + S2(r12)* F11 / 2* F21 / 2.

[0110] In some implementations, the ultrasonic map 29 is transformed from the top view perspective into a camera image plane perspective of the camera 4. The ultrasonic feature encoder module 1 1 may then be applied to the transformed ultrasonic map 29 instead. The transformation from the top view perspective into the camera image plane perspective is illustrated in Fig. 6 and Fig. 7. Fig. 6 shows a pattern 33 with different contours 34, 35, 36, 37 in the top view perspective. Fig. 7 shows a transformed pattern 33’, wherein the contours 34, 35, 36, 37 are mapped to the camera image plane perspective of a fisheye camera resulting in distorted contours 34’, 35’, 36’, 37’.

Claims

Claims1 . Method for parking slot detection, wherein a camera image (8) depicting an environment of a motor vehicle (1 ) is received from a camera (4) mounted to a motor vehicle (1 ); at least one ultrasonic sensor signal (9) is received from at least one ultrasonic detector (5, 6) mounted to the motor vehicle (1); a two-dimensional ultrasonic map (29) in a top view perspective is generated depending on the at least one ultrasonic sensor signal (9); parking slot proposal data comprising a position of a proposed parking slot (18) in the environment, which is equipped with an infrastructure device (22) of a predefined device type, is generated by applying a trained artificial neural network, ANN, (7) to input data, which depends on the camera image (8) and the ultrasonic map (29).

2. Method according to claim 1 , wherein for at least one parking slot (17, 18, 19, 20) in the environment, a respective parking slot feature map is generated, which comprises a position of the respective parking slot, by applying a parking slot detection module (13) of the ANN (7) to first input data, which depends on the camera image (8); and the parking slot proposal data is generated by applying a decoder module (15) of the ANN (7) to intermediate data, which comprises the generated parking slot feature maps, wherein the proposed parking slot (18) one of the at least one parking slot (17, 18, 19, 20).

3. Method according to claim 2, wherein the parking slot feature map comprises an occupancy status of the respective parking slot (17, 18, 19, 20); and the occupancy status of the proposed parking slot (18) indicates that the proposed parking slot (18) is unoccupied.

4. Method according to one of claims 2 or 3, wherein the parking slot feature map comprises a position of a center point (25) of the respective parking slot (17, 18, 19, 20) and / or a respective position of one or more boundary points (24) on a boundary of the respective parking slot (17, 18, 19, 20).

5. Method according to one of claims 2 to 4, wherein for at least one infrastructure device (21 , 22, 23) of the predefined device type in the environment, a respective infrastructure device feature map is generated, which comprises a position of the respective infrastructure device, by applying an infrastructure device detection module (14) of the ANN (7) to the first input data; and the intermediate data comprises the generated infrastructure device feature maps.

6. Method according to claim 5, wherein the infrastructure device feature map comprises a position of a center point (27) of the respective infrastructure device (21 , 22, 23) and / or a respective position of one or more boundary points (26) on a boundary of the respective infrastructure device (21 , 22, 23).

7. Method according to one of claims 2 to 6, wherein at least one ultrasonic feature map is generated by applying an ultrasonic feature encoder module (11) of the ANN (7) to second input data, which depends on the ultrasonic map (29); and the intermediate data depends on the at least one ultrasonic feature map.

8. Method according to claim 7, wherein an ultrasonic map transformation module is applied to the ultrasonic map (29) to transform the ultrasonic map (29) from the top view perspective into a camera image plane perspective of the camera (4); and the second input data comprises the transformed feature map.

9. Method according to claim 7, wherein a feature transformation module is applied to the at least one ultrasonic feature map to transform the at least one ultrasonic feature map from the top view perspective into a camera image plane perspective of the camera (4); and the intermediate data comprises the transformed at least one ultrasonic feature map.

10. Method according to one of the preceding claims, wherein the at least one ultrasonic sensor signal (9) is generated by the at least one ultrasonic detector (5, 6) based on reflected portions of ultrasonic pulses emitted into the environment by at least one ultrasonic emitter (5, 6) of the motor vehicle (1 ).11 . Method according to claim 10, wherein the ultrasonic map (29) is given by a plurality of grid values, each grid value corresponding to a respective grid cell of a predefined spatial grid in the top view perspective, wherein for each of the grid cells and for each emitter-detector pair of the at least one ultrasonic emitter (5, 6) and the at least one ultrasonic detector (5, 6) a corresponding travel distance from a position of the ultrasonic emitter (5, 6) of the emitter-detector pair via a position of the grid cell to a position of the ultrasonic detector (5, 6) of the emitter-detector pair is computed; a signal value of the ultrasonic sensor signal (9), which is generated by the ultrasonic detector (5, 6) of the emitter-detector pair, is computed depending on the travel distance; and the grid value of the grid cell is computed depending on the signal value.

12. Method according to claim 11 , wherein for each emitter-detector pair, a respective angular weighting function (28) is provided; for each grid cell, each signal value is multiplied with the respective angular weighting function (28) evaluated at the position of the grid cell to obtain a respective weighted signal value; and for each grid cell, the grid value is computed depending on a sum of the weighted signal values obtained for the respective grid cell.

13. Method according to claim 12, wherein the angular weighting function (28) depends on at least one beta-distribution.

14. Method according to one of the preceding claims, wherein the predefined device type corresponds to a charging station for an electric vehicle and the motor vehicle (1) is an electric vehicle.

15. Method for parking a motor vehicle (1 ), wherein a method according to one of the preceding claims is carried out and at least one control signal for guiding the motor vehicle (1) at least in part automatically to the proposed parking slot (18) is generated; and / or visual driver assistance information indicating the proposed parking slot (18) to a driver of the motor vehicle (1) is displayed on a display device of the motor vehicle (1).

16. Data processing apparatus comprising at least one computing unit (3), which is adapted to carry out a method according to one of claims 1 to 14.

17. Electronic vehicle guidance system (2) for a motor vehicle (1 ) comprising a data processing apparatus according to claim 16, wherein the at least one computing unit (3) is configured to generate at least one control signal for guiding the motor vehicle (1) at least in part automatically to the proposed parking slot (18); and / or control a display device of the motor vehicle (1 ) to display visual driver assistance information indicating the proposed parking slot (18) to a driver of the motor vehicle (1).

18. Electronic vehicle guidance system (2) according to claim 17 comprising the at least one camera (4) and / or the at least one ultrasonic detector (5, 6).

19. Computer program product comprising instructions, which, when executed by a data processing apparatus, cause the data processing apparatus to carry out a method according to one of claims 1 to 15.

Images