How to use convolutional neural networks to detect environmental information describing parking spaces

Abstract

To provide a method, a sensor device, a vehicle, a computer program product, and a computer readable storage medium that detect peripheral environment information that describe a parking space using a convolutional neural network.SOLUTION: A method comprises: assigning measurement data to one or more objects in a peripheral environment using a peripheral environment sensor of the vehicle 1; creating, on the basis of the measurement data, a peripheral environment map describing the peripheral environment; identifying the measurement data spatially relative to the vehicle in the peripheral environment map; feeding the peripheral environment map to a convolutional neural network; and causing the neural network to classify and locate, on the basis of the peripheral environment map, a vehicle parking space 9, detect structure descriptions 15a to 18c defining each parking space and generate structure description peripheral environment information based on the classified and located parking space.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a method for detecting surrounding environment information describing at least one parking space using a sensor device of a vehicle, the sensor device comprising at least one surrounding environment sensor and a computing device. The present invention also relates to a method for training a convolutional neural network, a sensor device, a vehicle, a computer program product and a computer readable storage medium. [Background technology]

[0002] Modern vehicles can use environmental sensors to obtain information about various objects in the vehicle's environment, which can be, for example, other traffic participants, e.g., other moving or stationary vehicles, or non-moving structures, e.g., curbs, walls, marks, etc. The information obtained in this way can be used to provide various driving assistance functions.

[0003] For example, from such object-based environment description further information about available parking spaces in the vehicle's environment can be derived. Also, algorithms are generally used to detect vacant parking spaces from detected objects and / or their positions in the environment according to set rules and dependencies. In this case, the rules are set, for example, by means of lookup tables and / or by case-specific queries, i.e. if conditions.

[0004] This process has the disadvantages that creating and checking these relatively complex and extensive rules is time-consuming and that the parking space identification is generally limited to the scenarios queried by the rules. Also, the deployment of the rules and their adaptation to the various requirements of different vehicles and / or parking space detection is costly to implement. From other technical fields, so-called deep learning methods are known, which provide an evaluation of image data by means of methods from the artificial intelligence field, for machine evaluation of image data or computer-based vision.

[0005] An example of such a method is the "You Only Look Once Method" (also called YOLO method) described in the paper "Patent Document 1", which allows for the simultaneous classification and localization of objects in an image by using a grid in a separate neural network, where the neural network finds class probabilities assigned to the objects and frames surrounding the objects in the image and localizing the objects, each of which surrounds a separate classified object and whose edges each extend parallel to the edges of the rectangular evaluated image. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Redmon, Joseph, et al. “You only look once: Unified, real-time object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition(2016) Summary of the Invention [Problem to be solved by the invention]

[0007] The object of the present invention is to show an improved method for detecting surrounding environment information describing a parking space, which in particular allows an improved adaptability of the parking space identification to different requirements and an improved description of different scenarios. [Means for solving the problem]

[0008] According to the invention, this problem is solved by the method as described in the opening paragraph, which method comprises the steps of: - recording measurement data describing an environment surrounding the vehicle by means of at least one environment sensor, the measurement data being assigned to one or more objects in the environment of the vehicle; - creating at least one environment map describing the vehicle's environment based on the measurement data, the measurement data being spatially localized with respect to the vehicle in the environment map; - evaluating at least one environment map by a computing device, the environment map being fed to a convolutional neural network, the neural network classifying and locating parking spaces present in the vehicle's environment based on the environment map, and the neural network detecting, for each of the parking spaces, at least one structural description of at least a part of a structure defining the parking space based on the environment map; - generating surrounding environment information of at least one structural description based on the classified and located parking spaces.

[0009] The method according to the invention is based on measurement data detected by means of one or more surrounding environment sensors of the vehicle, where the measurement data describes objects in the surrounding environment of the vehicle. The objects may in particular be other vehicles, infrastructure objects, buildings or structures, and may be lane markings and / or parking spot marks. In particular, the measurement data generally describes objects or parts of objects that may define a parking space. The measurement data may be recorded by means of one or more different types of surrounding environment sensors.

[0010] The measurement data, for example the measurement points to which the distance to the vehicle is assigned, may depend on the type of surrounding environment sensor. Such measurement points may be generated, for example, by means of ultrasonic sensors, radar sensors or lidar sensors. In this case, the measurement points may, for example, specify the distance to a point on the surface of an object. Additionally, the measurement points may also be assigned a direction, so that the relative position of the detection location, i.e. of the object surface, with respect to the vehicle is obtained from the measurement points. In the case of a sensor configured as a camera, the measurement data may describe, for example, lane markings or parking spot marks as objects and their distance or position with respect to the vehicle.

[0011] Based on the recorded measurement data, which in particular describe the current surroundings of the vehicle, an environment map is then created. The measurement data is also spatially determined with respect to the vehicle, in particular in a rectangular map in which the vehicle is located at the center. The measurement data may then be plotted on a two-dimensional map, which for example constitutes a plane extending along the length and width of the vehicle. The environment map may for example be generated as an occupancy map. It is also possible to use other map types and / or representation types which replicate the arrangement of the measurement points relative to the vehicle. The environment map preferably comprises a spatial grid, on which the measurement data is plotted or to which the measurement points are assigned. In this way, the surroundings of the vehicle described by the measurement points can be shown on the environment map.

[0012] The present invention is based on the recognition that such an environment map may be understood as an image, and that the deep learning method used for image evaluation may also be applicable to such an environment map. Thus, according to the present invention, a convolutional neural network (CNN) is used to evaluate the environment map. The convolutional neural network may be stored, for example, in a computing device of the sensor device, and thus the convolutional neural network may be provided with the environment map as input data. Alternatively, the convolutional neural network may be stored in a further computing device, for example a central control device, that communicates with the sensor device, and this further computing device is particularly implemented separately from the sensor device, i.e. the computing device of the sensor device.

[0013] The neural network is used to evaluate an environment map for parking spaces present in the vehicle's environment, i.e., for vacant parking spaces present in the vehicle's environment, and for one or more structures that at least partially define the parking spaces, and the neural network classifies and localizes parking spaces in the vehicle's environment based on the environment map, whereby for each parking space, at least one structural description of at least a portion of the structures that define the parking space is detected.

[0014] In particular, for the evaluation, only one separate convolutional neural network is used, which classifies the parking spaces as well as estimating their locations and detects at least one structural description for each parking space. In other words, the neural network is used to perform an image evaluation method based on the YOLO method on the surrounding environment map. It is also possible to provide a further neural network downstream of the convolutional neural network, which evaluates the output data detected by the convolutional neural network.

[0015] The classification of the parking spaces may in particular be performed by assigning classification probabilities to different parts of the environment map, in other words by indicating classification probabilities for different partial areas of the environment map respectively, which indicate with what probability the part relates to a parking space.

[0016] In comparison to the conventional YOLO method described in the introduction, the localization of an identified parking space is not performed by inserting a rectangular frame surrounding the parking space, always oriented along the image edge, but instead one or more structures that each at least partially define the parking space are precisely detected, where the structures are each mathematically described by at least one structural description.

[0017] The structural description may in particular describe the extension of the structure or parts of the structure in relation to the vehicle, such that the environmental information advantageously includes a determination of the presence and relative positioning of structures that define the parking space, e.g. objects, marks, etc. In this case, different scenarios may advantageously be identified based on the environmental map and made available as environmental information.

[0018] The neural network can advantageously be configured to detect different types of structures based on the surrounding environment map, so that an accurate description of the surroundings of the parking spaces classified by means of the neural network can be obtained as parking space information. Based on the parking spaces classified by the neural network and their relative arrangement, the surrounding environment information is then formed, for example, the surrounding environment information comprising at least parking spaces whose classification probability exceeds a predefined threshold, for which at least one detected structural description is also identified in each case.

[0019] Advantageously, in this way, surrounding environment information is generated, which allows the measurement data of different types of sensors to be taken into account and evaluated. This allows the method to be used flexibly in different types of vehicles and / or in vehicles that are configured differently, in particular with regard to their sensors. Furthermore, the method can be adapted to different scenarios, in particular by correspondingly training the neural network used, so that surrounding environment information can be generated that provides a highly accurate description of the vehicle's surroundings even in a number of different environments. Advantageously, the method can be easily extended to new scenarios by feeding the new scenarios to the neural network for training.

[0020] Based on the detected structural description for each classified parking space, the surrounding environment information can be flexibly used for many further applications, for example, for driving assistance systems. In particular, the surrounding environment information can cover all common entrance parking scenarios, so that further planning and / or execution of the entrance parking process can be easily performed based on the surrounding environment information.

[0021] The surrounding environment information can be used in particular for controlling actuators of the vehicle and / or displays of the vehicle. The vehicle actuators can for example be actuators arranged for steering the vehicle widthwise and / or the vehicle lengthwise. The at least one actuator can for example be a drive motor of the vehicle and / or a steering drive of the vehicle. Here, the control of the actuators can for example be performed depending on trajectory information describing the driving maneuver of the vehicle, generated from the surrounding environment information.

[0022] Depending on the surrounding environment information, for example, an automatic or partially automatic control of the entrance parking process of the vehicle can be performed, in particular by controlling at least one actuator of the vehicle. During the entrance parking process, the vehicle can move forwards and / or backwards and, between and / or during this movement, a steering angle change can be performed with respect to a set steering angle, in particular also multiple movements or multiple partial movements in different directions. In this case, the entrance parking process can be performed in particular in such a way that the vehicle is positioned in a parking space described by at least one surrounding environment information.

[0023] Additionally or alternatively to the actuators, at least one display device of the vehicle can also be controlled, so that for example the driver of the vehicle can be made aware of the detected parking spaces in the vehicle's surroundings, for example by means of corresponding image representations. Here, the display device can for example be a display arranged in the interior space of the vehicle, a head-up display, a virtual side mirror, etc. The representation on the display device can for example be used to enable the driver to select one of the detected parking spaces as a destination for the entrance parking process. Additionally or alternatively, assistance means for moving the vehicle to the detected and / or selected parking space, for example virtual lines and marks, which enable the driver to control the vehicle to move to this parking space, can be displayed on the display device. In the case of a vehicle configured for driverless travel, the display device can be located outside the vehicle and can for example be implemented as a portable device and / or as part of a fixed computing device.

[0024] The vehicle may be, for example, a motor vehicle, for example a car or a truck. The vehicle may also be, for example, a combination vehicle with a towing vehicle and one or more trailers. Alternatively, the method may also be used with further types of vehicles. The vehicle may in particular be a mobile robot that is at least locally free to move, in particular autonomously.

[0025] The parking space may be, for example, a parking location for a vehicle. It is also conceivable that the parking space is a charging station for the vehicle and / or suitable for unloading a part of the vehicle, for example a trailer. It is also conceivable that the parking space is used for loading and unloading of a load carried with the vehicle and / or an object transported by the vehicle. In this context, the surrounding environment information may, for example, describe the structure of a logistics facility in the surrounding environment of the vehicle, which allows for highly accurate loading and unloading of the load or transported object.

[0026] The sensor device of the vehicle may comprise one or more ambient environment sensors, which are arranged on the vehicle so as to record the immediate surroundings of the vehicle. The sensor device further comprises a computing device which may be part of the vehicle. Alternatively, the computing device may be a computing device external to the vehicle, which for example communicates with the at least one ambient environment sensor via a communication link, in particular wirelessly. Ambient environment information detected by the computing device external to the vehicle may be transmitted back to the vehicle via the communication link, so that the control of actuators and / or display devices of the vehicle as mentioned above can be effected depending on the ambient environment information.

[0027] The creation of the surrounding environment map based on the measurement data of the at least one surrounding environment sensor may likewise be performed by a computing device, alternatively this step may be performed by a further computing device in the vehicle and / or a further computing device external to the vehicle, which further computing device transmits the surrounding environment map to the computing device.

[0028] The neural network used to evaluate the environment map is a convolutional neural network (CNN) stored in the computing device or in a storage device of the computing device. The evaluation of the environment is thus performed using artificial intelligence or using deep learning methods. The convolutional neural network is configured or trained in particular to classify and localize available parking spaces in the environment of the vehicle based on the environment map and to detect, for example by regression, each of the parking spaces relative to the vehicle. The neural network is also configured or trained to detect, for each of the parking spaces, at least one structural description of at least a part of the structure that defines the parking space based on the environment map. In this way, the parking space detection is performed based on the environment map using a process that is adaptable and improveable by machine learning. This allows the parking space detection to be flexibly adapted to different scenarios by correspondingly training the neural network. Advantageously, an easily implementable adaptability of the parking space detection is thus achieved.

[0029] In the sense of the present invention, the concept "machine learning" (ML) denotes the use of algorithms for the analysis of training data, learning based on this training data and the subsequent determination or prediction of initially unknown features of the input data to be evaluated. In this context, both supervised and unsupervised learning can be used. In particular, the labeling strategy or the labeling of the training data can be of crucial importance for the analysis.

[0030] In the sense of the present invention, the concept of "deep learning" (DL) comprises a cascade of multiple layers of non-linear processing units (often artificial neural networks) for extracting and transforming features (or for extracting and transforming parameters). The concept of "artificial neural network" (ANN) or "convolutional neural network" (CNN) comprises a network of artificial neurons that are abstractly connected to each other for information processing, like the nervous system of a living organism. The neurons are illustrated as nodes and their connections as edges. The last (node) layer of the network is called the output layer, and the "invisible" (node) layer located before it is called the hidden layer.

[0031] Such an artificial neural network may be one-layer (output layer), two-layer (output layer and hidden layer for improving abstraction) or multi-layer (at least one output layer and several hidden layers for improving abstraction) structure.Also, the artificial neural network may be configured to have forward (feed-forward) and / or backward edges (recurrent connections) in its data transmission (feedback; feedback network).

[0032] In the sense of the present invention, the notion of a "computer-implemented method" denotes a schedule that can be implemented and executed on the basis of a data processor. A data processor, e.g. a computer, a computer network or other programmable device, can process data using programmable arithmetic rules. With regard to a method, the essential properties can be implemented, for example, by a novel program, several novel programs, an algorithm, etc.

[0033] According to the present invention, the surrounding environment map may be configured to include a map grid, measurement data is assigned to cells of the map grid, and for each cell the number of measurement points of the measurement data assigned to the cell, the height of an object described based on the measurement data assigned to the cell, the type of object described based on the measurement data assigned to the cell and / or at least one signal characteristic of the measurement data assigned to the cell may be identified.

[0034] For example, in the case of measurement data originating from a sensor type that evaluates reflected echoes of a transmitted signal, the number of echoes reflected from a cell by an object may each be stored as a signal characteristic. Sensors of this type may be, for example, ultrasonic, radar or lidar sensors. Here, the echoes may in particular each indicate a measurement point. It is also conceivable that, based on the echoes or the measurement points of such sensors, information regarding the height of the object from which the echoes are reflected can also be determined. Height information describing this height of the object may also be assigned to the individual cells of the map grid of the surrounding environment map for each measurement point or averaged over several measurement points.

[0035] As a signal characteristic, for example, the intensity of the echo or measurement point assigned to the cell may be stored in the environment map, where the intensity may be determined for the measurement points individually or on average. Depending on the sensor type, further characteristics of the measurement data may also be entered in the environment map, in particular characteristics that allow inferences about the object type and / or the detection quality.

[0036] The type of object described on the basis of the measurement data for each cell may also be specified in the environment map and assigned to the individual cells. The type of object may be detected on the basis of the measurement data, for example on the basis of measurement data obtained via an environment camera. In this way, objects that cannot be recorded or cannot be recorded well via ultrasonic, radar or lidar sensors, such as lane markings or parking space marks, can also be written into the environment map.

[0037] In this case, these information are assigned in particular to those cells which correspond to the object positions whose positions are detected via the sensor device, in particular to the surrounding environment cells which correspond to the parts of the space in the surrounding environment of the vehicle that are not blocked by objects, so that they can be left free without being assigned measurement data.

[0038] According to the invention, the neural network may be configured to overlay on the surrounding environment map an evaluation grid comprising a plurality of cells, each cell being assigned at least one frame, and in the surrounding environment information, for each of the frames, a classification as a parking space, an orientation of the parking space assigned to the frame relative to the frame and / or at least one structural description of a structure defining the parking space may be identified, which is described based on measurement data.

[0039] Here, the frame, which may also be referred to as a box, has in particular a rectangular shape. The frames of the cells of the evaluation grid can each be positioned and / or oriented in the same way with respect to the cell. The frames each represent a first estimate of possible parking spaces, which can each be assigned classification information, e.g. a classification probability, regarding their classification as parking spaces by the neural network. This in turn allows at least a basic location estimate of the possible parking spaces in the environment map. For each frame classified as a parking space, one or more structure descriptions can be detected and assigned to the frame in the environment information. In particular, for each frame classified as a parking space, a structure associated with the respective parking space can be set in this way.

[0040] Additionally, at least for frames classified as parking spaces, further characteristics describing the precise positioning and / or orientation of the parking spaces and / or their respective widths and lengths can be detected by the neural network using regression and also stored or further processed as part of the surrounding environment information. Furthermore, for example, a rotation angle by which each frame needs to be rotated to match the actual parking space can be set. Additionally or alternatively, further parameters describing the geometric relationship can also be set, such as a scale factor between the frame and the parking space.

[0041] In a preferred embodiment of the invention, an evaluation grid may be used such that each cell is assigned a number of differently positioned and / or differently oriented frames, where the frames may be, for example, two rectangular frames rotated by 90° relative to each other, making it easier to detect the relative orientation of the parking space to the vehicle, since it can be based initially on frames that already have a high match or high classification probability with the parking space described by the environment map.

[0042] It is also possible that each cell of the evaluation grid includes more than two frames and / or that the cells are oriented differently with respect to each other. In particular, the evaluation grid may have a first rectangular frame with a longitudinal edge extending in the length direction of the vehicle and two second frames, the longitudinal edges of the second frames each extending in the width direction of the vehicle. The two frames may then each be arranged in a cell offset in the length direction of the vehicle. In this case, a grid may be used in which the cells have edges extending parallel to the length direction of the vehicle and edges extending parallel to the width direction of the vehicle.

[0043] The more frames the evaluation grid contains, the more accurately the arrangement and / or extension lines can be detected or replicated in the surrounding environment information using the structure described by the structural description. Furthermore, the accuracy of the localization and detection of the relative arrangement of the parking space with respect to the vehicle can be improved.

[0044] In a preferred embodiment of the invention, at least one contour line is used as the structural description, the contour line may be configured to be formed by one or more line segments and / or two or more points.

[0045] The use of contour lines describing a structure or a part of a structure allows the aspects of the structure that define a parking space to be shown with high accuracy, so that, for example, driving maneuvers or trajectory planning can be performed efficiently and with high accuracy based on the surrounding environment information, where the description of the contour lines as polylines consisting of multiple line segments and / or two or more points allows for a compact duplication of the structure description, thus enabling similarly compact surrounding environment information, advantageously having a positive impact on memory and / or computational requirements for processing the surrounding environment information.

[0046] According to the invention, the structural description may be configured to describe at least one edge of the object defining the parking space facing the vehicle, at least one edge of the object defining the parking space that defines the parking space in the length direction and / or width direction, at least one edge of the object defining the entrance to the parking space, at least one edge of the object facing the entrance and / or at least one parking spot marking.

[0047] The object defining the parking space with its edge facing the vehicle may for example be a further vehicle located next to the parking space. Furthermore, the object may be a building, an infrastructure component, etc. Such an object may have at least one further edge defining the parking space in length and / or width. The object defining the entrance to the parking space may for example be a pillar, a large stone, etc. that needs to be driven around when entering the parking space.

[0048] The structure described by the structural description may be an object located opposite the entrance to the parking space. Such an object may be, for example, a curb, a wall or a further vehicle. Furthermore, the structural description may also describe individual edges of the parking space and / or parking space marks that indicate the contour of the parking space.

[0049] The structures described by the structure description each either physically define a parking space or, for example, define a parking location provided. The description of this structure as part of the surrounding environment information makes it possible to detect, for different scenarios resulting from any combination of said structures, respectively, appropriate trajectories that allow the movement of the vehicle into a parking space between the structures, for example.

[0050] The structural description of the object may also include height information, if such information is present, for example, as part of the measurement data and / or can be detected from a map of the surrounding environment by a neural network. For example, for a structure located opposite the entrance to a parking space, the curbs and / or the walls may also each be described by the structural description, if, for example, the parking space is firstly defined by one or more curbs, followed by a wall located at a large distance from the entrance. In such a case, for example, the vehicle may be parked in such a way that the vehicle partially goes over the curb without hitting the wall (so-called boot-over scenario). Additionally or alternatively, the vehicle may be parked partially on the curb, if the curb has a height that allows the vehicle to go over the curb.

[0051] By taking into account the structures that define the entrances, scenarios such as, for example, underground parking lots can be described, where the available parking locations are defined laterally by pillars or the like. Advantageously, the surrounding environment information thus allows different parking scenarios to be detected and indicated as part of the surrounding environment information. In this context, advantageously, the use of neural networks makes it possible to handle a large number of different scenarios efficiently and accurately.

[0052] According to the present invention, the surrounding environment information may be configured to include, for at least a portion of the classified parking spaces, parking area information describing the arrangement and / or orientation of the parking areas available in the parking spaces and / or at least one parking space type classification identifying at least one parking space type classification.

[0053] Here, the parking area information may specify which parking areas are available within the identified parking space. The parking area information may take into account, for example, that a parking area in the parking space for the vehicle may be reduced due to a part of a structure. Furthermore, for example, the above-mentioned curb climbing or parking with a part of the vehicle on the curb may result in a parking area that is larger than the space between the structures that define the parking space. The parking area information may then be determined, for example, by a neural network using regression as a position estimate for the parking space.

[0054] The parking space type classification may specify for a classified parking space, for example, whether it is a parallel parking space or a parallel parking space. For example, for each classified parking space, the surrounding environment information may also include a classification as a parallel parking space and a classification as a parallel parking space, each classification being specified, for example, as a classification probability. In this case, a parallel parking space is a parking space that can be parked along the driving direction of the road on which the vehicle is located. A parallel parking space is a parking space that can be parked across the driving direction of the road on which the vehicle is located. The distinction between parallel parking spaces and parallel parking spaces may be made particularly simply, for example, by providing a frame in the evaluation grid with two rectangles rotated by 90° with respect to each other.

[0055] Additionally, different types of parallel parking spaces may be classified differently. For example, parallel parking spaces that can be driven forward by a vehicle and parallel parking spaces that can be driven backward by a vehicle may each have an independent classification. Which direction a parallel parking space can be driven in by a vehicle can also be derived via the orientation of the parking space relative to the vehicle, which can also be detected by a neural network. In this way, a classification can be identified in the surrounding environment information, which significantly eases subsequent use of the surrounding environment information for performing a parking maneuver or detecting a trajectory for such a maneuver.

[0056] The surrounding environment information detected by the convolutional neural network can be supplemented with position information in a downstream step. The position information can take into account that the parking space and / or the essentially available parking area in the parking space is generally larger than the area occupied by the vehicle after parking in. The target position can therefore be located in particular completely inside the parking space and / or inside the parking area, for example maintaining a certain safety distance to the edge or edge of the parking space. The target position can also be described, for example, by a rectangle, which has an edge length smaller than the parking space or parking area and / or is rotated by an angle relative to the parking space or parking area.

[0057] In a preferred embodiment of the invention, a sensor device with a plurality of ultrasonic sensors, at least one lidar sensor, at least one radar sensor and / or at least one environment camera may be used as the at least one environment sensor. The sensor device may in particular comprise a plurality of the above sensor types and / or a combination of the above sensor types. The respective measurement data of the various sensors may in particular be plotted or integrated in the same environment map.

[0058] The present invention further provides a method for training a convolutional neural network, comprising the steps of: - providing at least one training data set including a plurality of training environment maps each describing an environment of the vehicle based on training measurement data, the training measurement data being spatially located with respect to the vehicle in the training environment map; - assigning ground truth describing at least one parking space in the vehicle's environment to at least a portion of a training environment map, the ground truth classifying and localizing at least one section of the training environment map as a parking space, the parking space ground truth identifying at least one structural description of at least a portion of a structure defining the parking space; - superimposing on the training environment map evaluation grids each including a plurality of cells, each cell of the evaluation grid being assigned at least one frame and each frame being assigned at least one structural description; - detecting, for each frame, the frame that has the maximum overlap with respect to at least one parking space described by the ground truth; - optimizing a neural network for a match between at least one structural description of a frame having a maximum overlap and a structural description described by ground truth.

[0059] The method for training a convolutional neural network according to the invention can in particular be used for training a neural network used in the method for detecting surrounding environment information describing at least one parking space according to the invention. By training, the internal parameters of the neural network can be adjusted, so that the neural network can be configured or trained to evaluate a surrounding environment map generated on the basis of measurement data by the method for detecting surrounding environment information describing at least one parking space according to the invention.

[0060] The creation of the training environment map based on the training measurement data is in particular performed in a similar manner to the creation of the environment map based on the measurement data described above. The training measurement data are each identified in the training environment map with respect to the position of the vehicle. The training measurement data may have been detected at an earlier time point, for example by means of a sensor device of the vehicle comprising at least one environment sensor, and may then be used for training. Alternatively, training measurement data created and / or calculated differently may also be used.

[0061] The training environment maps are then each assigned a ground truth, where for at least a portion of the training environment maps the ground truth describes one or more parking spaces in the environment map. Additionally, in this case the ground truth also identifies at least one structural description for each parking space, which describes a structure that at least partially defines the parking space described by the ground truth. For further training environment maps that are not included in the at least a portion of the training environment maps, a ground truth that does not describe a parking space can be used. This makes it possible, for example, to improve the false positive rate of parking space identification.

[0062] The ground truth describing at least one parking space may also include further parameters of the parking space to be detected by the neural network, such as the location of the parking space localization and / or the relative orientation of the parking space with respect to the vehicle described in the training environment map, in which case the ground truth represents the results based on which the evaluation of the neural network is optimized during training.

[0063] The training environment map is then overlaid during training with an evaluation grid, which may also be referred to as a training evaluation grid with respect to training the neural network, each of which includes a number of cells, where each of the cells of the evaluation grid is assigned at least one frame or at least one box, where the frames represent an initial estimate of possible parking spaces.

[0064] Then, from the frames in the evaluation grid, a frame is selected that has a maximum overlap with the parking space described by the ground truth, in which, in particular, for each parking space described in the ground truth, a frame that has a maximum overlap is determined, where the overlap, which may also be referred to as Intersection over Union (IoU), specifies to what extent each frame in the evaluation grid overlaps with one of the parking spaces described by the ground truth.

[0065] After detecting each frame with the maximum overlap with one or more of the parking spaces described by the ground truth, the neural network is optimized with respect to at least one structural description of the frame with the maximum overlap and the structural description described by the ground truth. The structural descriptions of the frames that do not have the maximum overlap may be discarded or left empty, respectively, because the frames are not assigned with a parking space, and therefore no structural description exists in the ground truth.

[0066] The neural network may now also be optimized with respect to the correspondence in relative positioning between each frame with the maximum overlap and the parking space described by the ground truth and assigned to the frame. For example, the ground truth may also include the orientation of the parking space identified by the ground truth, which allows for the optimization of the orientation detected by the neural network of one of the frames accordingly. In this way, the neural network can be efficiently trained for the identification of the parking space and the surroundings that respectively define the parking space in the surrounding environment map, in particular for use in the method of detecting surrounding environment information describing at least one parking space according to the invention.

[0067] According to the invention, it may be arranged to assign to each cell a number of differently positioned and / or differently oriented frames and / or to determine in which cell the center point of at least one parking space described by the ground truth is located, respectively, in order to detect the overlap, and to select as the frame with the maximum overlap the frame of the cell with the maximum overlap with the parking space described by the ground truth. Firstly, once it is detected in which cell of the evaluation grid the center point of the parking space described by the ground truth is located, the number of frames in which the respective overlap is detected can be reduced, which requires fewer calculation steps and thus allows efficient training.

[0068] In a preferred embodiment of the invention, an evaluation grid may be used such that each cell is assigned a plurality of differently positioned and / or differently oriented frames and / or a classification of parking spaces may be configured to include a classification as a parallel parking space and a classification as a parallel parking space.

[0069] During training as well as during detection of the surrounding environment information describing at least one parking space, multiple frames can be used per cell of the evaluation grid, each of which may in particular be rectangular and located at different positions and / or orientations relative to the edges of the rectangular cell of the evaluation grid, which can be used, for example, to distinguish between parallel and parallel parking spaces and thus also to train the neural network with respect to the corresponding classification or distinction.

[0070] According to the invention, at least one contour line is used as the ground truth structural description, the contour line being formed by one or more line segments and / or two or more points, and / or the ground truth structural description may be configured to describe at least one edge of the object defining the parking space facing the vehicle, at least one edge of the object defining the parking space defining the parking space in the length direction and / or width direction, at least one edge of the object defining the entrance to the parking space, at least one edge of the object facing the entrance and / or at least one parking spot mark. In this way, the neural network can also be trained on a preferred structural description in the form of a contour line, and a precise indication of the relevant structural description in the surrounding environment information can be achieved for a subsequent use of the surrounding environment information, in particular for the parking process.

[0071] In a preferred embodiment of the invention, the ground truth may comprise at least one parking space type classification identifying, for at least a portion of the classified parking spaces, parking area information describing the arrangement and / or orientation of the parking areas available in the parking space and / or at least one parking space type classification, and the optimization of the neural network may be configured to also be performed with respect to the parking area information and / or the parking space classification. In this way, the neural network may advantageously be configured to detect the parking area information and / or the parking space classification as part of the surrounding environment information based on the surrounding environment map.

[0072] The invention also relates to a sensor device comprising at least one surrounding environment sensor and a computing device, the computing device being configured to perform the method of the invention for detecting surrounding environment information describing at least one parking space.

[0073] Furthermore, the invention relates to a vehicle equipped with at least one sensor device according to the invention.

[0074] The invention also relates to a computer program product comprising instructions for causing a computing device to perform the method according to the invention for detecting surrounding environment information describing at least one parking space.

[0075] Furthermore, the invention relates to a computer readable storage medium comprising a computer program product according to the invention. The computer readable storage medium may for example be a non-transitory recording medium, such as a CD, a DVD, a floppy disk, a flash memory or the like.

[0076] All advantages and features described above for the method according to the present invention for detecting environmental information describing at least one parking space are also valid for the method according to the present invention for training a convolutional neural network and vice versa.

[0077] Furthermore, all advantages and configurations described with respect to one of the methods according to the invention are also valid for the sensor device according to the invention, the vehicle according to the invention, the computer program product according to the invention and the computer-readable storage medium according to the invention, and vice versa.

[0078] Further advantages and details of the invention emerge from the embodiment examples described below and from the figures, which are schematic and are as follows: [Brief description of the drawings]

[0079] [Figure 1] FIG. 1 shows an embodiment of a vehicle according to the invention comprising an embodiment of a sensor device according to the invention. [Diagram 2] FIG. 2 is a diagram of a surrounding environment map illustrating an example embodiment of a method for detecting surrounding environment information describing at least one parking space according to the present invention. [Diagram 3] FIG. 3 is a diagram of a surrounding environment map overlaid with an evaluation grid for illustrating an example embodiment of a method for detecting surrounding environment information describing at least one parking space according to the present invention and an example embodiment of training a neural network. [Figure 4] FIG. 4 is a diagram of a parking space and multiple structural descriptions assigned to the parking space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0080] An example embodiment of a vehicle 1 according to the invention is shown in Fig. 1. The vehicle 1 may be, for example, a motor vehicle, such as a car or a truck. The vehicle 1 may also be, for example, a combination vehicle comprising a towing vehicle and one or more trailers. Alternatively, the method may also be used with further types of vehicles.

[0081] The vehicle 1 is equipped with a sensor device 2, which includes a number of surrounding environment sensors 3 and a computing device 4. The surrounding environment sensors 3 of the sensor device 2 are configured, for example, as ultrasonic sensors. Additionally or alternatively, at least some of the surrounding environment sensors 3 may also be configured as surrounding environment cameras, as radar sensors and / or as lidar sensors.

[0082] In this example embodiment, the computing device 4 of the sensor device 2 is shown as part of the vehicle 1. The computing device 4 may be integrated in one of the surrounding environment sensors 3 or may be a computing device external to the vehicle, in which case the computing device external to the vehicle may communicate with the surrounding environment sensor 3 and / or further devices of the vehicle 1, in particular via a wireless communication link.

[0083] The surrounding environment sensors 3 can be used to detect measurement data based on the surrounding environment of the vehicle 1. The measurement data describes objects present in the surrounding environment of the vehicle 1. The objects can be, for example, other traffic participants, such as vehicles. Also non-moving objects can be detected as objects, such as infrastructure objects, parts of buildings, etc.

[0084] The computing device 4 of the sensor device 2 is configured to execute a method for detecting surrounding environment information describing at least one parking space in the surrounding environment of the vehicle 1. Depending on the surrounding environment, for example an automatic or partially automatic control of the vehicle 1 can be performed, in particular by controlling at least one actuator (not shown) of the vehicle 1. During the entry parking process, the vehicle 1 can move forward and / or backward, can perform steering movements and in particular also perform multiple partial movements or multiple movements. In this case, the entry parking process can in particular be performed in such a way that the vehicle is positioned in the parking space described by the at least one surrounding environment information.

[0085] Additionally or alternatively to the actuators, a display device (not shown) of the vehicle 1 can be operated, for example to make the driver of the vehicle 1 aware of the detected parking space in the vehicle's environment, for example by means of a corresponding graphical representation. Here, the display device can be, for example, a display arranged in the interior space of the vehicle 1, a head-up display, a virtual side mirror, etc.

[0086] The representation on the display device may be used, for example, to enable the driver to select one of the detected parking spaces as a destination for the entrance parking process. Additionally or alternatively, assistance means for moving the vehicle 1 to the detected and / or selected parking space may be displayed on the display device, which assistance means enables the driver to move the vehicle 1 to this parking space.

[0087] When detecting ambient environment information using the computing device 4 or in a computer-implemented method, firstly, measurement data describing the ambient environment or surroundings of the vehicle 1 are detected using one or more ambient environment sensors 3, and this measurement data is assigned to one or more objects in the ambient environment of the vehicle 1. Then, based on the measurement data, at least one ambient environment map 5 describing the ambient environment of the vehicle 1 is created, and the measurement data is spatially located in the ambient environment map 5 with respect to the vehicle 1.

[0088] An example embodiment of such an environment map 5 is shown in Fig. 2. The environment map 5 is a rectangular map in which the vehicle 1 is located in the center. In the environment map 5, the measurement data are spatially determined with respect to the vehicle 1. The measurement data are shown present as measurement points 6, which may be detected, for example, by means of the environment sensor 3 configured as an ultrasonic sensor. Here, the measurement points 6 of the measurement data each describe a signal echo reflected by an object in the environment of the vehicle 1.

[0089] The individual measurement points 6 each comprise a distance measurement relative to the vehicle 1, which may for example be based on a time-of-flight measurement. Additionally, a direction may also be determined for each measurement point 6, so that the reflection position, i.e. the object, assigned to the measurement point 6 can be spatially determined relative to the vehicle 1.

[0090] The surrounding environment map 5 further comprises a map grid 7, and the measurement data are assigned to individual cells 8 of the map grid 7. The larger the number of cells 8, the better the spatial resolution of the surrounding environment information. On the other hand, an increasing number of cells 8 also requires a larger memory to process the surrounding environment map 5.

[0091] For each cell 8, for example the number of measurement points 6 of measurement data assigned to that cell 8 may be determined and stored in the surrounding environment map 5. Also, for each individual cell 8, the height of an object described on the basis of the measurement data assigned to the cell 8, the type of such object and / or at least one further signal characteristic of the measurement data assigned to the cell may be determined.

[0092] The signal characteristic may for example relate to the intensity of the individual measurement points 6. In this case, for example, it is possible to determine a signal characteristic, for example the intensity, detected for all measurement points of one cell 8. Correspondingly, this may also be done for information regarding the object height and / or object type.

[0093] In addition to the measurement points 6 by the surrounding environment sensor 2 configured as an ultrasonic sensor, the measurement points 6 by the surrounding environment sensor 3 configured as a radar sensor and / or a lidar sensor may correspondingly also be input to the surrounding environment map 5. Objects detected via other types of sensors, for example via the surrounding environment sensor 3 configured as an surrounding environment camera, may also be input to the surrounding environment map 5. In this way, objects, for example lane markings, parking spot marks, etc. may be input to the surrounding environment map 5.

[0094] The computing device 4 then evaluates at least one surrounding environment map 5, which is fed to a convolutional neural network implemented by the computing device 4, which classifies and localizes the parking spaces 9 present in the surrounding environment of the vehicle 1 based on the surrounding environment map 5, and which detects for each of the parking spaces 9 at least one structural description of at least a part of the structure defining the parking space 9 based on the surrounding environment map 5. Based on the classified parking spaces 9 and the at least one structural description, the computing device 4 or a further computing device (not shown) then generates surrounding environment information describing the at least one parking space 9 in the surrounding environment of the vehicle 1. Advantageously, the surrounding environment map 5, for example two-dimensional image data, can then be processed by means of a neural network.

[0095] A neural network is configured to overlay an evaluation grid 11 comprising a number of cells 10 onto the environment map 5, with each cell 10 being assigned at least one frame 12. The environment map 5 with the overlaid evaluation grid 11 is shown in Figure 3, where the map grid 7 of the environment map 5 is not shown for clarity.

[0096] In this example, each cell 10 comprises three respective rectangular frames 12, 13, 14. Frames 12, 13 are rotated by 90° with respect to frame 14, so that frames 12, 13 are also oriented differently with respect to cell 10 than frame 14. In this example, frames 12, 13 each enable the detection of a parallel parking space, while frame 14 enables the detection of a parallel parking space for vehicle 1. Here, when using three frames 12, 13, 14, it is taken into account that in general two parallel parking spaces can be located over the length of a parallel parking space.

[0097] Each cell 10 may also be assigned a further frame (not shown), which may, for example, have the same orientation as one of the frames 12, 13, 14 but be offset in the x and / or y directions along one of the sides of the cell 10.

[0098] In the surrounding environment information, for each of the frames 12, 13, 14 of all cells 10 of the evaluation grid 11, a classification as a parking space is identified. In particular, only for the frames 12, 13, 14 classified as parking spaces 9, at least one characteristic description of at least a part of the structure defining the parking space 9 is additionally identified. Additionally, for each frame 12, 13, 14 classified as a parking space, an orientation of the parking space assigned to the frame 12, 13, 14, described on the basis of measurement data, can be identified with respect to the frame 12, 13, 14. This orientation can be detected by a neural network, for example by means of regression.

[0099] In Fig. 4, a parking space 9 is shown with multiple structural descriptions 15-18 assigned to the parking space 9. The structural descriptions 15-18 each describe a different structure that partially defines the parking space 9, the structure being in terms of objects and marks. A neural network detects the structural descriptions 15-18 based on the surrounding environment map 5.

[0100] Starting from the scanning direction 19, in which the vehicle 1 records the measurement data by means of its surrounding environment sensors 3, for example, measurement data accumulation occurs in each of the first and second areas 20 and 21. In this case, the measurement data each result from an object which laterally defines the parking space 9. Furthermore, measurement data may also be present in the third area 22, which result from one or more objects which define the parking space 9 on the opposite side 24 to the entrance 23.

[0101] Contours are used as structural descriptions 15-18, each of which includes one or more line segments and / or is formed by two or more points. Here, structural descriptions 15a, 15b each describe, for example, an edge of an object defining a parking space facing vehicle 1, an edge of the object defining the parking space in the width direction, and an edge of the object defining an entrance 23 of parking space 9. In this case, each of these edges of the object is described by a line segment. Thus, the contour is a polyline, which may include a plurality of interconnected line segments.

[0102] The objects described at least partially by the structural descriptions 15a, 15b, respectively, can be other vehicles, for example. The different line segments detected as part of the structural descriptions 15a, 15b can be used to enter the parking space 9 with high accuracy based on the surrounding environment information, since, with respect to the trajectory planning, the extensions of the outer edges of the objects are precisely described, in particular in the region of the entrance 23.

[0103] The structural descriptions 16a, 16b each describe a parking spot mark that indicates an provided parking spot. The structural descriptions 16a, 16b may be determined on the basis of measurement data recorded by means of an environmental sensor 3 of the vehicle 1, configured as a camera, for example by means of a neural network.

[0104] The structure descriptions 17a, 17b each describe a structure which laterally defines an entrance 23 of the parking space 9. Furthermore, the structure descriptions 17a, 17b each include a description of an edge of the structure facing the vehicle 1 and of a further edge of the structure which runs in the length direction of the parking space 9. The structure described by the structure descriptions 17a, 17b may for example be a pillar, a column or a large stone.

[0105] Corresponding structural descriptions 17c, 17d can also be found for the structure located on the opposite side 24 of the parking space 9 to the entrance.

[0106] On the opposite side 24 to the entrance 23, the structural descriptions 18a, 18b, 18c each describe a structure which delimits the parking space 9 opposite the entrance 23. Such structures may be, for example, a curb, an outer wall, an inner wall, etc. Here, the structural descriptions 18a, 18b, 18c may each also be assigned height information in the surrounding environment information, so that several consecutively located objects can be differentiated by means of their height differences. This makes it possible, based on the surrounding environment information, for example, to plan for partially climbing over such objects and / or for at least partially climbing over such objects with the vehicle front.

[0107] The structural descriptions 15-18 allow the neural network to detect only a portion of each classified parking space 9, and can also be supplemented with further structural descriptions, which advantageously cover the structures relevant for planning the parking operation, i.e. the movement of the vehicle 1 into the parking space 9, in the surrounding environment of the parking space 9 for a number of different scenarios.

[0108] Here, it is not necessary that each structural description is available for each classified parking space 9. Depending on the detected scenario, a small number of objects and / or marks may be available. For example, the availability of all or at least some of said structures may be checked by a neural network, which identifies the presence of the corresponding structures and thus the validity of the structural description describing the structures, respectively, based on the detection probability and / or a value derived from the detection probability.

[0109] The neural network may also include parking area information 25, shown only diagrammatically in this embodiment, identifying an area available for parking the vehicle 1. This parking area information 25 may be detected based on an orientation generated by the neural network and assigned to each frame 12, 13, 14 and / or a scaling generated by the neural network and assigned to each frame 12, 13, 14. The area illustrated in Fig. 4 can for example be used when the object described by the structural descriptions 17a-17d is not available.

[0110] The structural descriptions 15 to 18 may be replicated in the surrounding environment information based on the data structure shown in Table 1, for example.

[0111] [Table 1]

[0112] Here, the line "ParkingBox (PB)" indicates the vertices (x1, y1); (x2, y2) etc. of the parking area information 25. For the structural description, the lines "FirstObject (FO)" and "SecondObject (SO)" thus duplicate the defined points of the structural descriptions 15a, 15b, respectively, which are constructed as contour lines. The lines "FirstParkMarker (FPM)" and "SecondParkMarker (SPM)" thus contain the coordinates of the end points of the parking spot marks of the structural descriptions 16a, 16b.

[0113] For example, the rows "CurbObject1 (CO1)" and "CurbObject2 (CO2)" respectively show two vertices of the structural descriptions 18a, 18b, and 18c. The rows "FirstObjectBoulders (FOB)" and "SecondObjectBoulders (SOB)" show, for example, two of the structural descriptions 17a-17d. Further structural descriptions 17a-17d may also be shown in the surrounding environment information, or only two of the structural descriptions 17a-17d may be used, since a parking space is generally defined only by two pillars or the like, and the rows "FirstObjectBoulders (FOB)" and "SecondObjectBoulders (SOB)" have, for example, structural descriptions 17a and 17b or 17c and 17d duplicated therein, depending on the scenario.

[0114] Here, the values ​​of the structural descriptions 15-18 may each be set as a percentage with respect to the cells 10 of the evaluation grid 11 such that the values ​​are between 0 and 1. In this case, the values ​​may each be set with respect to a reference position, for example with respect to the center point of the parking area information 25, in particular with respect to the vertices and / or other reference points of the parking area information 25 describing a rectangular box. Such setting of the coordinates of the structural descriptions 15-18 allows setting small values ​​between 0 and 1, which in particular facilitates training of the neural network with the corresponding training data. In this case, it is possible to avoid the occurrence of gradients with too steep an ascent during training and / or the occurrence of untrained nodes in the network.

[0115] Each of the structure descriptions 15-18 and the parking area information 25 is assigned a probability P in the surrounding environment information, which indicates the probability of availability of each structure. Alternatively, the value P may also be set to 1 or 0, for example, if the calculated probability exceeds or is below a predefined threshold, respectively.

[0116] For each frame 12, 13, 14 classified as a parking space, the surroundings information may additionally or alternatively specify an orientation with respect to the parking area information 25, which specifies the angle between the respective frame 12, 13, 14 and the actual parking area within the parking space 9, which is described in the surroundings map on the basis of the measurement data. In this way, in addition to the arrangement of the parking space 9 in the vertical and horizontal directions, all further orientations can also be taken into account and shown. Additionally, a scale factor can then be set for showing the frame 12, 13, 14 in the actual parking space.

[0117] For each frame 12, 13, 14 classified as a parking space 9, the surrounding environment information may further include one or more parking space type classifications, which may indicate, for example, whether or how likely it is for a parallel or parallel parking space. A parking space 9 may be classified as such if the classification probability calculated by the neural network exceeds a predefined threshold. Thus, a parking space may be classified as a parallel parking space or as a parallel parking space if the classification probability assigned to the corresponding one of the frames 12, 13, 14 exceeds a predefined threshold. The parking space type classification may further indicate, for example, whether it is for a parallel parking space to be driven in a forward or reverse direction.

[0118] The generated ambience information may for example be a tensor indicating for each frame 12, 13, 14 whether it concerns a parking space 9 or whether it is a parallel or parallel parking space. For each frame 12, 13 classified as a parking space 9, further information may be set. For each cell 10 of the evaluation grid 11, the ambience information may then be, for example, a vector y cell may include the vector y cell teeth, y cell =[P1<parkingslotparameters>C par1 C per_fw1 C per_bw1 C void P2...P3...] T where values ​​assigned to frame 12 are indicated, for example, with a subscript "1", values ​​assigned to frame 13, most of which are not shown for the sake of clarity, with a subscript "2", and values ​​assigned to frame 14, most of which are not shown for the sake of clarity, with a subscript "3", etc. The individual values ​​or components of the vector are written as follows: P i : an indicator as to whether the corresponding frame 12, 13, 14 relates to a parking space 9 (e.g., 0 or 1) <Parking Slot Parameters> : The structural descriptions 15 to 18 and parking area information 25 shown in Table 1, for example, belong to the parking space assigned to each frame. C par : Class probability that a parallel parking space exists C per_fw : Class probability that there is a parallel parking space in the forward direction C per_bw : Class probability that there is a parallel parking space in the backward direction C void : Class probability that a parking space does not exist.

[0119] For example, indicator P is a probability C par ,C per_fw or C per_bw may be set to 1 if one of the probability C void may be set to 0 if exceeds a certain threshold.

[0120] For example, the surrounding environment information output in tensor format is A×B×(y cell ) or A×B×(n*i), where A and B denote the number of rows and columns of the evaluation grid, and thus their product denotes the number of cells in the evaluation grid, and y cell are vectors assigned to each cell, and y cell In general, y has n*i components, where n is the number of frames per cell and i is the number of pieces of information detected in a cell per frame. cell For example, n=3 and i=65.

[0121] In an example embodiment of a method for training a convolutional neural network, first at least one training data set is provided, each of which includes a plurality of training environment maps based on training measurement data, each of which describes the environment of the vehicle 1, where the measurement data is spatially located in the training environment maps with respect to the vehicle 1. The training method is particularly useful for training a convolutional neural network implemented in a computing device 4 for carrying out the example embodiment described above.

[0122] Each training environment map is assigned a ground truth describing at least one parking space 26 in the environment of the vehicle 1, the ground truth classifying and locating at least one section of the training environment map as a parking space 26. The ground truth also identifies structural descriptions 15-18 associated with the immediate surroundings of the parking space 26. It is also conceivable that only a portion of the training environment map is assigned a ground truth describing at least one parking space 26, while other portions of the training environment map are each assigned a ground truth that does not, for example, describe the parking space 26. In this manner, the false positive rate of parking space identification can be improved or reduced.

[0123] In FIG. 3, a parking space 26 set by ground truth is shown. The training environment map basically corresponds to the environment map 5 described above. The convolutional neural network can be trained, for example, using a Dempster-Shafer grid map created on the basis of radar data as the training environment map, so that the method described above can be used to detect parking spaces on the basis of radar data. Additionally or alternatively, a training environment map created on the basis of measurement data from an ultrasonic sensor, a lidar sensor, an environment camera or any combination of these sensor types can also be used. In this case, basically each type of training environment map can be used, including occupied and empty areas.

[0124] Also, during training, an evaluation grid 11, each including a number of cells 10, is superimposed on the training environment map shown in Fig. 3 above, and each cell 10 of the evaluation grid 11 is assigned at least one frame 12, 13, 14 corresponding thereto. Then, each frame 12, 13, 14 with a maximum overlap is found with respect to at least one parking space 26 described by the ground truth. Then, the neural network is optimized for a match between the structural description of at least one of the frames 12, 13, 14 with the maximum overlap and the structural description described by the ground truth.

[0125] To detect overlaps, it is determined in which cell 10 of the evaluation grid 11 the center point 27 of at least one parking space 26 described by the ground truth is located, which in the example shown in FIG.

[0126] Then, the frame 12, 13, 14 of the cell that has the greatest overlap with the parking space 26 described by the ground truth is selected as the frame with the greatest overlap. To find each overlap, for each frame 12, 13, 14 of the cell 10 in which the center point 27 is located, the overlap or its IoU (Intersection over Union) value for its area is calculated, respectively.

[0127] In the example shown in Fig. 3, the overlap with the right-hand vertically oriented frame 13 is the largest, and so it is selected as the optimization target of the neural network. Thus, based on the selection of frames 12, 13, 14 and the arrangement of the vehicle 1 in the surrounding environment map 5, it can be detected or trained whether it concerns a parallel parking space or a parallel parking space as described above. Furthermore, the arrangement between the parking space 26 and the vehicle 1, if classified as a parallel parking space, can also be detected or trained as to the entrance parking direction, i.e. whether the vehicle 1 should drive forward or backward into the parking space 26.

[0128] Here, at least one contour may be used as the ground truth structural description, which as described above is formed by one or more line segments and / or two or more points. The ground truth structural description may describe at least one edge of an object defining the parking space 26 facing the vehicle 1, at least one edge of an object defining the parking space 26 defining the parking space 26 in the length direction and / or width direction, at least one edge of an object defining the entrance 23 of the parking space 26, at least one edge of an object facing the entrance 23 and / or at least one parking spot mark.

[0129] The evaluation grid 11 can be used such that each cell 10 is assigned three or more frames 12, 13, 14. In this case, the frames 12, 13, 14 can be positioned differently and / or oriented differently. Different measurements for the cells 10 can then be trained as well, such as different widths, different lengths, different positions of the center points of the frames 12 and the parking spaces 26, the associated orientation between the frames 12 and the parking spaces 26, which can also be detected by the neural network using regression.

[0130] The orientation of the parking space 26 can also be approximated using the frames 12, 13, 14, and differently positioned and / or differently oriented localization and / or orientation parameters can be initialized using two or more frames 12, 13, 14 per cell 10. In this regard, the more frames 12, 13, 14 are used, the more accurately the neural network can localize and detect the relative arrangement of the parking space 26 with respect to the vehicle, detect parking area information 25 describing vacant parking areas in the parking space, and detect a structure description describing a structure that at least partially defines the parking space 26.

[0131] The optimization of the neural network can be performed using a loss function, which can be found for frames 12, 13, and 14, the vector y cell In addition, or alternatively, a softmax function can be used to minimize the squared error for all variables in , and the class probabilities Cpar, Cperp fw , Cperp_bw, Cvoid and / or classification probability P n Only the error in can be minimized.

[0132] Loss function

[0133]

number

[0134] If y1=1, it means that frame 12, which is assigned the subscript “1”, is classified as a parking space. In this case, the squared error loss function is used to calculate the squared error loss function for each variable i, i.e., the vector y cell Each component y n If y1=0, it means that frame 12, which is assigned the subscript “1”, is classified as “non-parking space”, and in this case, the indicator or classification probability P n That is, the vector y cell This is then carried out for further frames 13 and 14. This application relates to the invention described in the claims, but also includes the following as other aspects. 1. A method for detecting surrounding environment information describing at least one parking space (9) using a sensor device (2) of a vehicle (1), the sensor device (2) comprising at least one surrounding environment sensor (3) and a computing device (4), the method comprising: - recording measurement data describing the surrounding environment of the vehicle (1) by means of the at least one surrounding environment sensor (1), the measurement data being assigned to one or more objects in the surrounding environment of the vehicle (1); - creating at least one environment map (5) describing the environment of the vehicle (1) based on the measurement data, the measurement data being spatially identified in the environment map with respect to the vehicle (1); - evaluating the at least one environment map (5) by the computing device (4), the environment map (5) being fed to a convolutional neural network, the neural network classifying and localizing parking spaces (9) present in the environment of the vehicle (1) based on the environment map (5), the neural network detecting for each of the parking spaces (9) at least one structural description (15-18) of at least a part of a structure defining the parking space (9) based on the environment map; - generating said surrounding environment information based on said classified and located parking spaces (9), said surrounding environment information of said at least one structural description (15-18). 2. 2. The method according to claim 1, characterized in that the surrounding environment map (5) comprises a map grid (7), the measurement data are assigned to cells (8) of the map grid (7), and for each cell (8) the number of measurement points (6) of the measurement data assigned to the cell (8), the height of an object described on the basis of the measurement data assigned to the cell (8), the type of object described on the basis of the measurement data assigned to the cell (8) and / or at least one signal characteristic of the measurement data assigned to the cell (8) are identified. 3. 3. The method according to claim 1 or 2, characterized in that the neural network is configured to overlay on the surrounding environment map (5) an evaluation grid (11) comprising a plurality of cells (10), each of which is assigned at least one frame (12, 13, 14), and in the surrounding environment information, for each of the frames (12, 13, 14) a classification as a parking space, an orientation of the parking space assigned to said frame (12, 13, 14) relative to said frame (12, 13, 14) described on the basis of the measurement data and / or at least one structural description (15-18) of a structure defining said parking space (9) are identified. 4. The method according to any one of 1 to 3 above, characterized in that at least one contour line is used as the structural description (15 to 18), said contour line being formed by one or more line segments and / or two or more points. 5. 5. The method according to any one of claims 1 to 4, characterized in that the structural description (15 to 18) describes at least one edge of an object defining the parking space (9) facing the vehicle (1), at least one edge of an object defining the parking space (9) defining the parking space (9) in the length direction and / or in the width direction, at least one edge of an object defining an entrance (23) of the parking space (9), at least one edge of an object facing the entrance (23) and / or at least one parking spot mark. 6. 6. The method according to any one of 1 to 5 above, wherein the surrounding environment information includes, for at least a portion of the classified parking spaces (9), parking area information (25) describing the layout and / or orientation of the parking areas available in the parking spaces and / or at least one parking space type classification identifying at least one parking space type classification. 7. The method according to any one of 1 to 6 above, characterized in that as the at least one surrounding environment sensor (3), an ultrasonic sensor device having a plurality of ultrasonic sensors, at least one lidar sensor, at least one radar sensor and / or a sensor device (2) having at least one surrounding environment camera is used. 8. In particular, a method for training a convolutional neural network for use in a method according to any one of 1 to 7 above, comprising: - providing at least one training data set including a plurality of training environment maps each describing an environment of a vehicle (1) based on training measurement data, said training measurement data being spatially identified in said training environment map with respect to said vehicle (1); - assigning a ground truth describing at least one parking space (26) in the vehicle's (1) environment to at least a portion of the training environment map, the ground truth classifying and localizing at least one section of the training environment map as a parking space (26), the ground truth for the parking space (26) identifying at least one structural description (15-18) of at least a portion of a structure defining the parking space (9); - a step of superimposing on said training environment map evaluation grids (11), each of which includes a plurality of cells (10), wherein each cell (10) of said evaluation grid (11) is assigned at least one frame (12, 13, 14), and each frame is assigned at least one structural description (15-18); - detecting, with respect to said at least one parking space (26) described by said ground truth, each of said frames (12, 13, 14) having a maximum overlap; - optimizing the neural network with respect to a match between the at least one structural description (15-18) of the frame (12, 13, 14) having the maximum overlap and a structural description (15-18) described by the ground truth. 9. 9. The method according to claim 8, characterized in that each cell (10) is assigned a plurality of differently positioned and / or differently oriented frames (12, 13, 14) and / or in determining in which of the cells (10) is located a center point (15) of each of the at least one parking space (26) described by the ground truth in order to detect an overlap, and in selecting the frame (12, 13, 14) of the cell (10) having the maximum overlap with the parking space (26) described by the ground truth as the frame (12, 13, 14) having the maximum overlap. 10. 10. The method according to claim 8 or 9, characterized in that at least one contour line is used as the structural description (15-18) of the ground truth, the contour line being formed by one or more line segments and / or two or more points, and / or the structural description (15-18) of the ground truth describes at least one edge of an object defining the parking space (26) facing the vehicle, at least one edge of an object defining the parking space (26) in the length direction and / or width direction, at least one edge of an object defining an entrance (23) of the parking space (26), at least one edge of an object facing the entrance and / or at least one parking spot mark. 11. 11. The method according to any one of claims 8 to 10, characterized in that the ground truth comprises, for at least a portion of the classified parking spaces (26), parking area information (25) describing the arrangement and / or orientation of the parking areas available in the parking spaces (26) and / or at least one parking space type classification identifying at least one parking space type classification, and optimization of the neural network is also performed with respect to the parking area information (25) and / or the parking space classification. 12. A sensor device comprising at least one surrounding environment sensor (3) and a computing device (4), said computing device (4) being configured to perform the method according to any one of 1 to 7 above. 13. A vehicle comprising at least one sensor device (2) as described above in 12. 14. A computer program product comprising instructions for causing a computing device (4) to carry out the method according to any one of 1 to 7 above. 15. 15. A computer readable storage medium comprising a computer program product as recited in claim 14. < / parkingslotparameters>

Claims

1. 1. A method for detecting surrounding environment information describing at least one parking space (9) using a sensor device (2) of a vehicle (1), said sensor device (2) comprising at least one surrounding environment sensor (3) and a computing device (4), comprising: - recording measurement data describing the surrounding environment of the vehicle (1) by means of the at least one surrounding environment sensor (1), said measurement data being assigned to one or more objects in the surrounding environment of the vehicle (1); - creating at least one environment map (5) describing the environment of the vehicle (1) based on said measurement data, said measurement data being spatially identified in said environment map with respect to said vehicle (1); - evaluating said at least one environment map (5) by said computing device (4), said environment map (5) being fed to a convolutional neural network, said neural network classifying and locating parking spaces (9) present in the environment of said vehicle (1) based on said environment map (5), said neural network detecting for each of said parking spaces (9) at least one structural description (15-18) of at least a part of a structure defining said parking space (9) based on said environment map; - generating said surrounding environment information according to said classified and located parking spaces (9) and said at least one structural description (15-18).

2. 2. The method according to claim 1, characterized in that the surrounding environment map (5) comprises a map grid (7), the measurement data are assigned to cells (8) of the map grid (7), and for each cell (8) the number of measurement points (6) of the measurement data assigned to the cell (8), the height of an object described on the basis of the measurement data assigned to the cell (8), the type of object described on the basis of the measurement data assigned to the cell (8) and / or at least one signal characteristic of the measurement data assigned to the cell (8) are identified.

3. 2. The method according to claim 1, characterized in that the neural network is configured to overlay on the environment map (5) an evaluation grid (11) comprising a plurality of cells (10), each of which is assigned at least one frame (12, 13, 14), and in the environment information a classification as a parking space, an orientation of the parking space described on the basis of the measurement data and assigned to said frame (12, 13, 14) with respect to said frame (12, 13, 14) and / or at least one structural description (15-18) of a structure defining said parking space (9) are identified for each of said frames (12, 13, 14).

4. 2. The method according to claim 1, characterized in that at least one contour line is used as the structural description (15-18), said contour line being formed by one or more line segments and / or two or more points.

5. 2. The method according to claim 1, characterized in that the structural description (15-18) describes at least one edge of an object defining the parking space (9) facing the vehicle (1), at least one edge of an object defining the parking space (9) defining the parking space (9) in the length direction and / or in the width direction, at least one edge of an object defining an entrance (23) of the parking space (9), at least one edge of an object facing the entrance (23) and / or at least one parking spot mark.

6. 2. The method of claim 1, wherein the surrounding environment information comprises, for at least a portion of the classified parking spaces (9), parking area information (25) describing a layout and / or orientation of the parking areas available in the parking spaces and / or at least one parking space type classification specifying a classification of at least one parking space type.

7. 2. The method according to claim 1, characterized in that as the at least one surrounding environment sensor (3), an ultrasonic sensor arrangement (2) with a plurality of ultrasonic sensors, at least one lidar sensor, at least one radar sensor and / or at least one surrounding environment camera is used.

8. A method for training a convolutional neural network, in particular for use in a method according to any one of claims 1 to 7, comprising: - providing at least one training data set comprising a plurality of training environment maps each describing an environment of a vehicle (1) based on training measurement data, said training measurement data being spatially identified in said training environment map with respect to said vehicle (1); - assigning a ground truth describing at least one parking space (26) in the vehicle's (1) environment to at least a portion of the training environment map, the ground truth classifying and locating at least one section of the training environment map as a parking space (26), the ground truth for the parking space (26) identifying at least one structural description (15-18) of at least a portion of a structure defining the parking space (9); - superimposing on said training environment map evaluation grids (11) each including a plurality of cells (10), each cell (10) of said evaluation grid (11) being assigned at least one frame (12, 13, 14) and each frame being assigned at least one structural description (15-18); - finding each of said frames (12, 13, 14) that has a maximum overlap with respect to said at least one parking space (26) described by said ground truth; - optimizing said neural network for a match between said at least one structural description (15-18) of said frame (12, 13, 14) having a maximum overlap and a structural description (15-18) described by said ground truth.

9. 9. The method according to claim 8, characterized in that each cell (10) is assigned a plurality of differently positioned and / or differently oriented frames (12, 13, 14) and / or that in order to detect an overlap, it is determined in which cell (10) a center point (15) of each of the at least one parking space (26) described by the ground truth is located, and the frame (12, 13, 14) of the cell (10) having the maximum overlap with the parking space (26) described by the ground truth is selected as the frame (12, 13, 14) with the maximum overlap.

10. 9. The method according to claim 8, characterized in that at least one contour line is used as the structural description (15-18) of the ground truth, the contour line being formed by one or more line segments and / or by two or more points and / or the structural description (15-18) of the ground truth describes at least one edge of an object defining the parking space (26) facing the vehicle, at least one edge of an object defining the parking space (26) defining the parking space (26) in a length direction and / or a width direction, at least one edge of an object defining an entrance (23) of the parking space (26), at least one edge of an object facing the entrance and / or at least one parking spot mark.

11. 9. The method according to claim 8, characterized in that the ground truth comprises, for at least a portion of the classified parking spaces (26), parking area information (25) describing a layout and / or an orientation of the parking areas available in the parking spaces (26) and / or at least one parking space type classification identifying at least one parking space type classification, and the optimization of the neural network is also performed with respect to the parking area information (25) and / or the parking space classification.

12. A sensor arrangement comprising at least one ambient environment sensor (3) and a computing device (4), said computing device (4) being configured to perform the method according to any one of the claims 1 to 7.

13. A vehicle comprising at least one sensor device (2) according to claim 12.

14. A computer program product comprising instructions for causing a computing device (4) to carry out the method according to any one of claims 1 to 7.

15. A computer readable storage medium comprising the computer program product of claim 14.

Images