Method and device for recognizing parking space of vehicle by means of machine learning method

Abstract

The invention relates to a computer-implemented method, in particular for detecting a parking space for a vehicle (1), comprising the following steps: providing a camera image (B); evaluating the camera image (B) by means of a data-based parking space recognition model (11), the parking space recognition model (11) being designed to assign a camera image to a representation of a parking space in the surroundings of the vehicle (1) detected by the camera image (B); wherein the representation of the parking space specifies a description of the orientation of the entry line (E) into the identified parking space in the vehicle coordinate system.

Description

Technical Field

[0001] The invention relates to a method for identifying a parking space or a parking gap for a vehicle based on an object recognition algorithm. Background Art

[0002] The automatically detected possible parking spaces for the vehicle are an essential prerequisite for downstream methods for the automatic parking of the vehicle.

[0003] The recognition of parking spaces is usually based on the evaluation of a camera image representing the vehicle's surroundings. For example, as is known from "Context-Based Parking Slot Detection With a Realistic Dataset", IEEE Access, Volume 8, 2020 by H. Do et al.: Based on a camera image of the surroundings, the coordinates of the four corner points of the parking space can be determined with the aid of a two-stage method. Here, the first stage serves to roughly identify the area where a parking space may be present, including identifying its orientation with the aid of a first neural network, while the second stage serves to identify the parking space in detail within the approximate area of ​​the possible parking space specified by the first stage. The output of this second stage corresponds to the four corner points of the identified quadrilateral parking space, which improves the previous recognition method that can determine the rectangular parking space area and specify the width, height and center coordinates for it.

[0004] However, the disadvantage of the two-stage design of parking space recognition according to the above prior art is that the artificial neural network of the multi-stage design often loses the acquired information or does not pass it to the next stage. Complex end-to-end training across the two neural networks can alleviate this problem, but it will also cause the errors or inaccuracies of the first stage to be amplified in the subsequent stages.

[0005] The publication DE 102010047162 A1 discloses a method for identifying free parking spaces by analyzing image data recorded in a vehicle by at least one image detection device, comprising the steps of: detecting and storing an input image of the lateral surroundings of a vehicle during the movement of the vehicle; determining and tracking reference points by analyzing the detected input image; reconstructing a 3D model of the lateral surroundings of the vehicle in the form of a 3D point cloud by using the reference points; and analyzing the 3D point cloud to identify free parking spaces. In order to be able to select a suitable free parking space particularly easily and quickly, the 3D point cloud of the lateral surroundings of the vehicle is output as an output image, wherein a dimensionally accurate 3D model of the vehicle is inserted into the output image at the location of the mapping of the identified free parking space.

[0006] The publication DE 112020000369 T5 discloses a method for identifying a parking space by means of a neural network, wherein data representing displacement values ​​for corner points of an anchor shape are determined from image data representing the parking space by using a neural network. Corner points of a warped polygon are determined from the displacement values ​​to the corner points of the anchor shape, and first distances between the corner points of the warped polygon and ground truth corner points of the parking space are calculated. Summary of the invention

[0007] According to the invention, a method for marking a parking space, in particular for a subsequent automatic parking method, is provided according to claim 1 , and a corresponding device according to the independent parallel claim.

[0008] Further refinements are specified in the dependent claims.

[0009] According to a first aspect, a method for identifying a parking space for a vehicle is provided, the method comprising the following steps:

[0010] -Provide camera images;

[0011] - evaluating the camera image with the aid of a data-based parking space recognition model, wherein the parking space recognition model is designed to assign the camera image to a representation of a parking space in the surroundings of the vehicle captured by the camera image, wherein the representation of the parking space specifies a description of the position of an entry line into the recognized parking space in a vehicle coordinate system.

[0012] Evaluating camera images to identify parking spaces in the vehicle's surroundings is crucial for automated parking methods. For this purpose, object recognition algorithms based on convolutional neural networks are often used, for example in the form of the YOLOv3 method.

[0013] For parking space recognition, as mentioned at the outset, multi-stage recognition methods are proposed in the prior art, which provide a first neural network for context recognition and search range limitation and a further neural network for actually identifying the position of the parking space. The above-mentioned disadvantages of such multi-stage systems for parking space recognition during the training process can be overcome by the above-mentioned method in that, on the one hand, a single-stage neural network is implemented, which directly outputs parameters for defining a parking space based on the camera image, which performs parking space recognition based on the camera image data. On the other hand, the parameters for defining a parking space are selected in such a way that they represent a particularly suitable parking space representation, which is used, for example, for a subsequent automatic parking method based on this.

[0014] Although it is known from the prior art to identify a parking space by the four corner points of a quadrilateral parking space area, the above method applies a parking space representation which is characterized by the definition of the orientation of the parking space entry line. Further instructions defining the parking space can then be oriented at the entry line.

[0015] Furthermore, the orientation of the entry line can be specified as the length and orientation of the distance vector between the center of the entry line and the angle of the entry line relative to the center of the grid cell in the grid line network, which places the object recognition algorithm on the camera image. In particular, the orientation of the entry line can also be specified by the width of the entry line.

[0016] The entry line can be determined by the distance and direction relative to the center of the grid cell with which the area detected by the camera image is gridded, and by the width of the detected entry line.

[0017] The parking space recognition model can be designed with a neural network or other data-based model suitable for image evaluation and in particular has a YOLOv3 architecture, which locates the recognized parking spaces in a grid line network.

[0018] It may be provided that the representation of the parking space is further specified by lateral boundary lines, the position and orientation of which are defined based on the position of the access line, in particular by the length of the respective lateral boundary line and the lateral boundary line angle to the access line.

[0019] The above representation of the parking space as a positioning result of the parking space recognition model is advantageous because such a representation of the parking space is directly based on the properties of the parking space. Therefore, these properties of the parking space do not need to be determined using the four corner points of the parking space as in the examples described in the prior art. It is thereby possible to optimize the appropriate selection of the loss parameters (parameters of the cost function for training the model) for training the parking space recognition model in view of the quality of the recognized values. For example, a more accurate detection of the orientation of the entry line may therefore be more important than the detection of the length of the side boundary line. This aspect can be taken into account accordingly by reducing the share of the side boundary line length considered in the total loss for training the parking space recognition model.

[0020] Since the properties of the detected parking spaces can be better evaluated by the above-mentioned representation of the parking spaces, the loss function used to train the parking space recognition model can be more specifically adapted to the needs of parking space recognition. In particular, by using the angles α and β between the entry line and the two side boundary lines, parking spaces extending parallel to the driving direction, at right angles or obliquely can be automatically identified. By detecting the center of the entry line, parking can be performed in the parking space with the help of an automatic parking system even if the width value of the entry line and the length value of the side boundary line are not detected with high accuracy. For example, the free area can be measured by an object recognition system, for example, by an ultrasonic sensor, and the parking space can be used by using the located entry line, wherein the entry line is defined by the distance and angle to the center of the grid cell, for example, when using the YOLOv3 method.

[0021] Furthermore, the above-described representation of a parking space enables the elements of the parking space to be determined individually and largely independently of one another. The side boundary lines are therefore respectively described by an angle relative to the approach line and by the length of the side boundary line and are therefore independent of one another. It is therefore possible to calculate confidence measures independently of one another for approach line parameters, such as the distance vector between the center of the nearest grid cell and the approach line center, the orientation of the distance vector and the width of the approach line, and for side boundary line parameters, wherein the side boundary line parameters include the angle between the side boundary line and the approach line and the length of the side boundary line. With the confidence measures, the parking space selection can be improved by the automatic parking system in the case of a plurality of identified parking spaces.

[0022] According to one specific embodiment, the representation of the parking space can be used by an automatic parking algorithm in order to automatically control a parking process of the vehicle, in particular by determining a parking trajectory.

[0023] According to another aspect, a method for training a data-based parking space recognition model is provided, the method comprising the following steps:

[0024] - providing a training data set which in each case assigns a camera image of the vehicle's surroundings to a representation of a parking space recorded in the camera image, wherein the representation of the parking space comprises a specification of the position of an approach line and the position and orientation of lateral boundary lines adjoining the approach line;

[0025] - training a data-based parking space recognition model using a loss function, wherein the loss function weights the position of the entry line more than the position and orientation.

[0026] Additionally, the penalty function may evaluate the difference between the angles of the lateral boundary lines relative to the entry line. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The embodiments are explained in more detail below with reference to the accompanying drawings, wherein:

[0028] Figure 1 A schematic diagram of a motor vehicle with a camera system and an assistance system for detecting the surroundings is shown;

[0029] Figure 2 a schematic diagram showing a functional diagram of a parking space recognition system; and

[0030] Figure 3 A schematic diagram of a parking space including variables representing the parking space is shown. DETAILED DESCRIPTION

[0031] Figure 1 A schematic diagram of a vehicle 1 equipped with an assistance system 2 is shown. The assistance system 2 is connected to a camera system 3 so that the assistance system 2 can be provided with one or more camera images of the surroundings of the vehicle 1, in particular a part of the surroundings in the forward and reverse directions.

[0032] The camera image provided by the camera system 3 is evaluated in the assistance system 2 with the aid of a parking space recognition model and a representation of the position of the parking space in the surroundings is determined. The parking space representation can then be used, for example, with the aid of an automatic parking algorithm that can be implemented in the assistance system 2 in order to automatically park the vehicle 1 in the recognized parking space.

[0033] Figure 2 A schematic diagram of an automatic parking system 10 is shown, which is designed with the aid of a parking space recognition model 11 and a parking algorithm 12. The parking space recognition model 11 is designed in the form of a data-based model with an object recognition architecture. For example, the parking space recognition model 11 can be configured with the aid of a YOLOv3 network architecture. On the input side, the parking space recognition model receives a camera image B of the vehicle's surroundings. The camera image B can in particular be a composite image from a plurality of cameras 3 and is provided as an RGB image.

[0034] The YOLOv3 network architecture has a multi-layer convolutional neural network in a manner known per se, which locates possible parking spaces within a grid line network that maps the vehicle's surroundings in the horizontal direction. The grid line network is related to the vehicle coordinate system. Thus, objects, such as free parking spaces, are located in the grid line network that maps the surroundings by the parking space recognition model 11 based on the coordinates of the grid cells.

[0035] On the output side, entry line parameters EP and side boundary line parameters SP are output, which parameters make it possible to describe the orientation of the parking space relative to the vehicle based on geometric parameters related to the grid line network in such a way that the automatic parking algorithm 12 based thereon can perform the parking process in a simple manner. The parking algorithm 12 accordingly determines the parking trajectory S along which the vehicle 1 is to be moved into the parking space in a manner known per se.

[0036] As combined Figure 3 The parking space is represented as described. The positioning is not performed by determining the corner points of the parking space as in the prior art method, but by determining entry line parameters describing the orientation and width of the entry line of the free parking space. The entry line parameters EP include the distance of the center point Cxy of the entry line E from the center M of the grid cell of the grid line network G and the direction of the distance vector phi. In addition, the entry line parameters EP include the width W of the entry line E.

[0037] The side boundary line parameter SP determines the orientation of the side boundary lines S1, S2 as a function of the orientation of the entry line. Starting from the end point of the entry line E, the side boundary lines S1, S2 can extend, which are defined by the lengths L1, L2 and the side boundary line angles α, β, respectively. Thus, a parking space can be clearly defined even in the case of a non-rectangular shape.

[0038] The representation of the parking space by the approach line parameter EP and the side boundary line parameter SP contains in a direct and simple manner the usability properties of the parking space, which can be further processed by the automatic parking algorithm 12 .

[0039] Additionally, the representation of parking space locations enables the use of a single-stage parking space recognition model 11 and the definition of a loss function for training the data-based parking space recognition model 11 in terms of real parking space attributes. The loss function represents the cost function for a conventional training algorithm for a data-based model.

[0040] The training of parking space recognition model 11 can be performed in a conventional manner based on training methods known per se, for example based on back propagation of a predefined loss function.

[0041] The loss function generally takes into account the deviation of the target output of the parking space recognition model 11 from the actual output. In order to weigh the importance of the parking space properties in terms of the parking process, the loss function can give a higher evaluation to the consideration of the entry line parameter EP and a correspondingly lower evaluation to the side line parameters. In addition, the loss function can take into account the deviation of the side boundary line angles α and β in a simple manner, so that the greater the deviation between the side boundary line angles α and β, the greater the loss. Specific value ranges can also be defined. These value ranges are common for the side line angles α and β and the entry line width W, so that a high loss share can be set for values ​​that deviate from these value ranges.

Claims

1. A computer-implemented method, in particular for identifying a parking space for a vehicle (1), comprising the following steps: -Provide a camera image (B); - evaluating the camera image (B) with the aid of a data-based parking space recognition model (11), wherein the parking space recognition model (11) is designed to assign the camera image to a representation of a parking space in the surroundings of the vehicle (1) detected by the camera image (B), wherein the representation of the parking space specifies a description of the orientation of an entry line (E) into the recognized parking space in the vehicle coordinate system.

2. The method according to claim 1, wherein: The orientation of the entry line is specified as the length and orientation of the distance vector (Cxy) between the center of the entry line (E) and the angle of the entry line relative to the center of a grid cell in the grid line network (G).

3. The method according to claim 2, wherein: The orientation of the entry line (E) is further specified by the width (W) of the entry line (E).

4. The method according to claim 2 or 3, wherein: The parking space recognition model (11) is designed using a neural network and in particular has a YOLOv3 architecture, which locates the recognized parking spaces in the grid line network (G).

5. A method according to any one of claims 2 to 4, wherein the representation of the parking space is further specified by side boundary lines (S1, S2), the position and orientation of the side boundary lines being defined based on the position of the entry line (E), in particular by the length (l1, l2) of the corresponding side boundary lines (S1, S2) and the side boundary line angles (α, β) relative to the entry line (E).

6. The method according to any one of claims 1 to 5, wherein: The representation of the parking space is used by an automatic parking algorithm in order to automatically control a parking process of the vehicle (1), in particular by determining a parking trajectory.

7. A computer-implemented method for training a data-based parking space recognition model (11), the method comprising the following steps: - providing a training data set, which in each case assigns a camera image (B) of the vehicle's surroundings to a representation of a parking space recorded in the camera image, wherein the representation of the parking space comprises a specification of the position of an approach line (E) and the position and orientation of lateral boundary lines (S1, S2) adjoining the approach line (E), - training a data-based parking space recognition model (11) using a loss function, wherein the loss function weights the position of the entry line (E) more than the position and orientation.

8. The method according to claim 6, wherein: The loss function also evaluates the difference between the lateral boundary line angles (α, β) relative to the entry line (E).

9. An apparatus for performing the method according to any one of claims 1 to 8.

10. A computer program product comprising instructions which, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 8.

11. A machine-readable storage medium comprising instructions which, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 8.

Images