Detection of available parking spots

Abstract

To provide a computer-implemented method for detecting potentially available parking spots.SOLUTION: Receiving at least one image (I (E)) showing a vehicle driving environment from a camera system, determining at least one non-moving vehicle (Vnm) located in a parking spot in the vehicle driving environment, assigning a vehicle bounding box (BBV) to a non-moving vehicle (Vnm) in a parking spot, determining the presence of at least one characteristic feature (CF) within the vehicle bounding box (BBV) indicating a departure of the vehicle from the parking spot, assigning a departure probability (PL) to the identified non-moving vehicle (Vnm) based on the presence of the at least one determined characteristic feature (CF), and outputting information about the non-moving vehicle (Vnm) and the assigned departure probability (PL).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for detecting potentially available parking spots based on camera images acquired from a camera system comprising at least one camera, a computer program embodied to perform the method according to the invention, and a computer-readable storage medium, which can preferably be used, for example, in an advanced driver assistance system to perform parking functions. [Background technology]

[0002] Advanced driver assistance systems (ADAS) for vehicles are based on processing various data sensed by various ADAS sensors, such as radar sensors, LiDAR sensors, ultrasonic sensors, cameras, etc. Using ADAS sensors, information about the vehicle's surrounding environment can be obtained, which is then used to perform various ADAS functions. On the one hand, ADAS functions can involve assisting the driver while the control of the vehicle remains with the driver. On the other hand, depending on the automation level, fully autonomous vehicles can be realized. Known ADAS functions are, for example, various methods for detecting and / or classifying objects and / or obstacles around the vehicle, lane detection and / or lane departure detection methods, rain detection methods, as well as various parking assistance functions.

[0003] In the case of parking assistance functions, methods are known that assist in parking a vehicle, for example by assisting in the search for a parking slot, by assisting in the parking process, or even by autonomously carrying out the parking process once an available parking slot is found.

[0004] Neural network-based parking slot detection has been proposed, for example, by [1].

[0005] However, while current approaches primarily focus on detecting parking slots that are available and vacant, there is also a need to detect parking slots that may become vacant, i.e., be vacated. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Q. Huy et al., “CNN-based Two-Stage Parking Slot Detection Using Region-Specific Multi-Scale Feature Extraction,” 2021, available at arXiv:210806185 [Non-patent document 2] O. Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” available at arXiv:1505.04597 [Non-patent document 3] J. Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection,” available at arXiv:1506.02640 [Non-patent document 4] F. Li et al., “DINO: Towards a Unified Transformer-based Framework for Object Detection and Segmentation,” available at arXiv:26206.02777 [Non-Patent Document 5] A Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” available at arXiv arXiv:2010.11929 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, the objective technical problem underlying the present invention is to provide an option to determine parking spots that are about to become vacant or parking slots that are about to become available. [Means for solving the problem]

[0008] This object is achieved by a method according to claim 1, a computer program according to claim 15 and a computer-readable (storage) medium according to claim 16.

[0009] Regarding the method, the problem underlying the invention is to provide a computer-implemented method for detecting potentially available parking spots based on camera images acquired from a camera system comprising at least one camera, the method comprising: receiving at least one image from a camera system that is indicative of a vehicle's surroundings; - identifying at least one non-moving vehicle located in a parking spot in the vehicle's surroundings and assigning a vehicle bounding box to the non-moving vehicle in the parking spot; - determining the presence of at least one characteristic feature within the vehicle bounding box that is indicative of a vehicle exiting the parking spot; assigning a departure probability to the identified non-moving vehicle based on the presence of the at least one determined characteristic feature; and - outputting information about non-moving vehicles and assigned departure probabilities.

[0010] The camera system is preferably a vehicle camera system, the vehicle being any vehicle moving in road traffic, for example a car, a motorcycle, a bus, etc.

[0011] First, a non-moving vehicle located in a parking slot is detected. That is, a parking spot that may soon become available is detected not based on the boundary of the parking lot, but rather through the vehicle located there. Next, a characteristic feature of the vehicle when it leaves the parking spot is determined, and an exit probability is determined based on this. Information regarding the identified non-moving vehicle and the exit probability can be output, for example, to the vehicle driver or output as input to a subsequently executed parking function. For example, the output information can show an indication of a parking spot as a potentially available spot. In this case, an exit probability can be determined and assigned to the parking spot.

[0012] This procedure is particularly advantageous in urban environments and / or densely parked roads, public outdoor parking lots, and indoor parking lots, where finding an available parking spot is often difficult. The present invention provides, inter alia, a method for automatically finding soon-to-be-available parking spots, which can be used to assist the driver or can be input to an advanced driver assistance system that provides parking assistance.

[0013] In a preferred embodiment of the present invention, the at least one non-moving vehicle is identified by performing an object detection process, in particular using an object detector. Any object detection process can be used in this regard. If an object detector is applied, the object detector can be based on a machine learning model and can be any suitable known object detector.

[0014] Advantageously, identifying the at least one non-moving vehicle further comprises performing a depth estimation process based on the image sequence received from the camera system, this additional depth estimation making it possible to reliably distinguish between moving and non-moving vehicles.

[0015] The method is typically performed on streets or public parking areas where parking is dense. In these and similar cases, non-moving vehicles are typically parked vehicles, and other reasons for non-moving vehicles, such as being stopped at traffic lights or in a traffic jam, can be ignored.

[0016] In a preferred embodiment, the step of identifying at least one non-moving vehicle located in a parking spot in the vehicle's surroundings and / or the step of determining the presence of at least one characteristic feature within the vehicle's bounding box that is indicative of the vehicle's departure from the parking spot includes using at least one neural network. The neural network is preferably a convolutional neural network (CNN) or a transformer neural network. In some embodiments, a single multi-tasking neural network is deployed, while in other embodiments, multiple neural networks are used for different purposes.

[0017] Preferred examples of suitable networks include the U-net introduced in Non-Patent Document 2, and the YOLO network proposed in Non-Patent Document 3, particularly YOLOv8 or VOLOv9. On the other hand, examples of suitable transformer networks are the Mask DINO network introduced in Non-Patent Document 4, and the Vision Transformer (ViT) network described in Non-Patent Document 5.

[0018] Advantageously, the characteristic feature indicative of the departure of the vehicle from the parking spot is a vehicle light or a human being, in particular a human head. The vehicle light may be a front light, a back light or an indicator light. The detected human being is preferably a human being detected inside the vehicle.

[0019] With regard to the method according to the invention, it can be further verified that the detected features belong to the detected non-moving vehicle and can therefore be assigned as characteristic features.

[0020] In this regard, in a preferred embodiment, determining the presence of at least one characteristic feature includes detecting the feature, assigning a feature bounding box to the feature, and characterizing the feature as the characteristic feature if a ratio between the feature bounding box and the vehicle bounding box is within a predetermined ratio range.

[0021] More preferably, determining a characteristic feature having the form of a human head comprises detecting a human head, and the detected head is characterized as a characteristic feature if it is detected in a predetermined upper part of the vehicle bounding box, i.e., the detected human head is only considered if its head bounding box is located in an upper part of the vehicle bounding box, for example in the upper half or upper third thereof.

[0022] Also advantageously, determining the characteristic feature having the form of a vehicle light includes detecting a vehicle light, and the detected vehicle light is characterized as the characteristic feature if it is detected in a predetermined lower part of the vehicle bounding box, i.e., the detected vehicle light is only considered if the vehicle light bounding box is located at the lower part of the vehicle bounding box.

[0023] Finally, advantageously, determining the presence of at least one characteristic feature includes calculating a first distance from the camera system to the feature and a second distance from the camera system to the non-moving vehicle, and characterizing the feature as a characteristic feature if the difference between the first distance and the second distance is less than or equal to a predetermined threshold distance.

[0024] In another preferred embodiment, the step of assigning a departure probability to the identified non-moving vehicle includes predetermining at least one probability value indicative of a particular characteristic feature or a combination of two different characteristic features. For example, a list of potential characteristic features can be generated, and a probability value can be assigned to each characteristic feature of the simultaneously determined combination of at least two characteristic features.

[0025] Regarding the final method step regarding the output of relevant information regarding potentially available parking spots, there are several options available. For example, an indication can be provided to the vehicle driver in the form of an acoustic or visual signal about potentially available parking spots. Information can also be provided to an advanced driver assistance system embodied to perform parking functions.

[0026] Also, in this regard, it is advantageous that outputting the information regarding the non-moving vehicle and the departure probability includes, in particular, displaying to the vehicle driver, a camera image including the vehicle bounding box and the departure probability. Also, multiple vehicle bounding boxes can be simultaneously assigned an exit probability. In the latter case, the driver or the parking function can select one of the potentially available parking slots based on at least one criterion, such as the exit probability or the distance between the vehicle and a potentially available parking slot.

[0027] Preferably, the camera system further comprises a thermal camera for measuring the temperature of features in images received from the camera, such a thermal camera being able to provide additional information regarding the temperature of the non-moving vehicle.

[0028] In this regard, advantageously, the probability of departure is further determined based on the measured temperature of the feature, for example, a relatively high temperature of a non-moving vehicle may indicate recent use of that vehicle and may reduce the probability of departure for that particular vehicle.

[0029] The method according to the invention may be used in any of the described embodiments, particularly preferably in an advanced driver assistance system that performs parking functions.

[0030] The problem underlying the present invention is further solved by a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the present invention, and by a computer-readable (storage) medium comprising instructions executable by at least one processor for carrying out the method according to the present invention or on which the computer program according to the present invention is stored.

[0031] The invention and its preferred embodiments will be further described with reference to the following drawings. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 shows a flow diagram illustrating a method for detecting potentially available parking spaces. [Figure 2] FIG. 2 illustrates the determination of characteristic features. [Figure 3] Figure 3 shows an image of the vehicle's surroundings with indications of potentially available parking spots, where like elements are given like reference numerals. DETAILED DESCRIPTION OF THE INVENTION

[0033] In particular, when there are few or no parking spaces available, e.g., on streets with dense on-street parking, in dense outdoor or indoor parking lots, etc., vehicle drivers do not necessarily look for an empty slot, but for parking slots or spaces that may become available. The present invention provides an option for automatically detecting parking slots that will soon become available. The proposed approach is illustrated by the flow diagram shown in Figure 1.

[0034] In a first step, at least one image I(E) showing a vehicle environment (E) is received from a camera system (not shown). In the received image I(E), at least one non-moving vehicle V located in a parking spot of the vehicle environment E is detected. nm is identified.nm The identification of the non-moving vehicle V can be performed by an object detection process, for example, using an object detector OD, among others. Also, the vehicle bounding box BBV is used to identify the non-moving vehicle V. nm is assigned to.

[0035] Optionally, a depth estimation process d based on the image sequence I received from the camera system can be further performed. In this way, moving or dynamic vehicles can be compared with non-moving vehicles V nm By analyzing the depth based on the image sequence I received from the camera system, the relative position and velocity of the vehicle and other moving vehicles can be calculated. 1. For example, the vehicle bounding box BB determined as the (own) vehicle can be easily distinguished from the vehicle bounding box BB. V The distance between the vehicle and the non-moving vehicle V can be determined in successive images I. If the distance determined based on the depth information does not match the displacement indicating the vehicle is moving, the detected vehicle is classified as a non-moving vehicle V. nm and can be ignored in subsequent analyses.

[0036] In the third step, the vehicle bounding box BB V The presence of at least one characteristic feature CF indicative of the departure of the vehicle from the parking spot within the parking spot is determined. This step can also be performed by an object detection process, for example, using, among other things, an object detector OD. The characteristic feature CF can be an active light, for example, a headlight, a backlight, an indicator light, and, among other things, a light emitted by a non-moving vehicle V. nm It is the human head inside.

[0037] In a fourth step, the identified non-moving vehicle V is identified based on the presence of at least one determined characteristic feature. nm The dropout probability PL(CF,V nm ) is assigned. That is, the assigned defection probability PL depends on whether the characteristic feature CF is detected at all and the category of the characteristic feature, e.g., the type of feature, the image I, the vehicle bounding box BB. V It depends on the position inside.

[0038] Finally, the non-moving vehicle V nm and the departure probability PL may be output to the vehicle driver, for example, or provided as input for a subsequently executed parking function.

[0039] Vehicle bounding box BB V The determination of the characteristic feature CF indicative of the departure of a vehicle from a parking spot in the parking lot and the assignment of the departure probability PL based thereon are further illustrated in Fig. 2. For the proposed method, any one of the characteristic features CF can be checked. Fig. 2 exemplarily illustrates the case of four characteristic features CF1-CF4, where the first characteristic feature CF1-CF3 is indicative of a vehicle light, i.e., CF1 is indicative of a backlight, CF2 is indicative of an indicator light, CF3 is indicative of a headlight, while CF4 is indicative of a human head.

[0040] In the first step, the vehicle bounding box BB V An object detection process OD in is performed here for each characteristic feature CF1 to CF4, exemplarily using an object detector OD.

[0041] In a second step, a case distinction is performed based on different combinations of the characteristic features CF1 to CF4 that are considered. In the illustrated embodiment, the vehicle bounding box BB V Within, the first case indicates a situation where any active vehicle lights are detected (CF1 to CF3->t is true) but no human head is detected (CF4->f is false), the second case indicates a situation where any vehicle lights (CF1 to CF3) and a human head (CF4) are detected, the third case indicates a situation where no vehicle lights (CF1 to CF3) are detected but a human head (CF4) is detected, and finally, the fourth case indicates a situation where no vehicle lights (CF1 to CF4) and no head (CF4) are detected, i.e., no characteristic feature CF is detected at all.

[0042] In a subsequent third step, the different detected vehicle lights are distinguished: front lights (CF1), rear lights (CF2) and head lights (CF3). Note that in other embodiments, the second and third steps can also be combined into a single method step.

[0043] In the next optional fourth step, the determined characteristic features CF1 to CF4 are actually detected by the non-moving vehicle V. nm and the vehicle bounding box BB v Validation can be performed to verify whether a vehicle is associated with a non-moving vehicle V. nm and the determined characteristic feature CF. The validation may take into account position information, depth information, and / or size information for the characteristic features CF1-CF4, or a combination of these, or only selected characteristic features CF. The characteristic features CF1-CF4 and the vehicle bounding box BB. V If the association of vehicles can be confirmed, these are non-moving vehicles V nm This is taken into account when assigning the defection probability PL to each

[0044] For example, the validation may focus on the fact that for each considered characteristic feature CF, the ratio between the vehicle bounding box BBV and the feature bounding box BBF should be in a certain range. Otherwise, the determined characteristic feature FC may correspond to a different vehicle. Alternatively or additionally, the feature may be a feature that corresponds to the identified non-moving vehicle V. nm Positional requirements can also be defined to ensure that the determined vehicle lights are actually characteristic of the vehicle bounding box BB. V The detected human head should preferably be located at the bottom of the vehicle bounding box BB. V It is also possible to take into account estimated depth information, i.e., the determined characteristic feature CF should be located at the top of the identified non-moving vehicle V. nm It should be in close proximity to

[0045] 2, in the fifth step, different departure probabilities PL1 to PLN are determined for each case considered in the second step regarding the case classification of combinations of characteristic features CF1 to CF4. These departure probabilities PL1 to PLN are assigned to each combination of characteristic features CF, and finally, the departure probabilities PL1 to PLN are determined for ... nm and the assigned departure probability PL may be provided to the vehicle operator or provided as input to a parking function that is subsequently executed.

[0046] For this example, the departure probability PL is assigned based on the detected vehicle lights distinguishing between different vehicle lights and based on the detected human head. If no characteristic features are detected, this situation corresponds to an departure probability PL = 0. For example, the highest departure probability PL can be assigned to the situation where an illuminated backlight is detected, since it is likely that the backlight will be illuminated when leaving a parking spot. The second highest departure probability PL can be assigned to the situation where an indicator light is illuminated, followed by the situation where the headlights are illuminated and / or the situation where a human head is detected. It should be understood that additional configurations and characteristic features CF can be taken into consideration. The method can be applied to various characteristic features CF or feature combinations and their associated departure probabilities PL.

[0047] Optionally, information about non-moving vehicles obtained from a thermal camera (not shown) may also be taken into account to determine the non-moving vehicle V nm A dropout probability PL can be determined or assigned to

[0048] In Figure 3, an image I(E) showing the vehicle's surroundings (E) received from the vehicle's camera system is shown. Several vehicles are shown in the image, each marked with a black vehicle bounding box BB. V, which is shown for one of the vehicles as an example. According to the method of the present invention, the white vehicle bounding box BBV is assigned a departure probability PL=50%. In this way, the vehicle bounding box BB V Using the image I(E) with the assigned departure probability PL, relevant information can be output to the vehicle driver or to a processing unit for further processing of the obtained information.

Claims

1. 1. A computer-implemented method for detecting potentially available parking spots based on camera images (I) acquired from a camera system comprising at least one camera, the method comprising: - receiving from said camera system at least one image (I(E)) representative of the vehicle's surroundings (E); - at least one non-moving vehicle (V) located in a parking spot in the vehicle environment; ｎｍ ) and calculate the vehicle bounding box (BB Ｖ ) to the non-moving vehicle (V ｎｍ ), - the vehicle bounding box (BB Ｖ determining the presence of at least one characteristic feature (CF) indicative of the vehicle's departure from the parking spot within the parking spot; - determining the identified non-moving vehicle (V) based on the presence of the at least one determined characteristic feature (CF); ｎｍ ) with a dropout probability (PL); and - the non-moving vehicle (V ｎｍ ) and the assigned dropout probability (PL).

2. said at least one non-moving vehicle (V ｎｍ 2. The method according to claim 1, wherein the objects are identified by performing an object detection process (OD), in particular by means of an object detector.

3. a depth estimation process (d) based on the image sequence (I) received from the camera system to estimate the depth of the at least one non-moving vehicle (V ｎｍ 3. The method of claim 2, wherein the first and second vertices are identified as the first and second vertices.

4. said at least one non-moving vehicle (V ｎｍ 4. The method according to claim 1, wherein the step of identifying a characteristic feature (CF) and / or the step of determining the presence of the at least one characteristic feature (CF) comprises using at least one neural network.

5. The method according to any one of claims 1 to 4, wherein the characteristic feature (CF) indicative of the departure of the vehicle from the parking spot is a vehicle light or a human being, in particular a human head.

6. Determining the presence of the at least one characteristic feature (CF) includes detecting features, determining a feature bounding box (BB), Ｆ ) to the feature, and assigning the feature bounding box (BB Ｆ ) and the vehicle bounding box (BB Ｖ 6. The method of claim 1, further comprising characterizing the feature as a characteristic feature (CF) if its ratio to the characteristic feature (CF) is within a predetermined ratio range.

7. Determining a characteristic feature (CF) having a human head shape includes detecting a human head, and the detected head fits into the vehicle bounding box (BB). Ｖ 7. The method of claim 1, wherein the CF is characterized as a characteristic feature (CF) when detected at a predetermined upper portion of the CF.

8. Determining characteristic features (CF) having the form of vehicle lights includes detecting vehicle lights, and the detected vehicle lights are located within the vehicle bounding box (BB). Ｖ 8. The method of claim 1, wherein the CF is characterized as a characteristic feature (CF) if it is detected in a predetermined lower part of the CF.

9. Determining the presence of the at least one characteristic feature (CF) comprises determining a first distance from the camera system to the feature (CF) and a distance from the camera system to the non-moving vehicle (V). ｎｍ 9. The method of claim 1, further comprising: calculating a first distance to the feature; and a second distance to the feature; and characterizing the feature as a characteristic feature (CF) if the difference between the first distance and the second distance is less than or equal to a predetermined threshold distance.

10. Based on the presence of the at least one determined characteristic feature (CF), the identified non-moving vehicle (V ｎｍ 10. The method of claim 1, wherein assigning the churn probability (PL) of a characteristic feature (CF) comprises predetermining at least one probability value indicative of a particular characteristic feature (CF) or a combination of two different characteristic features (CF).

11. The non-moving vehicle (V ｎｍ ) and the departure probability (PL) is outputted. Ｖ 11. The method according to any one of claims 1 to 10, comprising displaying said camera image, in particular to a vehicle driver, containing said probability of departure (PL) and said departure probability (PL).

12. The method of any one of claims 1 to 11, wherein the camera system further comprises a thermal camera for measuring the temperature of features in the images received from the camera.

13. The method of any one of claims 1 to 12, wherein the probability of departure (PL) is further determined based on the measured temperature of the feature.

14. Use of the method according to any one of claims 1 to 13 in an advanced driver assistance system, in particular for performing parking functions.

15. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 13.

16. A computer-readable (storage) medium comprising instructions executable by at least one processor for carrying out the method of any one of claims 1 to 13 or having stored thereon the computer program of claim 15.

Images