Image Semantic Segmentation for Parking Space Detection
The automated parking system uses cameras and neural networks to detect and label parking spaces in images, projecting onto occupancy grids, addressing the challenge of space identification for autonomous vehicles.
Patent Information
- Application Number
- JP2023540010
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-12-30
- Filing Date
- 2021-12-23
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2041-12-23
AI Technical Summary
Automated driving systems face challenges in accurately distinguishing parking spaces from other vacant spaces using sensors and images, particularly for autonomous vehicles.
An automated parking system utilizing cameras, sensor arrays, and a neural network to detect and label features in images, generate masks, and project them onto an occupancy grid to identify parking spaces, considering horizontal edges and probabilities for vacant spaces.
Effectively locates and navigates vehicles to suitable parking spaces, even in the absence of visible markers, by accurately identifying and sizing parking spaces using image recognition and occupancy grids.
Smart Images

Figure 0007591660000001 
Figure 0007591660000002 
Figure 0007591660000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to automated driving systems, and more particularly to a system for detecting suitable parking spaces.
[0002] background Autonomous vehicles utilize information collected from a variety of sensors and communication devices. Such devices include GPS, radar, lidar, cameras, and other sensors that measure the vehicle's motion. To park a vehicle, a parking space needs to be distinguished from other vacant spaces. Sensing vacant spaces within an image of the environment near the vehicle is a challenging task for an autonomous vehicle.
[0003] The background discussion provided herein is intended to generally set forth the context of the present disclosure. To the extent described in this Background section, the work of the currently named inventors and aspects of the description that may not have qualified as prior art at the time of filing are not admitted, expressly or impliedly, as prior art to the present disclosure.
[0004] overview An automated parking system for an automated vehicle according to an exemplary embodiment of the present disclosure includes, among other things, at least one camera that captures images in the vicinity of the vehicle, at least one sensor array that detects objects in the vicinity of the vehicle, and a controller that is configured to generate a mask of an area in the vicinity of the vehicle from the images captured by the at least one camera, the mask including labeled features identified in the captured images, and project the generated mask onto an occupancy grid generated using information from the at least one sensor array, where the controller is configured to identify locations of parking spaces according to vacant spaces defined in the generated mask.
[0005] In another exemplary embodiment of the automated parking system described above, the controller includes a neural network configured to label features in the acquired images and is further configured to detect edges of objects in the generated mask to define a size of the parking space.
[0006] In another exemplary embodiment of any of the automated parking systems described above, detecting the edges includes detecting horizontal edges of a parked vehicle within the parking space and horizontal edges of a parking space marker.
[0007] In another exemplary embodiment of any of the automated parking systems described above, the controller is configured to define a best line across the defined parking space based on a detected horizontal edge of the parked vehicle and a horizontal edge of the parking space.
[0008] In another exemplary embodiment of any of the automated parking systems described above, the controller is further configured to determine that the parking space is free in response to detecting a free space on a side of the best line opposite the vehicle.
[0009] In another exemplary embodiment of any of the automated parking systems described above, the controller is further configured to associate a probability that the parking space is vacant and to select a parking space having a predetermined probability for parking the vehicle.
[0010] In another exemplary embodiment of any of the automated parking systems described above, the controller is further configured to determine a size of the parking space using dimensions determined from the occupancy grid after the mask is projected onto the occupancy grid.
[0011] In another exemplary embodiment of any of the automated parking systems described above, the controller is further configured to select a center of the parking spot to be utilized as a destination point of the determined path to guide the vehicle to the parking space.
[0012] In another exemplary embodiment of any of the automated parking systems described above, the controller is configured to define a parking space based on a selection by a driver of the vehicle in response to an absence of a visible parking marker in the generated mask.
[0013] A method for detecting parking spaces for an autonomous vehicle according to another example embodiment of the present disclosure includes, inter alia, acquiring an image of an area near the vehicle, generating a mask of the area near the vehicle that includes labeled features identified in the acquired image, defining free spaces in the generated mask as parking spaces for the vehicle, and projecting the generated mask onto an occupancy grid of the area near the vehicle to identify locations of the parking spaces relative to the vehicle.
[0014] In another embodiment of the above-described method for detecting parking spaces, edges of objects in the generated mask are detected to define the size of the parking spot.
[0015] In another embodiment of any of the above-described methods for detecting a parking space, detecting edges of the object comprises detecting horizontal edges of a parked vehicle and horizontal edges of a parking space marker within the parking spot.
[0016] Another embodiment of any of the above-described methods for detecting a parking space comprises defining a best line across the defined parking space based on the detected horizontal edges of the parked vehicle and the horizontal edges of the parking space.
[0017] Another embodiment of any of the above-described methods of detecting a parking space comprises determining that the parking space is free in response to detecting a free space on a side of the vest line opposite the vehicle.
[0018] Another embodiment of any of the above methods of detecting a parking space comprises associating a probability that the parking space is vacant and selecting a parking space having a predetermined probability for parking the vehicle.
[0019] Another embodiment of any of the above-described methods of detecting a parking space further comprises determining a size of the parking space using dimensions determined from the occupancy grid after the mask is projected onto the occupancy grid.
[0020] Another embodiment of any of the above methods of finding a parking space comprises selecting a center of the parking spot to define a path for moving the vehicle to the parking spot.
[0021] In another embodiment of any of the above-described methods of detecting a parking space, acquiring an image of the area near the vehicle further comprises acquiring an image from a camera disposed in at least one side mirror of the vehicle.
[0022] In another embodiment of any of the above-mentioned methods of detecting a parking space, the defining of a free space in the acquired image further comprises selecting an unoccupied area as a parking space in response to an absence of visible parking markers in the generated mask.
[0023] In another embodiment of any of the above-described methods of detecting a parking space, selecting the vacant area further comprises prompting an operator of the vehicle to assign a location within the vacant area as a parking space. Although the different examples have specific components illustrated, embodiments of the disclosure are not limited to those specific combinations. Some of the components or features of one example can be used in combination with the features or components of another example.
[0024] These and other features disclosed herein can be best understood from the following specification and drawings, the following of which is a brief description. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 illustrates a schematic diagram of a vehicle including an exemplary parking space system for an automated driving system. [Diagram 2] FIG. 1 illustrates a schematic diagram of an exemplary vehicle in a parking lot for capturing images and labeling objects. [Diagram 3] FIG. 2 shows a schematic diagram of an example image with labeled objects. [Figure 4] FIG. 2 shows a schematic diagram of an example image with identified best lines. [Diagram 5] FIG. 1 illustrates a schematic diagram of a mask applied to an occupancy grid and identified available parking spaces. [Figure 6] FIG. 1 shows a schematic diagram of a selected area within an open area with no markers.
[0026] Detailed Description Referring to FIG. 1, a vehicle 20 is shown generally including an exemplary parking space detection system 28 that provides information utilized by an automated driving system 26. The parking space detection system 28 captures images of the vicinity of the vehicle 20 using a side camera 24 and a front camera 40. An artificial intelligence algorithm, such as an exemplary neural network 34, analyzes the images and applies labels to objects detected in the images. The labeled images are then projected onto an occupancy grid of objects detected in the vicinity of the vehicle to identify vacant parking spaces. The occupancy grid may be generated using information from the cameras 24, 40 along with other sensor arrays on the vehicle, such as a radar device 22. As will be appreciated, other sensor devices may be utilized within the contemplation of this disclosure.
[0027] The parking space detection system 28 and the driving system 26 are part of a controller 30 that is configured to execute the disclosed methods and algorithms. The exemplary controller includes a processing device 32, a memory 36, and an artificial intelligence algorithm in the form of a neural network 34. Although the neural network 34 is disclosed as an example, other image recognition systems and methods that operate according to other artificial intelligence algorithms may be implemented within the contemplation of the present disclosure.
[0028] The controller 30, processing device 32 and memory device 36 are shown diagrammatically and may be part of an overall vehicle controller or a dedicated controller for the autonomous driving system 26. The controller 30 and processing device 32 may be hardware devices for executing software, particularly software stored in memory 36. The processing device 32 may be a custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with a computing device, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions.
[0029] Memory 36 may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements. Additionally, memory 36 may incorporate electronic, magnetic, optical, and / or other types of storage media. It should be noted that memory may have a distributed architecture, where various components are located remotely from each other, but are accessible by the processor.
[0030] The software in memory 36 may include one or more separate programs, each containing an ordered list of executable instructions for implementing the disclosed logical functions and operations. A system component realized as software may be interpreted as a source program, an executable program (object code), a script, or other entity comprising a sequence of instructions to be executed. If constructed as a source program, the program is translated via a compiler, assembler, interpreter, etc., which may or may not be contained within memory.
[0031] Input / output devices (not shown) that may be coupled to the system I / O interface may include, but are not limited to, input devices such as, for example, a keyboard, a mouse, a scanner, a microphone, a camera, a proximity device, etc. Additionally, the input / output devices may include, but are not limited to, output devices such as, for example, a printer, a display, etc. Finally, the input / output devices may further include devices that communicate as both inputs and outputs such as, but are not limited to, modulators / demodulators (modems; for accessing another device, system, or network), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc.
[0032] When the parking space detection system 28 is in operation, the processor 32 may be configured to execute software stored in the memory 36, to communicate data to and from the memory 36, and to generally control the operation of the system 28 in accordance with the software. The software in the memory may be read, in whole or in part, by the processor, perhaps buffered within the processor, and then executed.
[0033] With continued reference to Figure 1 and with reference to Figure 2, the controller 30 is configured to generate a mask of the area near the vehicle 20 from images captured by the cameras 24, 40. The controller 30 is further configured to assign labels to features in the captured images. In this example, objects in the images are identified utilizing a neural network 34. As will be appreciated, other image recognition algorithms and systems may be utilized and are within the contemplation of the present disclosure. The captured images are used to generate a mask that is subsequently projected onto an occupancy grid.
[0034] Figure 2 shows a schematic of a vehicle 20 in a parking lot 42 including other vehicles 44 parked within the parking spaces and an empty space 48. In this example, the empty space 48 is located between parking markers 46. The diagram shown in Figure 2 is a visual projection of the information obtained from the cameras 24, 40 onto a bird's eye view. In this example, one of the side mirrors 24 is utilized to capture the image along with the front view camera 40.
[0035] The captured images are used to create a mask of the scene, in this case the parking lot, comprising the vehicles 44, spaces 48 and markers 46. The images are labeled using the neural network 34.
[0036] In this disclosed example, a trailer 38 is coupled to the vehicle 20. The presence of the trailer 38 is recognized by the controller 30, and suitable parking areas are evaluated based on the vehicle configuration including the trailer 38. The exemplary system and method operates to detect an available space suitable for parking the vehicle based on the vehicle configuration including the presence of a trailer, as shown in this example.
[0037] 3, the image is utilized to generate a mask, shown generally at 50. Mask 50 includes labels for vehicles 44, markers 46 and spaces 48. Labeling of objects within mask 50 is the first step in defining mask 50, which is then projected onto an occupancy grid.
[0038] Referring to FIG. 4, the mask 50 is further labeled by identifying features in the image to distinguish between vacant spaces that are actually parking spaces and vacant spaces that are not parking spaces. In the example disclosed herein, the edges of the vehicle 44, indicated at 58, are identified. The forward edge 56 of each marker 46 is also identified. Edges 58 and 56 are forward and closest to the vehicle 20, and they indicate which portions of the vacant space are parking spaces and which portions are part of a path through the parking lot 42. In this example, edges 58 and 56 provide points for generating a best line 52. An exemplary best line 52 is a horizontal line created through the identified edges. As will be appreciated, other edge and line directions can be utilized to address other parking space orientations and configurations. In this example, the best line 52 is utilized to distinguish between vacant spaces that are parking spaces 48 and vacant spaces that are not parking spaces, indicated at 54. In this example, the controller 30 is configured to recognize the vacant space 48 on the side of the best line 52 opposite the vehicle 20 as a parking space. Any vacant spaces on the same side of the vest line 52 are identified as not being parking spaces.
[0039] The vest line 52 is also used to determine the size of the free space 48 so that the system can verify whether the vehicle 20 will fit within the spot 48.
[0040] Referring to FIG. 5, the mask 50 is then projected onto an occupancy grid, shown generally at 60. Projecting the mask 50 onto the occupancy grid 60 places vacant parking spaces, as shown at 62, in positions in the real world near the vehicle. As will be appreciated, the mask 50 includes labels for the captured image, and the occupancy grid 60 includes real world correlations for objects and features identified in the image. The mask 50 projected onto the occupancy grid 60 will include at least the vest line 52 and the vacant space identification. Other labels, such as vehicles 44, may also be projected onto the occupancy grid 60. The occupancy grid 60 may be implemented as a two-dimensional grid or a three-dimensional grid. The mask 50 may be modified to fit either a two-dimensional grid or a three-dimensional grid.
[0041] Once the mask 50 is projected onto the occupancy grid 60, candidate parking spaces 62 are identified and centerlines 64 are defined in the spots to be used by the automated driving system 26. The automated driving system 26 uses this information to define a path for entering the space 62.
[0042] With reference to FIG. 6, the disclosed method and system can also operate in areas where there are no parking markers or other vehicles. FIG. 6 is a schematic diagram of an empty area that may be adjacent to a campground or other area where parking markers are not utilized. In this case, the user would be prompted to place a box 68 on a captured image of the empty area 66 without objects defining the parking space. The box 68 provides a definition of the parking space with a mask. As will be appreciated, the box 68 is shown as an example, but any shape or area selection feature can be utilized within the contemplation of the present disclosure. Thus, rather than having the neural network 34 use objects in the image to determine an appropriate parking space, the driver defines the space for parking. The defined space indicated by the box 68 is then projected onto an occupancy grid and utilized to generate a path for moving the vehicle 20 into the space.
[0043] Thus, the disclosed example parking space detection system 28 utilizes semantic segmentation to detect free spaces and vehicles from images captured by the cameras 24, 40 on the vehicle 20. Features and objects in the images are labeled at the edges of the vehicle and parking markers are identified. The edges of the vehicle and the parking markers are used to define a horizontal best line 52. The image containing the free space labels and best line 52 is projected onto an occupancy grid, which orients the free spaces relative to the vehicle 20 and identifies available, vacant parking spaces.
[0044] Although the different non-limiting embodiments are illustrated as having particular components or steps, the embodiments of the disclosure are not limited to those particular combinations. Some of the components or features from any of the non-limiting embodiments can be used in combination with features or components from any of the other non-limiting embodiments.
[0045] It should be understood that like reference numerals identify corresponding or similar elements throughout the several views, and although particular component arrangements are disclosed and illustrated in these exemplary embodiments, it should be understood that other arrangements can benefit from the teachings of the present disclosure.
[0046] The above description should be interpreted as illustrative and not limiting in any way. Those skilled in the art will appreciate that certain modifications may fall within the scope of the present disclosure. For these reasons, the following claims should be studied to determine the true scope and content of the present disclosure.
Claims
1. 1. An automated parking system for a motor vehicle, comprising: at least one camera for acquiring images of the vicinity of the vehicle; at least one sensor array for detecting objects in the vicinity of the vehicle; A controller; It is equipped with the controller is configured to generate a mask of an area near the vehicle from the images captured by the at least one camera, the mask including labeled features identified in the captured images, and project the generated mask onto an occupancy grid generated using information from the at least one sensor array; the controller is configured to identify locations of parking spaces according to vacant spaces defined in the generated mask; The controller is further configured to detect edges of objects in the generated mask to define a size of a parking space; said detecting edges includes detecting horizontal edges of a parked vehicle in a parking spot and horizontal edges of a parking space marker; the controller is configured to define a best line across a defined parking space based on the detected horizontal edges of the parked vehicle and the horizontal edges of the parking space markers. Automatic parking system.
2. The automated parking system of claim 1 , wherein the controller includes a neural network configured to label features in the captured images.
3. 3. The automated parking system of claim 1, wherein the controller is further configured to determine that a parking space is free in response to detecting a free space on a side of the best line opposite the vehicle.
4. 4. The automated parking system of claim 3, wherein the controller is further configured to associate a probability that the parking space is vacant and to select the parking space having a predetermined probability for parking the vehicle.
5. 5. The automated parking system of claim 4, wherein the controller is further configured to determine a size of the parking space using dimensions determined from the occupancy grid after the generated mask is projected onto the occupancy grid.
6. 6. The automated parking system of claim 5, wherein the controller is further configured to select a center of the parking space to be utilized as a destination point of a path determined to guide the vehicle to the parking space.
7. 2. The automated parking system of claim 1, wherein the controller is configured to define the parking space based on a selection by a vehicle operator in response to an absence of a visible parking marker in the generated mask.
8. 1. A method for detecting a parking space for an autonomous vehicle, comprising: acquiring an image of an area near the vehicle; generating a mask of the area near the vehicle that includes labeled features identified in the captured image; defining free spaces within the generated mask as parking spaces for the vehicle; - projecting the generated mask onto an occupancy grid of the area near the vehicle to identify the location of the parking space relative to the vehicle; Including, the step of defining a free space within the generated mask further comprises the step of selecting an unoccupied area as a parking space in response to an absence of visible parking markers within the generated mask. How to detect parking spaces.
9. The method of claim 8 further comprising detecting edges of objects in the generated mask to define a size of the parking space.
10. 10. The method of claim 9, wherein detecting edges of objects includes detecting horizontal edges of a parked vehicle within a parking spot and horizontal edges of a parking space marker.
11. 11. The method of claim 10, further comprising the step of defining a best line across a defined parking space based on the detected horizontal edges of the parked vehicle and the horizontal edges of the parking space markers.
12. The method of claim 11 , further comprising the step of determining that a parking space is free in response to detecting a free space on an opposite side of the vehicle from the best line.
13. 13. The method of finding a parking space of claim 12, further comprising the steps of associating a probability that a parking space is vacant and selecting the parking space having a predetermined probability for parking the vehicle.
14. A method for detecting a parking space as described in claim 13, further comprising a step of determining a size of the parking space using dimensions determined from the occupancy grid after the generated mask is projected onto the occupancy grid.
15. The method of claim 14 further comprising the step of selecting a center of the parking space to define a path for moving the vehicle into the parking space.
16. 9. The method of detecting a parking space of claim 8, wherein the step of capturing an image of the area near the vehicle further comprises capturing an image from a camera disposed in at least one side mirror of the vehicle.
17. 9. The method of finding a parking space of claim 8, wherein the step of selecting a vacant area further comprises the step of prompting an operator of a vehicle to assign a location within the vacant area as the parking space.
Citation Information
Patent Citations
Detection method and device of parking space and direction and angle thereof, equipment and medium
CN110706509A
Track calculation device
JP2012153324A
Parking position detection system, and automatic parking system using the same
JP2017147629A
Outside recognition device for vehicle
JP2019066333A
Real-time robust surround view parking space detection and tracking
US10867192B1
