Parking space detection method and system

Abstract

A method includes obtaining a first image from a first camera disposed on a vehicle and generating a bird's-eye view (BEV) image using the first image. The method further includes processing the BEV image using a machine learning model to generate parking space prediction data. The parking space prediction data includes a first center coordinate of a first available parking space, a first parking space confidence, and first corner displacement data. The first corner displacement data includes a first pair of relative coordinates positioning a first corner relative to a first center coordinate and a second pair of relative coordinates positioning a second corner relative to the first center coordinate. The method further includes determining a first location of the first available parking space using the parking space prediction data, and parking the vehicle in the first available parking space when the first parking space confidence satisfies a threshold.

Description

Background Art

[0001] Parking a vehicle is known to be a stressful task for drivers. Consequently, various automotive companies have been working to automate the task of parking through in-vehicle driver assistance systems. In order to accurately park a vehicle, it is necessary to identify available parking spaces.

[0002] Typically, parking space detection is performed using one or more sensor systems on the vehicle, combined with computational methods for processing the sensory data received from the sensor systems. The sensor systems may include emitters (e.g., lasers) and one or more sensors. A non-exhaustive list of example sensor systems includes: cameras; LiDAR; radar; and ultrasonic sensors. However, despite the availability of sensory data, many challenges exist in accurately and efficiently detecting available parking spaces.

[0003] The computational methods used to process vehicle sensor data and identify available parking spaces are often slow and cannot be applied in real time. Furthermore, many parking space detection systems can only detect parking spaces with limited conditions, such as spaces with certain orientations, spaces with clear boundaries, or spaces bounded by other occupied parking spaces. The inability of parking space detection systems to quickly and consistently identify available parking spaces in different situations has led to widespread distrust in driver assistance systems. Summary of the Invention

[0004] This summary is provided to introduce some concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Embodiments disclosed herein generally relate to a method that includes obtaining a first image from a first camera mounted on a vehicle and generating a bird's-eye view (BEV) image using the first image. The method also includes processing the BEV image using a machine learning model to generate parking space prediction data. The parking space prediction data includes first center coordinates of a first available parking space, a first parking position confidence, and first corner displacement data. The first corner displacement data includes a first relative coordinate pair locating a first corner relative to the first center coordinate and a second relative coordinate pair locating a second corner relative to the first center coordinate. The method also includes determining a first position of a first available parking space using the parking space prediction data and, when the first parking position confidence satisfies a threshold, parking the vehicle in the first available parking space.

[0006] Embodiments disclosed herein generally relate to a computer-implemented method for training a machine learning model. The method includes obtaining a plurality of bird's-eye view (BEV) images and identifying available parking spaces in the plurality of BEV images. Furthermore, for each identified available parking space in the BEV image, the method also includes: determining a ground-truth parking representation for the identified available parking space, wherein the ground-truth parking representation includes center coordinates and corner displacements; calculating an envelope width and an envelope height using the center coordinates and the corner displacements; and matching the identified available parking space to one or more anchor boxes using the envelope height and the envelope width. Once the ground-truth parking representation has been determined for each identified available parking space in a given BEV image, the method also includes generating a target data structure for the given BEV image, the target data structure including the center coordinates and the corner displacements for each identified available parking space in the given BEV image. The method also includes generating a training dataset comprising the plurality of BEV images and their associated target data structures, and training a machine learning model using the training dataset, wherein the machine learning model is configured to receive one or more BEV images directly from the plurality of BEV images.

[0007] Embodiments disclosed herein generally relate to a system comprising a vehicle, a first camera mounted on the vehicle, a bird's-eye view (BEV) image, a machine learning model, and a computer. The computer comprises one or more computer processors and is configured to obtain a first image from the first camera, construct a BEV image from the first image, and process the BEV image using the machine learning model to generate parking space prediction data. The parking space prediction data comprises first center coordinates of a first available parking space, a first parking position confidence, and first corner displacement data. The first corner displacement data comprises a first relative coordinate pair locating a first corner relative to the first center coordinate and a second relative coordinate pair locating a second corner relative to the first center coordinate. The computer is further configured to determine a first position of a first available parking space using the parking space prediction data and, when the first parking position confidence satisfies a threshold, park the vehicle in the first available parking space without driver assistance.

[0008] Other aspects and advantages of the claimed subject matter will be apparent from the following description and appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Now, specific embodiments of the disclosed technology will be described in detail with reference to the accompanying drawings. For consistency, the same elements in various drawings are represented by the same reference numerals.

[0010] Figure 1 A parking lot is depicted according to one or more embodiments.

[0011] Figure 2AA perpendicular parking space orientation is depicted according to one or more embodiments.

[0012] Figure 2B Parallel parking space orientation is depicted according to one or more embodiments.

[0013] Figure 2C Fishbone parking space orientation is depicted in accordance with one or more embodiments.

[0014] Figure 3 Depicted are fields of view of multiple cameras disposed on a vehicle according to one or more embodiments.

[0015] Figure 4 Depicted are images acquired from a plurality of cameras disposed on a vehicle, according to one or more embodiments.

[0016] Figure 5 Depicted are example bird's eye view (BEV) images in accordance with one or more embodiments.

[0017] Figure 6 Systems are described in accordance with one or more embodiments.

[0018] Figure 7A Depicted are parking spaces defined using absolute coordinates according to one or more embodiments.

[0019] Figure 7B Depicted are parking spaces defined with center-relative coordinates according to one or more embodiments.

[0020] Figure 7C Depicted is a fishbone-shaped parking space defined by center relative coordinates according to one or more embodiments.

[0021] Figure 8 Depicted are output data structures of a machine learning model according to one or more embodiments.

[0022] Figure 9 Depicted are example BEV images labeled with predicted parking space data in accordance with one or more embodiments.

[0023] Figure 10 A neural network is depicted in accordance with one or more embodiments.

[0024] Figure 11 Depicted are a ground truth parking representation and two anchor boxes in accordance with one or more embodiments.

[0025] Figure 12 Depicted are a ground truth parking representation and a predicted parking representation in accordance with one or more embodiments.

[0026] Figure 13 Flowcharts are depicted according to one or more embodiments.

[0027] Figure 14 Flowcharts are depicted according to one or more embodiments.

[0028] Figure 15 Flowcharts are depicted according to one or more embodiments.

[0029] Figure 16 Systems are described in accordance with one or more embodiments. DETAILED DESCRIPTION

[0030] In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth to provide a more thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In other cases, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0031] Throughout this application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any nouns in this application). The use of ordinal numbers does not imply or create any specific ordering of the elements, nor does it limit any element to only a single element, unless expressly disclosed, such as with the use of the terms "before," "after," "single," and other such terms. Rather, ordinal numbers are used to distinguish elements. For example, a first element is different from a second element, and a first element may contain more than one element and come after (or before) a second element in the ordering of the elements.

[0032] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "an acoustic signal" includes reference to one or more of such acoustic signals.

[0033] Terms such as "about," "substantially," and the like mean that the recited features, parameters, or values ​​need not be achieved precisely, but rather that deviations or variations (including, for example, tolerances, measurement errors, measurement precision limitations, and other factors known to those skilled in the art) may occur in amounts that do not negate the effect that the features are intended to provide.

[0034] It should be understood that one or more steps shown in the flowcharts may be omitted, repeated, and / or performed in a different order than shown. Therefore, the scope of the disclosure herein should not be considered limited to the specific arrangement of steps shown in the flowcharts.

[0035] Although multiple dependent claims are not introduced, it will be apparent to one of ordinary skill that the subject matter of the dependent claims of one or more embodiments may be combined with other dependent claims.

[0036] exist Figure 1-16 In the following description, in the various embodiments disclosed herein, any component described with respect to a figure may be equivalent to one or more components of the same name described with respect to any other figure. For the sake of brevity, the description of these components will not be repeated with respect to each figure. Therefore, each embodiment of the components of each figure is incorporated by reference and is assumed to be optionally present in each other figure having one or more components of the same name. In addition, according to the various embodiments disclosed herein, any description of the components of the figures will be interpreted as an optional embodiment, which can be implemented in addition to, in combination with, or in place of the embodiment described with respect to the corresponding components of the same name in any other figure.

[0037] Embodiments disclosed herein describe methods and systems for detecting parking spaces around a vehicle using one or more images obtained from one or more cameras mounted on the vehicle. The parking space detection methods and systems disclosed herein identify available or partially available parking spaces using a parking representation consisting of at least two corners, where each corner of the parking representation represents a specific corner of the parking space. For example, where the parking representation is a polygon with four corners, each of the four corners is explicitly designated as one of the following: entry-left, entry-right, end-left, and end-right. Furthermore, the corner designations of the parking space are mutually exclusive. Thus, the parking representation can directly conform to any orientation of the parking space, regardless of the relative viewpoint of the vehicle, without requiring rotation or affine operations, while encoding the corner relationships.

[0038] Figure 1 An example parking lot (100) is depicted that includes parking spaces (102). Figure 1 The marking lines do not extend to every parking space (102). A parking space (102) may be occupied by a parked vehicle (204) or other object (e.g., a trash container (104)), or may be unoccupied and available. In some instances, the parking lot (100) may be outdoors (as shown) or indoors, such as in a parking garage (not shown). In other instances, the parking space (102) may be located near a road. The surface material of the parking space (102) depends on its location and construction. A non-exhaustive list of parking space (102) surface materials may include asphalt, concrete, brick, and paving materials.

[0039] Typically, a parking space (102) has an associated parking space orientation. Figures 2A-2C Parking spaces (102) are depicted with various parking space orientations. Also, for clarity, not all parking spaces (102) are labeled. Figures 2A-2CEach depicts a searching vehicle (202) or a vehicle intended to be parked in an available parking space, and one or more parked vehicles (204) already parked in a parking space (102). Figures 2A-2C Each depicts at least one empty and available parking space ( 205 ). Figure 2B Another vehicle (203) is also depicted that is neither parked nor searching for an available parking space (102). In particular, Figure 2A A parking space (102) is depicted, which is available and occupied by a parked vehicle (204), having a vertical parking space orientation (206). Figure 2B A parking space (102) is depicted adjacent to a road (209), wherein the parking space (102) has a parallel parking space orientation (208). Figure 2C A parking space (210) is shown having a fishbone or diagonal space orientation.

[0040] Figures 2A-2C It is further shown that the parking space (102) can be marked in a variety of ways. For example, Figure 2A As shown, a solid line may be used to mark or define a parking space (102). Alternatively, as shown Figure 2B and 2C As shown, the parking space (102) may be marked with a dashed line. In other cases, some or all of the markings defining the parking space (102) may be implicit (e.g., curb, parking space entrance, etc.), clearly shown, or ambiguous (e.g., faded). Additionally, the markings defining the parking space (102) may partially surround the parking space (102), such as Figure 2A and 2C As shown, or the parking space (102) can be completely surrounded, as Figure 2B In addition, Figure 2A As shown, an "L corner" marking (212) or a "T corner" marking (214) may be used to indicate one or more corners of a parking space (102). In other examples, no corner markings are used (e.g., Figure 2C ).

[0041] In general, a parking space (102) can be configured with any combination of the features described above. That is, a parking space (102) can have any combination of surface material (asphalt, concrete, etc.), setting (e.g., indoor or outdoor, parking lot or street adjacent), space orientation (perpendicular (204), parallel (208), herringbone (210)), marking line style (dashed, solid, mixed), corner designation (T-corner, L-corner, etc.), and marking closure (partially enclosed, fully enclosed). One of ordinary skill in the art will recognize that alternative and / or additional features, such as vehicle entry angle and parking space width, can be used to define a parking space. Therefore, due to the breadth and descriptive nature of parking space (102) configurations, one of ordinary skill in the art will understand that not all parking space (102) configurations need be enumerated herein without imposing limitations on the present disclosure.

[0042] As will be discussed later in this disclosure, the parking space detection method and system disclosed herein are applicable to any configuration of parking spaces (102). Furthermore, the parking space detection method and system disclosed herein are robust to additional factors such as environmental conditions (e.g., nighttime, rain) and partial obstruction of the parking space (102).

[0043] Figure 3 A camera view (300) of the surroundings of a vehicle (301) is depicted. According to one or more embodiments, one or more cameras may be provided on the vehicle (301) in order to detect parking spaces (102) near the vehicle (301) (e.g., to search for a parking space (102) near the vehicle (202)). In one or more embodiments, one or more of the cameras may be equipped with a fisheye lens. In other embodiments, one or more of the cameras provided on the vehicle may be a pinhole (i.e., conventional) camera. Figure 3 In the example, the camera is not explicitly shown. However, in Figure 3 , the locations of four cameras disposed on the vehicle (301) are indicated by circles. Typically, each of the one or more cameras disposed on the vehicle (301) can be relatively small compared to the vehicle (301). Therefore, due to the small size of the cameras, the cameras can be disposed on the vehicle 301 such that the cameras are generally inconspicuous and do not obstruct any function of the vehicle 301. Furthermore, each of the one or more cameras can be concealed (e.g., tucked under a trim piece) and reinforced (i.e., able to withstand vibration, operate in a wide range of temperatures, be waterproof, etc.).

[0044] Figure 3 A front camera (304), two side cameras (306) and a rear camera (308) are depicted. The number of cameras provided on the vehicle (301) and the positions of the cameras are not limited to Figure 3. In other embodiments, fewer or more cameras may be used. For example, in one or more embodiments, a camera (or camera system) with surround view capability may be located on top of vehicle 301.

[0045] A camera has a field of view (FOV). FOV is used herein as a general term intended to indicate the extent of the observable world seen by a camera. Stated more technically, FOV is the solid angle over which a camera's detector is sensitive to electromagnetic radiation. Figure 3 The field of view of each of the four cameras on the vehicle (301) depicted in FIG is shown by a dashed closed curve. The front camera (304) has a front FOV (310) such that the front camera (304) can acquire images of the environment in front of the vehicle (301). Similarly, the side cameras (306) are associated with a left-facing FOV (312) or a right-facing FOV (314), where the terms "left" and "right" are relative to each other. Figure 3 Finally, the rear-facing camera (308) has a rear-facing FOV (316) such that the rear-facing camera (308) can capture images of the environment behind the vehicle (301). Figure 3 As shown, the fields of view of the cameras partially overlap, so that the entire surroundings of the vehicle (301) are captured by the cameras. In other words, a set of images acquired by the cameras mounted on the vehicle (301) at a given moment can be said to capture the surroundings of the vehicle (301) at that moment. As previously mentioned, in some cases, the entire surroundings of the vehicle (301) can be captured using a camera system mounted on top of the vehicle (301), the system consisting of one or more cameras.

[0046] continue Figure 3 For example, it should be noted that the placement of the one or more cameras disposed on the vehicle (301) (e.g., the height of the one or more cameras relative to the ground) can directly affect the field of view of the one or more cameras and the visibility of the parking spaces (102) near the vehicle (301). One of ordinary skill in the art will understand that the height of the one or more cameras disposed on the vehicle 301 can be adjusted and / or selected to optimize the field of view of the camera without departing from the scope of the present disclosure. In addition, the one or more cameras can be placed so that any viewpoint of the one or more cameras is not obstructed by the vehicle (301).

[0047] Figure 4 Four examples of images are depicted, using settings such as Figure 3Each image is acquired by one of four cameras on the vehicle (301) shown. Because each image is associated with a given camera located on the vehicle (301), the images can be described as a front image (402) (from the front camera (304)), a left image (404) (from the side camera (306) with a left FOV (312)), a right image (406) (from the side camera (306) with a right FOV (314)), and a rear image (408) (from the rear camera (308)). The example images have overlapping fields of view and form the surrounding camera view (300). The overlap of the fields of view is represented because the parked vehicle (204) appears in both the left image (404) and the rear image (408).

[0048] Figure 4 The example image is acquired at the moment when a vehicle (301) on which a camera is mounted passes through a parking lot (100). Figure 4 As shown, the parking lot (100) is outdoors and includes parking spaces (102), not all of which are located in the Figure 4 is marked. Figure 4 The parking spaces (102) have a vertical parking orientation (206) and are each marked with a partially enclosed solid line. Specifically, both side boundaries of each parking space (102) are marked with a solid line, the far end of the parking space (102) is implicitly marked by the curb, and the entrance side of the parking space (102) is unmarked. Additionally, in Figure 4 In the example image of FIG, an unusable area ( 420 ) of the parking lot ( 100 ) can be seen.

[0049] According to one or more embodiments, images acquired from one or more cameras disposed on a vehicle (301) are stitched together to create a bird's-eye view (BEV) image or representation of the vehicle's (301) surroundings. In one or more embodiments, the BEV image is created from the images acquired from the one or more cameras disposed on the vehicle (301) using a technique commonly referred to as inverse perspective mapping (IPM). Typically, IPM assumes that the vehicle's (301) surroundings are flat and, once calibrated, maps the pixels of the image onto a flat plane via a homographic projection. Figure 5 Depicted by Figure 4 The example BEV image (500) is constructed from an example image of the vehicle (301). The example BEV image (500) shows the surroundings of the vehicle (301) (parking lot (100)), including parking spaces (102), unavailable areas (420), and parked vehicles (204). Again, to avoid cluttering the figure, it should be noted that not every parking space (102) is labeled. As shown in the figure, IPM often results in deformation of non-planar objects. For example, the parked vehicle (204) appears stretched, as if extending to Figure 5 However, the BEV image generally does not strongly affect the representation of the parking space (102) and other planar objects, so that the BEV image can be used for parking space (102) identification and positioning.

[0050] In one or more embodiments, the vehicle (301) may be equipped with additional sensing systems. These sensing systems may include ultrasound systems and light detection and ranging (LiDAR) systems, each consisting of one or more sources (e.g., ultrasound transmitters, lasers) and receivers or sensors. In one or more embodiments, the sensing data received from the additional sensing systems supplements the BEV image and is used to assist in localizing the vehicle (301) and parking space (102) within the BEV image.

[0051] Figure 6A high-level overview of a parking space detection method according to one or more embodiments is shown in FIG. As shown, one or more surrounding images 602 are acquired using one or more cameras provided on a vehicle 301. Herein, the term surrounding image simply refers to an image that captures at least a portion of the surrounding environment of the vehicle (301). In one or more embodiments, the camera provided on the vehicle (301) is equipped with a fisheye lens and has an image resolution of 1280×800 pixels. Hereafter, for simplicity, it will be assumed that there are at least two surrounding images so that the surrounding images (602) can be referred to as plural without ambiguity. However, it is emphasized that the embodiments disclosed herein can operate with a single surrounding image (602) without limitation. Therefore, there is no requirement that the surrounding image (602) consist of more than one image. As the name implies, the surrounding image 602 is an image of the local environment of the vehicle 301 at a certain moment. The surrounding image (602) is processed using an inverse perspective mapping (IPM) module (604), which applies the IPM technique to the surrounding image (602) to form a BEV image (606). It should be noted that in one or more embodiments, the BEV image (606) can be formed without the IPM module (604), for example when using a surround view camera system mounted on top of the vehicle (301). In one or more embodiments, the resulting BEV image (606) has an image resolution of 640×640 pixels and spans a physical area of ​​25 meters by 25 meters. In this embodiment, a single size of a pixel corresponds to 3.9 centimeters. Furthermore, according to one or more embodiments, the vehicle (301) is centered within the BEV image (606). Thus, the BEV image (606) can depict a parking space (102) up to 12.5 meters from the vehicle (301). That is, in one or more embodiments, the detection area of ​​the vehicle (301) is 25 meters by 25 meters, with the detection area centered on the vehicle (301). Note that the detection area can be modified by adjustments to one or more cameras (e.g., camera type, camera placement, etc.) such that the detection area is not limited to 25 meters by 25 meters, but can be smaller or larger than this value in other embodiments. Figure 6Remaining consistent, the BEV image (606) is processed using a machine learning model (608) to produce parking space prediction data (610). The machine learning model (608) will be described in more detail below. However, it will now be described that the machine learning model (608) is trained and configured to accept the BEV image (606) as input and locate the parking space (102) depicted in the BEV image (606) regardless of the configuration of the parking space (102) and other environmental factors (e.g., time of day, weather conditions). The output of the machine learning model (608) is parking space prediction data (610), which contains the locations of the corners (two or more) of the parking representation for all identified parking spaces (102) shown in the given BEV image (606), as well as an indication of the parking space surface material and confidence level for each parking space (102). In addition, using the parking space prediction data (610), entry line data and parking space orientation can be determined for each parking space (102).

[0052] To better understand the output included in the parking space prediction data (610), it is useful to explain how the parking space (102) is represented by the parking representation. According to one or more embodiments, a quadrilateral parking representation is used to represent the parking space (102) by an area surrounding the parking space (102). The quadrilateral parking representation has four corners. In one or more embodiments, each corner of the parking representation indicates a specific corner of the parking space (102). That is, the four corners are explicitly indicated as one of the following: entry-left, entry-right, end-left, and end-right. Furthermore, the corner names of the parking space are mutually exclusive. Figures 7A-7C Each depicts a parking space (102) represented by a quadrilateral parking representation and a search vehicle (202) that intends to enter and park in the detected parking space (102). Again, it should be noted that not Figures 7A-7C All parking spaces (102) are marked to avoid unnecessarily cluttering the drawing.

[0053] Go to Figure 7A, a parking representation has four uniquely labeled corners; namely, a left entry corner (702), a right entry corner (704), a left end corner (706), and a right end corner (708). It should be noted that the designations of "left" and "right" can be relative to the orientation of the vehicle, relative to the BEV image (606), or relative to any other fixed reference, as long as the relative designations are applied consistently. Typically, when the corner names are known and the parking representation can use two or more corners, the parking representation is fully defined by describing the locations of its corners. Given that the pixels of the BEV image (606) are spatially distributed on a plane, the coordinate system is two-dimensional with axes parallel to the edges of the BEV image (606). For example, in one or more embodiments, the x-axis is defined as extending along the width of the BEV image (606) and the y-axis is defined as extending along the height of the BEV image (606). The origin of the coordinate system can be defined arbitrarily, as long as it remains fixed across all provided BEV images (606). In one or more embodiments, the origin of the coordinate system is located at the center of each provided BEV image (606). Additionally, it should be noted that the center of the BEV image (606) is typically aligned with the center of the rear axle of the vehicle (301), which is a common standard. The units of the coordinate system can be easily transformed between pixel space and physical space, so that the parking space detection method and system disclosed herein are not limited to any unit selection. Additionally, in one or more embodiments, the coordinate system can use normalized units. For example, the x-axis and y-axis coordinates can be given in pixel or physical space relative to the width and height of the BEV image (606), respectively. In one or more embodiments, and as will be described in more detail later in this disclosure, a cell grid overlaid on the BEV image (606) can be used to segmentally define the coordinate system used with the BEV image (606). In this case, the coordinate system can be normalized according to the width and height of each grid cell.

[0054] The locations of the corners of the parking representation may be described using absolute or center-relative coordinates. Figure 7A The parking space (102) is represented using absolute coordinates (701). In this case, by using the coordinate pair ( ) to directly locate each corner by specifying its position on the x and y axes. Again, these coordinates may be normalized and may be referenced to an origin defined by the BEV image (606) or an origin defined by a grid cell of the BEV image (606). To distinguish a coordinate pair of a corner from any other corner of the parking representation, each coordinate pair is labeled with an index according to the corner to which it is associated. In using the four corners 7A to 7C, the left entry corner (702) is indexed with the number 1, the right entry corner (704) uses index 2, the left end corner (706) uses index 3, and the right end corner (708) is indexed with the number 4. Thus, for example, using Figure 7A , the position of the right corner (708) is given by the coordinate pair (x4, y4). The assignment of corners to indices is arbitrary, and one of ordinary skill in the art will recognize that any choice of mutually exclusive indices may be used with the corners of the parking representation, provided that the choice is applied consistently, and therefore the indices shown herein do not represent a limitation on the present disclosure. In the absolute coordinates (701), assuming that the index coordinate pair (e.g., (x1, y1)) indicates which corner is being referenced and where that corner is located, the parking representation of the area surrounding the parking space (102) is fully defined using the index coordinate pairs (one for each corner of the parking representation). This is in Figure 7A , where a rectangular parking representation is drawn using coordinate pairs (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0055] Figure 7B The center relative coordinates (703) are used to depict the Figure 7A As before, the parking representation is fully defined by locating its specified (e.g., indexed) corners. Figure 7A Likewise, the left entrance corner (702) is given index 1, the right entrance corner (704) is given index 2, the left end corner (706) is given index 3, and the right end corner (708) is given index 4. Figure 7B In the example, the coordinate pair indicating the center point of the parking representation is used as the coordinate pair ( ). The coordinate pair representing the center of the parking representation is referred to herein as the center coordinate (712). The center coordinate (712) is related to the coordinate pair of the corner in absolute coordinate (701) as

[0056]

[0057] Where N indicates the number of corners used in the parking representation. 7A to 7C In the example shown, N = 4. However, in general, a parking representation can be defined using N ≥ 2. When defining a parking representation using two corners, the corners can be considered to be diagonally opposite corners of a rectangle. Therefore, the two corners can be used to define a parking representation that encloses a finite surface, rather than just defining line segments.

[0058] like Figure 7BAs shown, using the center relative coordinates (703), the corners of the parking representation are located relative to the center coordinates (712). That is, the coordinate pair of each corner is written as a relative coordinate pair (712) indicating the displacement from the center coordinates along the x and y axes. For example, using the center relative coordinates (703), the position of the left entrance corner (702) is represented by the relative coordinate pair ( ). Similar relative coordinate pairs can be formed for the remaining corners. Mathematically, there is a relationship between the relative coordinate pairs using the center relative coordinates of the corner (703) and the coordinate pairs using the absolute coordinates (701), according to the following relationship:

[0059]

[0060] The parking representation is a polygon with four corners. 7A to 7C In the example, the relationship between relative coordinate pairs and coordinate pairs is as follows:

[0061]

[0062] According to one or more embodiments, the center relative coordinates (703) are used to configure the machine learning model (608). Figure 7C Another example of a parking representation of a parking space (102) using center relative coordinates (703) is depicted. For simplicity, Figure 7C Not every parking space (102) and parking indication corner is marked. Figure 7C The parking indication in Figure 7B The rectangular parking representation in has the same definition, but is not rectangular. Figure 7C To emphasize the advantages of the present disclosure. Namely, regardless of the relative viewpoint of the vehicle 301, the parking representation can directly conform to and represent any orientation and shape (including non-rectangular shapes) of the parking space 102 without requiring additional rotation or affine operations. This provides a significant advantage over existing parking detection systems, which are limited to representing parking spaces (102) as rectangles or can only detect parking spaces (102) at a given orientation relative to the vehicle (301).

[0063] Additionally, an advantage of providing a specific designation to each corner (e.g., the left entry corner (702) rather than an undesignated corner) is that an entry line (710) indicating the boundary of the parking space (102) that the vehicle (301) (the search vehicle (202)) should enter) can be easily determined. According to one or more embodiments, the entry line (710) is defined by a straight line connecting the left entry corner (702) and the right entry corner (704). Mathematically, the entry line 710 of the parking representation is given by the following set of points:

[0064]

[0065] where λ is a continuous variable whose domain is [0,1].

[0066] In one or more embodiments, more than one entry line may be determined for any detected parking space (102). For example, in one or more embodiments, a parking representation is used with one or more additional object detectors capable of locating other objects such as vehicles and streets. In these cases, the spatial relationship between the parking space (102) and surrounding objects is used to determine one or more entry lines for the parking space (102). As an example, in Figure 7A In the example, the searching vehicle (202) has detected an available parking space (102) that is bounded on the top and bottom by open areas (e.g., streets) and on the sides by other available parking spaces. Therefore, this information can be used to determine that any line segment directly connecting adjacent corners of the parking representation can be considered a valid entry line.

[0067] Figure 8 The information included in the parking space prediction data (610) is described in more detail. Figure 5 As shown in the example BEV image (500), the BEV image (606) can contain more than one parking space (102). Typically, the BEV image (606) can contain zero or more parking spaces (102). The parking space prediction data (610) includes separate information for each detected parking space (701) in the BEV image (606) that is provided to the machine learning model (608). Specifically, for each detected parking space 801, the parking space prediction data 610 includes: predicted center coordinates 712; a pair of center-relative coordinates of each corner of the parking representation, collectively referred to as corner displacements (804); a confidence score between 0 and 1 indicating the visibility of each corner, where the confidence scores are collectively referred to as corner visibility (806); a car position confidence score (810) between 0 and 1 indicating the confidence that the parking representation formed using the center coordinates (712) and the corner displacements (804) accurately encloses the available parking space (102); and a classification prediction (814) of the parking space surface material. Furthermore, using the parking space prediction data (610), a categorical prediction of the parking space orientation (i.e., vertical, parallel, fishbone) can be calculated for each parking space (102). That is, in one or more embodiments, the orientation of each parking space 102 is determined by post-processing the parking space prediction data 610.

[0068] Upon receiving the BEV image (606), the machine learning model (608) outputs parking space prediction data (610). The parking space prediction data (610) contains all the information required to positively identify (according to a confidence level, vehicle location confidence (810)) and locate an available parking space (102) in the BEV image (606). The BEV image (606) represents the local surroundings of the vehicle (301). Furthermore, the BEV image (606) can be spatially mapped to the physical space surrounding the vehicle. Thus, the parking space prediction data 610 can be used to physically detect (identify and locate) an available parking space 102 local to the vehicle 102, and the parking space prediction data 610 can be used with an onboard driver assistance system to automatically (i.e., without driver input) park the vehicle 301.

[0069] Figure 9 Depicts Figure 5 Example BEV image (500) of FIG. 5 , wherein parking space prediction data (610) output by the machine learning model (608) has been used to label the example BEV image (500). Figure 9 As shown, using the center coordinates (712) and the corner displacements (804) (where four corners are used in this example) allows a quadrilateral parking representation to be drawn around each detected available parking space (102). Additionally, an entry line (710) is determined for each parking space (102), and Figure 9 The car position confidence (810) of each parking space (102) is marked in FIG. Figure 9 Detection of a blocked parking space (902) is also depicted. From the perspective of the vehicle (301), the blocked parking space is partially blocked by the parked vehicle (204). Therefore, for the parking representation representing the blocked parking space (902), the corner visibility of the left corner (706) and the right corner (708) is expected to be close to zero, or to be thresholded to zero, indicating that these corners are not visible. Finally, as shown from Figure 9 It can be inferred that the parking representation can be used to easily determine the surface area and entrance line width of the detected parking space (102). Note that in order to avoid Figure 9 Confusing, not every parking space (102) has a reference, nor Figure 9 Each parking space (102) has a reference (with a number label and line) to each corner, center coordinate (712), entry line (710), and car position confidence (810).

[0070] As described above, a machine learning model (608) is used to generate parking space prediction data (610) from the BEV image (606). Machine learning, broadly defined, is the extraction of patterns and insights from data. The phrases "artificial intelligence," "machine learning," "deep learning," and "pattern recognition" are often used interchangeably and synonymously throughout the literature. This ambiguity arises because the field of "extracting patterns and insights from data" has developed simultaneously and disjointly in many classical fields such as mathematics, statistics, and computer science. For consistency, this article will adopt the term machine learning or machine learning, however, those skilled in the art will recognize that the concepts and methods detailed below are not limited by this choice of nomenclature.

[0071] Those skilled in the art will recognize that the fields and concepts encompassed by machine learning are both too broad and too deep to be fully described herein. However, in order to provide the necessary context for the machine learning models employed by one or more embodiments of the present invention, a minimal description of neural networks and convolutional neural networks is provided in the following paragraphs. It is emphasized that the following description is intended to provide a rough understanding of some machine learning methods and models and should not be considered a limitation of the present disclosure.

[0072] One type of machine learning model is a neural network. A neural network can often be used as a subcomponent of a larger machine learning model. Figure 10 A diagram of a neural network is shown in FIG. At a high level, a neural network (1000) can be graphically depicted as consisting of nodes (1002) and edges (1004), where Figure 10 In the example, any circle represents a node, and the edge (1004) is Figure 10 Nodes (1002) can be grouped to form layers (1005). Figure 10 Four layers (1008, 1010, 1012, 1014) of nodes (1002) are shown, wherein the nodes (1002) are grouped into columns, however, the grouping need not be as Figure 10As shown. Edges (1004) connect nodes (1002). Edges (1004) may or may not connect to any node (1002), regardless of which layer (1005) the node (1002) is in. That is, nodes (1002) may be sparsely and residually connected. However, when every node (1002) in a layer is connected to every node in the adjacent layer (1002), the layer is said to be densely or fully connected. If all layers in the neural network (1000) are densely connected, the neural network (1000) may be referred to as a densely or densely connected (or fully connected) neural network (1000). The neural network (1000) will have at least two layers (1005), where the first layer (1008) is considered the "input layer" and the last layer (1014) is the "output layer". Any intermediate layers (1010, 1012) are typically described as "hidden layers". The neural network (1000) may have zero or more hidden layers (1010, 1012), and a neural network (1000) having at least one hidden layer (1010, 1012) may be described as a "deep" neural network or a "deep learning method." Thus, in some embodiments, the machine learning model is a deep neural network. Typically, the neural network (1000) may have more than one node (1002) in the output layer (1014). In this case, the neural network (1000) may be referred to as a "multi-objective" or "multi-output" network.

[0073] Nodes (1002) and edges (1004) have additional associations. That is, each edge is associated with a numerical value. The edge values, or even the edges (1004) themselves, are often referred to as "weights" or "parameters." When training the neural network (1000), numerical values ​​are assigned to each edge (1004). In addition, each node (1002) is associated with a numerical variable and an activation function. Activation functions are not limited to any class of functions, but traditionally follow the form

[0074]

[0075] where j is an index into the set of "incoming" nodes (1002) and edges (1004), and f is a user-defined function. The incoming nodes (1002) are the nodes that are considered as a graph (e.g. Figure 10 Some functions for f may include linear functions. , S-type function and the rectified linear unit function , however, many additional functions are often employed. Each node (1002) in the neural network (1000) may have a different associated activation function. Often, as a shorthand, an activation function consists of a function To describe. That is, by the linear function The constructed activation function can be simply referred to as a linear activation function without causing too much ambiguity.

[0076] When the neural network (1000) receives input, the input is propagated through the network according to the activation function and the incoming node (1002) values ​​and edge (1004) values ​​to calculate the value of each node (1002). That is, the value of each node (1002) can change for each input received. Occasionally, the node (1002) is assigned a fixed value, such as a value of 1, which is not affected by the input or changes according to the edge (1004) value and the activation function. The fixed node (1002) is often called a "bias" or "bias node" (1006). Figure 10 Shown in dotted circles.

[0077] In some embodiments, the neural network (1000) may include specialized layers (1005), such as normalization layers (batch or layer-by-layer normalization may occur) or dropout layers, or additional connection processes, such as cascading. One of ordinary skill in the art will appreciate that such variations are within the scope of this disclosure.

[0078] As noted, the training process of a neural network (1000) includes assigning values ​​to edges (1004). To begin training, edges (1004) are assigned initial values. These values ​​can be assigned randomly, according to a prescribed distribution, manually, or by some other assignment mechanism. Once the edge (1004) values ​​have been initialized, the neural network (1000) can act as a function such that it can receive inputs and produce outputs. In this way, at least one input is propagated through the neural network (1000) to produce an output. During training, a data set, often referred to as a training set or training data, is provided to the neural network (1000). The training set consists of inputs and associated target(s), where the target(s) represent "ground truth" or otherwise desired outputs. The inputs are processed by the neural network (1000), and the outputs of the neural network (1000) are compared to the targets associated with the input data. The comparison of the outputs of the neural network (1000) with the targets is often performed by a so-called "loss function," although other names for this comparison function are often employed, such as "error function," "objective function," "value function," and "cost function." Many types of loss functions are available, such as mean squared error functions, however, a general characteristic of a loss function is that the loss function provides a numerical estimate of the similarity between the output of the neural network (1000) and the associated target. In some embodiments, the loss function may be composed of multiple loss functions applied to different parts of the output-target comparison. The loss function may also be constructed to impose additional constraints on the values ​​assumed by the edges (1004), for example by adding penalty terms (which may be physics-based) or regularization terms. Typically, the goal of a training procedure is to change the values ​​of the edges (1004) to promote similarity between the output of the neural network (1000) and the associated targets on the dataset. Thus, the loss function is used to guide the changes made to the values ​​of the edges (1004), typically through a process known as "backpropagation."

[0079] While a full review of the backpropagation process is beyond the scope of this disclosure, a brief overview is provided. Backpropagation involves computing the gradient of the loss function with respect to the edge (1004) values. The gradient indicates the direction of change in the edge (1004) value that resulted in the largest change in the loss function. Because the gradient is local to the current edge (1004) value, the edge (1004) value is typically updated by a "step size" in the direction indicated by the gradient. The step size is often referred to as the "learning rate" and does not need to remain fixed during the training process. Additionally, the step size and direction can be informed by previously seen edge (1004) values ​​or previously calculated gradients. This approach to determining step direction is often referred to as a "momentum" based approach.

[0080] Once the edge (1004) values ​​have been updated or changed from their initial values ​​by the backpropagation step, the neural network (1000) will likely produce different outputs. Therefore, the following process is repeated: propagating at least one input through the neural network (1000), comparing the neural network (1000) output with the associated target to a loss function (wherein the comparison may be piecewise with more than one loss function), computing the gradient of the loss function with respect to the edge (1004) values, and updating the edge (1004) values ​​using a gradient-guided step until a termination criterion is reached. Common termination criteria are: reaching a fixed number of edge (1004) updates, otherwise known as an iteration counter; decrementing the learning rate; noting that the loss function does not change significantly between iterations; reaching a specified performance metric, as evaluated on the training data or a separate holdout dataset. Once the termination criteria are met and no further changes to the edge (1004) values ​​are attempted, the neural network (1000) is said to be "trained". Note that, depending on the construction of the loss function, in some cases, minimizing the loss function may be used to improve the similarity between the target and the neural network (1000) output. In other cases, the objective may be to maximize a loss function (if maximization is the objective, the loss function is often referred to as the objective or value function). One of ordinary skill in the art will appreciate that the tasks of maximization and minimization can be made equivalent through techniques such as inversion. In other words, when updating the edge (1004) value with a step size guided by the gradient, the step size may be in the direction of the gradient or in the opposite direction to the gradient, depending on the construction of the loss function.

[0081] The machine learning model architecture defines the entire structure of the machine learning model. For example, in the case of a neural network (1000), the number of hidden layers in the network, the type of activation function used, and the number of outputs must be specified. Additionally, the use and placement of specialized layers such as batch normalization must be defined. Each of these choices, such as the choice of how many hidden layers to use in the neural network (1000), is known as a hyperparameter of the machine learning model. In other words, the machine learning model architecture specifies the hyperparameters surrounding the machine learning model. Note that the machine learning model architecture does not describe the values ​​of the machine learning model's edges (weights, parameters). These must be learned during training or otherwise specified when using a pre-trained model.

[0082] Another type of machine learning model is a convolutional neural network (CNN). A CNN is similar to a neural network (1000) in that it can be technically represented graphically by a series of edges (1004) and nodes (1002) grouped together to form layers. However, it is more informative to think of a CNN as a structural grouping of weights; the term structure here indicates that the weights within a group have relationships. CNNs are widely used when the input data also has structural relationships (e.g., a spatial relationship where one element of the input is always considered to be "to the left" of another element of the input). For example, images have structural relationships because each pixel (element) has a directional relationship with respect to its neighbors.

[0083] The structured groupings or groups of weights are referred to herein as "filters". The number of weights in a filter is typically much smaller than the number of elements in the input (e.g., pixels in an image). In a CNN, a filter can be thought of as "sliding" or convolving with the input data to form an intermediate output or intermediate representation of the input data that still has a structural relationship. Similar to the neural network (1000), the intermediate output is typically further processed with an activation function. Many filters can be applied to the input data to form many intermediate representations. Additional filters can be formed to operate on the intermediate representations, thereby creating more intermediate representations. This process can be repeated as specified by the user. When convolving with the input, the filter can be strided so that some elements (e.g., pixels) of the input are skipped. For example, a group of intermediate output representations can be pooled by considering only the maximum value of the group in subsequent calculations. Strides and pooling can be used to downsample the intermediate representation. Similar to the neural network (1000), additional operations such as normalization, concatenation, dropout, and residual connections can be applied to the intermediate representation. In addition, the intermediate representation can be upsampled, for example, by techniques such as transposed convolutions. In a CNN, there is a "final" set of intermediate representations where no more filters act on them. In some instances, the structural relationships of the final intermediate representations are dissolved; a process known as "flattening." The flattened representations are typically passed to a neural network (1000), or at least a densely connected layer, to produce the final output. Note that in this context, the neural network (1000) is still considered part of the CNN. In other instances, the structural relationships of the final layer (i.e., the CNN output) are preserved, reorganized, or reshaped for interpretation. Similar to the neural network (1000), after initializing the filter weights and the edge (1004) values ​​(if any) of the internal neural network (1000), the CNN is trained using a backpropagation process according to a loss function.

[0084] According to one or more embodiments, the machine learning model (608) used in the parking space detection method and system disclosed herein is a CNN. In particular, in one or more embodiments, the architecture of the CNN is similar to the well-known "You Only Look Once" (YOLO) object detection model. Various versions of YOLO exist and differ in aspects such as the type of layers used, the resolution of the training data, etc. However, a defining characteristic of all YOLO versions is that multiple objects of different scales can be detected in a single pass. In addition, the most recent YOLO architecture divides the input image into grid cells, and each grid cell has one or more associated anchor boxes. In one or more embodiments, the machine learning model (608) follows the same layer structure as YOLOv4.

[0085] While the machine learning model (608) described herein may be modeled after YOLO, there are a number of key differences. The first difference is the way the target parking representations of the training data are matched to the anchor boxes. In order to match the target parking representations to the anchor boxes, a training dataset must be provided. According to one or more embodiments, a private dataset is collated and manually annotated using one or more vehicles (301), each equipped with one or more cameras capable of generating a view of the surroundings of the vehicle (301) (e.g., see Figure 3 ). The private dataset can be updated and / or enhanced with additional data (e.g., BEV images) as it becomes available. In one or more embodiments, the private dataset contains annotated BEV images (606), wherein each BEV image (606) in the private dataset spans an area of ​​25 meters by 25 meters, which is a significant increase from the publicly available dataset. In addition, the private dataset covers a wide range of parking space configurations. The private dataset is used to train, configure, and evaluate the machine learning model (608). As a common procedure, the private dataset is divided into a training dataset and a test dataset. In one or more embodiments, the private dataset is divided into a training dataset, a validation dataset, and a test dataset.

[0086] The training and test datasets consist of many input-target pairs, where the input is a BEV image (606) (e.g., the example BEV image (500)), and the associated target is a data structure formatted similarly to the parking space prediction data (610), but with ground truth, or the precise (manually determined or semi-automatically determined) location of each parking representation of the parking space (102) in the BEV image (606). In order to appropriately format the target data structure for each BEV image (606), each ground truth parking representation in a given BEV image (606) must be matched to at least one anchor box.

[0087] Figure 11A ground truth parking representation (1102) from an annotated BEV image (606) is depicted, where, in this example, the ground truth parking representation (1102) is defined using four corners. The center coordinates (712) and corner displacements (804) of the ground truth parking representation (1102) are known. Thus, the exact locations of the left entry corner (702), the right entry corner (704), the left end corner (706), and the right end corner (708) are known (or can be determined). Given the center coordinates 712, the ground truth parking representation 1102 is associated with a grid cell (not shown). Each grid cell is associated with K anchor boxes, where K is an integer greater than or equal to 1. Figure 11 Two anchor boxes are depicted, namely, a first anchor box (1104) and a second anchor box (1106). It should be noted that, according to one or more embodiments, although the ground truth parking representation (1102) is a non-rectangular polygon, the anchor boxes are rectangular. In order to match the ground truth parking representation (1102) with the anchor boxes, first, the envelope width (1110) and envelope height (1112) of the ground truth parking representation (1102) are calculated. The envelope width (1110) and envelope height (1112) are the width and height, respectively, of the smallest rectangular box (1108) that completely encloses the ground truth parking representation (1102). The envelope width (1110), w, is calculated as

[0088]

[0089] where x n is the absolute coordinate of the nth corner along the x-axis. For example, in the case of a parking representation with four corners, x1 is the absolute coordinate of the left entry corner (702) along the x-axis, x2 is the absolute coordinate of the right entry corner (704) along the x-axis, x3 is the absolute coordinate of the left end corner (706) along the x-axis, and x4 is the absolute coordinate of the right end corner (708) along the x-axis. Similarly, the envelope height (1112), h, is calculated as

[0090]

[0091] Wherein, the reference y indicates the absolute coordinate along the y-axis of the corner indexed from 1 to N. Note that although the term absolute coordinate is used, this term is intended to distinguish between center-relative coordinate systems. In practice, the absolute coordinates can be normalized relative to the width and height of the BEV image (606) or the width or height of the grid cell. In addition, the origin of the x-axis and y-axis can be placed relative to the BEV image (606) or the grid cell.

[0092] Since each anchor box is rectangular, it has a width and a height. To determine which anchor box should be associated with the ground truth parking representation, the width and height of each anchor box are compared with the envelope width (1110) and envelope height (1112) of the ground truth parking representation. Specifically, the anchor box compatibility score is calculated for each of the K anchor boxes as follows:

[0093]

[0094] where k is used as each anchor box in the K anchor boxes (i.e. 1≤ ≤ ), and and denotes the width and height of the k-th anchor box, respectively. According to one or more embodiments, anchor boxes with an anchor box compatibility score greater than a compatibility threshold are immediately removed from consideration. If a single anchor box satisfies the criteria set by the compatibility threshold, then this anchor box is associated with the ground truth parking representation (1102). In the event that more than one anchor box has a compatibility score less than the compatibility threshold, each of these anchor boxes is associated with the ground truth parking representation (1102). In one or more embodiments, the compatibility threshold is set to 2.

[0095] A second difference of the parking space detection system of the present disclosure is that during training, a custom loss function is used to update the parameters of the machine learning model (608). Custom loss function The form is as follows

[0096]

[0097] in , , , , and can all be considered as separate loss functions each corresponding to a portion of the parking space prediction data 610, and , , , and is a weighting term. Typically, a weighting term can be configured to give its associated loss term greater or lesser importance and / or scale the loss term so that the loss terms are on the same order of magnitude. In one or more embodiments, one or more weighting terms can be set to zero. That is, individual loss functions can be removed from the custom loss function without departing from the scope of this disclosure.

[0098] Loss Item and For evaluating the accuracy of the predicted parking representation in the parking space prediction data (610) relative to the ground truth parking representation (1102) represented in the target data structure. It is called the intersection of aggregated corners over union loss. The intersection of aggregated corners over union loss is defined as

[0099]

[0100] in represents the corner intersection on the union value. As can be seen, the aggregated corner intersection union loss is determined using all N corners of the parking representation. Specifically, The distance between the ground truth and the predicted parking representation is determined using an intersection-over-union (IoU) based function applied to the i-th corner of the ground truth and predicted parking representation. The IoU based function may be the original IoU function or any other IoU variant, such as the generalized intersection-over-union (GIoU) ​​function or the distance intersection-over-union (DIoU) function, without departing from the scope of the present disclosure.

[0101] To better understand the aggregated corner IoU loss and how to apply the IoU-based function to each corner of the parking representation, this paper provides a ) is used as an example of an IoU-based function. For this example, it is useful to know the Generalized Intersection over Union (GIoU) ​​function. GIoU is a metric that evaluates how close two convex shapes are in terms of shape and position. In general, for any two arbitrary convex shapes A and B given as , the Intersection over Union (IoU) of these two shapes is

[0102]

[0103] Furthermore, the smallest convex shape that contains A and B is C. Then, the generalized intersection-over-union (GIoU) ​​of two shapes containing A and B is determined according to the following formula:

[0104]

[0105] When using the GIoU function in the aggregated corner intersection over union loss, to compare a single ground truth parking representation (1102) with its predicted parking representation, the aggregated corner intersection over union loss is calculated as

[0106]

[0107] in is the generalized intersection-over-union of the two rectangles associated with the i corner of the ground truth and predicted parking representation. The generalized intersection-over-union of i corners can be best understood by illustrating . Figure 12Depicts a ground truth parking representation (1102) and an associated predicted parking representation (1201), wherein the parking representations are defined using four corners. The ground truth parking representation (1102) and the predicted parking representation (1201) are each defined by their respective center coordinates (center coordinate (712) and predicted center coordinate (1212), respectively) and their corner displacements. Specifically, in the case of a parking representation using four corners Figure 12 In the example of FIG, the ground truth parking representation (1102) is defined by the left entrance corner (702), the right entrance corner (704), the left end corner (706), and the right end corner (708). Similarly, the predicted parking representation 1201 is defined by the predicted left entrance corner 1202, the predicted right entrance corner 1204, the predicted left end corner 1206, and the predicted right end corner 1208. Figure 12 As shown, the predicted parking representation 1201 is not precisely aligned with the ground truth parking representation 1102. The purpose of the aggregated corner intersection-over-union loss is to quantify the alignment between the ground truth parking representation (1102) and the predicted parking representation (1201). As can be seen in the above equation, this is done by averaging the IoU-based values ​​for each corner of the ground truth and predicted parking representation. Using the right corner (i.e., i=4) as an example, Figure 12 A first rectangle (1214) and a second rectangle (1216) are depicted, the first rectangle (1214) having diagonally opposite corners positioned at the center coordinates (712) and the right end corner (708) of the ground truth parking representation (1102), and the second rectangle (1216) having diagonally opposite corners positioned at the predicted center coordinates (1212) and the predicted right end corner (1208) of the predicted parking representation (1201). The intersection of the first rectangle (1214) and the second rectangle (1216) is at Figure 12 The first rectangle (1214) and the second rectangle (1216) can be considered as two convex shapes, and the IoU-based value (i.e., GIoU) ​​between the first rectangle (1214) and the second rectangle (1216) can be calculated. In this example, the IoU-based function is the generalized intersection over union (GIoU) ​​function. Therefore, in Figure 12 In the example, the generalized union between the first rectangle (1214) and the second rectangle (1216) is called the generalized union of the right corners (corner i=4). In general, for Figure 12 In a similar manner, two rectangles can be formed for each of the four corners of the parking representation in [ 1102 ]. Thus, continuing with this example, the generalized intersection-over-union (GIU) of each corner can be determined using rectangles formed from the center coordinates to the associated corners of both the ground truth parking representation ( 1102 ) and the predicted parking representation ( 1201 ).

[0108] For a given pair of ground truth parking representations (1102) and associated predicted parking representations (1201), the aggregated corner intersection-over-union loss L can be calculated by averaging the corner intersection-over-union values ​​C of the corners defining the parking representations. It should be emphasized that Figure 12 Only the calculation of the generalized intersection-over-union (GIU) of the 4th corner is illustrated, however, the same process can be applied to the remaining corners to calculate Figure 12 The aggregated corner intersection-over-union (IoU) loss for the parking representation shown in . Furthermore, it is emphasized that while the aforementioned examples use the generalized intersection-over-union (GIoU) ​​function, the aggregated corner intersection-over-union (IoU) loss can use any IoU-based function (e.g., IoU, DIoU, GIoU, etc.) without departing from the scope of this disclosure.

[0109] It is called corner distance loss in this paper. Again, using Figure 12 As an example, the corner distance of the 4th corner is depicted (1220). Generally speaking, Corner distance Indicates the Euclidean distance between the i-corner of the ground truth parking representation (1102) and the i-corner of the predicted parking representation (1201). Mathematically, the i-corner distance Determined to be

[0110]

[0111] where x i and y i The absolute coordinates of the corner i along the x-axis and y-axis are indicated respectively, and the superscripts refer to the ground truth parking representation (1102) (gt) or the predicted parking representation (1201) (pred). Corner distance loss L CD The calculation is as follows:

[0112]

[0113] In one or more embodiments, the corner distance loss also includes clamping, thresholding, and scaling operations. Given a corner distance threshold, In one or more embodiments, the corner distance loss is defined as

[0114]

[0115] is the car position confidence loss. The car position confidence loss operates on the car position confidence (810) output by the machine learning model (608). The car position confidence (810) is a value between 0 and 1 that represents the confidence that the predicted parking representation (1201) represents the ground truth parking representation (1102). The parking space confidence (810) is similar to the objectness score in a standard object detection network. In one or more embodiments, the car position confidence loss is a standard binary cross entropy loss applied to the car position confidence (810) of the predicted parking representation (1201) while learning whether the ground truth parking representation (1102) is associated with the predicted parking representation (1201). In one or more embodiments, the car position confidence loss is a binary cross entropy loss with a logarithmic loss function.

[0116] is the loss of corner visibility. The corners of a parking representation or parking space (102) may be defined as visible or occluded. Within the parking space prediction data (610), the corner visibility (806) indicates the predicted state of each corner of the detected parking space (102), visible or occluded. In practice, corner visibility is a continuous variable between 0 and 1 indicating the likelihood of a given corner being visible. Thus, a binary cross entropy loss may be applied to each corner of the predicted parking representation (1102) and the associated ground truth parking representation (1201), if present, and the corner visibility loss is the average of the binary cross entropy losses at each corner.

[0117] at last, is a surface material loss. The surface material loss is applied to the predicted parking space surface material (814) and the known parking space (102) surface material. In one or more embodiments, the surface material loss uses a categorical cross entropy loss function.

[0118] According to one or more embodiments, the parking space prediction data (610) only includes the center coordinates (712), corner displacements (804), and vehicle position confidence (810) of each detected parking space (102). In this case, the custom loss function is defined as

[0119]

[0120] The parking space detection data (610) is likely to predict that many parking spaces (102) have low car position confidence (810) values. According to one or more embodiments, parking spaces (102) with low car position confidence (810) values ​​can be removed from consideration based on a user-provided confidence threshold. For example, in one or more embodiments, only parking spaces (102) defined by the parking space prediction data (610) and having a confidence (810) value greater than 0.6 are retained and / or considered. In other embodiments, post-processing techniques such as non-maximum suppression (NMS) can be applied to remove or filter out low-confidence predicted parking representations and merge overlapping predicted parking representations.

[0121] Figure 13 A training step of a machine learning model (608) according to one or more embodiments is depicted. For training, it is assumed that a training dataset comprising one or more BEV images (606) and associated target or real data (annotations) is provided. To implement the training step, at least one BEV image input (1302) must be provided. The BEV image input (1302) used for training is associated with ground-truth parking representation data (1304) and BEV image input metadata (1306). The ground-truth parking representation data (1304) describes the ground-truth parking representation of the available parking spaces (102) in the BEV image input (1302). The BEV image input metadata (1306) includes all additional information about the parking spaces in the BEV image input (1302). For example, the BEV image input metadata (1306) indicates the surface material and corner visibility of the parking spaces (102) in the BEV image input (1302).

[0122] A target data structure (1307) must be created for the BEV image input (1302). To create the target data structure (1307), information about each ground-truth parking representation in the BEV image input (1308) is encoded in the target data structure (1307). This is accomplished by referencing the ground-truth parking representation data (1304) for each (1308) ground-truth parking representation in the BEV image input (1302) according to the following steps. First, as shown in block 1310, the grid cell associated with the ground-truth parking representation is identified. Then, the ground-truth parking representation center coordinates are scaled relative to the grid cell. Next, in block 1312, the envelope width and envelope height of the ground-truth parking representation are calculated. Next, in block 1314, the ground-truth parking representation is matched to at least one rectangular anchor box according to the methods previously described herein. Finally, for each ground-truth parking representation, the corner displacements are transformed to match the scale of the associated anchor box, as shown in block 1316. Once these steps have been applied to each ground-truth parking representation in the BEV image input (1302), the scaled center coordinates and corner displacements of each ground-truth parking representation can be appended with the associated BEV image input metadata (1306) and injected into the appropriate location in the target data structure (1307) based on the identified grid cells and associated anchor boxes.

[0123] Additionally, the BEV image input (1302) is processed by the machine learning model (608) to generate parking space prediction data (610). In block 1318, the corner displacements in the parking space prediction data (610) are represented, scaled, or transformed according to the size of the associated anchor box for which they are predicted. In block 1320, the parking space prediction data (610) is compared to the target data structure (1307) using a custom loss L. As shown in block 1322, the gradient of the custom loss (608) is determined relative to the parameters of the machine learning model. Next, in block 1324, the parameters of the machine learning model are updated, guided by the gradient.

[0124] Note that although Figure 13The parameters of the machine learning model are depicted as being updated when evaluating a single BEV image input (1302), but in general, the updates can occur when processing any number of BEV image inputs (1302). That is, one of ordinary skill in the art will recognize that the training process can be applied to batches of inputs without imposing limitations on the present disclosure. Furthermore, the processes of generating the target data structure (1307) and producing the parking space prediction data (610) need not occur simultaneously or in parallel. In one or more embodiments, the target data structure (1307) for each BEV image input (1302) in the training dataset is determined and stored prior to training the machine learning model (608). Additionally, according to one or more embodiments, the training dataset can be augmented using any data augmentation technique known in the art. For example, the training dataset can be augmented by randomly applying one or more of the following techniques to each BEV image (606) in the training dataset: vertical flip; horizontal flip; random rotation, hue-saturation-value (HSV) color space adjustment.

[0125] It is emphasized that while the previous examples referenced a CNN (e.g., a YOLO architecture), the methods and techniques disclosed herein (e.g., custom losses, anchor box determination, etc.) are not limited to this choice of machine learning model. In one or more embodiments, the machine learning model (608) is a visual transformer (ViT) trained using at least the custom loss described above.

[0126] According to one or more embodiments, to improve the speed and efficiency of the parking space detection method and system disclosed herein, so that parking spaces (102) can be detected in real time using a computing system on a vehicle (301), the machine learning model (608) is implemented in a compiled computer language. In one or more embodiments, the machine learning model is implemented in C++.

[0127] In one or more embodiments, the predicted parking representation 1201 determined using the parking space prediction data 610 is post-processed using a Canny filter correction method. The Canny filter correction method slightly shifts the corners of the predicted parking representation (1201) to the closest point output by the Canny filter. As previously mentioned, the vehicle (301) can be equipped with additional sensing systems, such as ultrasonic systems and LiDAR systems. In one or more embodiments, the sensing data received from the additional sensing systems is used to refine the predicted parking representation 1201.

[0128] Figure 14A flowchart outlining a general process for training a machine learning model (608) according to one or more embodiments is depicted. In block 1402, a plurality of bird's eye view (BEV) images are collected. Each BEV image is constructed from one or more images acquired using one or more cameras disposed on a vehicle (301). The plurality of BEV images contain various parking space (102) configurations (e.g., parking space orientation, parking space surface material, marking line type, etc.) under various environmental conditions (e.g., nighttime, daytime, rainy, etc.) and settings (i.e., indoors, outdoors). In block 1404, an available parking space (102) is identified in each of the plurality of BEV images. Typically, a BEV image may contain zero or more available parking spaces (102). In one or more embodiments, the available parking spaces (102) are identified manually or by using a semi-automatic iterative process. In block 1406, each of the identified available parking spaces is represented using a ground truth parking representation. Each ground truth parking representation is represented by a center coordinate of the identified available parking spot (102) and two or more center-relative coordinate pairs corresponding to corners of the ground truth parking representation, wherein the ground truth parking representation is defined by the two or more corners. In block 1408, for each identified available parking spot (102) in a given BEV image, a grid cell of the BEV image associated with the identified parking spot (its ground truth parking representation) is identified. In block 1410, for each identified available parking spot (102), an envelope width and an envelope height are calculated. The envelope width and the envelope height correspond to the width and height, respectively, of a minimum rectangle that encloses the ground truth parking representation of the identified available parking spot (102). In block 1412, for each identified parking spot (102) in the given BEV image, its ground truth parking representation is matched to one or more anchor boxes. That is, each identified available parking spot (102) in the given BEV image is matched to at least one anchor box. Considering a single given ground-truth parking representation, matching is accomplished by comparing the envelope width and envelope height of the ground-truth parking representation with the width and height of the candidate anchor boxes. Note that in one or more embodiments, a candidate anchor box is an anchor box associated with a grid cell of the given ground-truth parking representation. However, in other embodiments, any anchor box may be considered a candidate anchor box regardless of any grid cell association of the given ground-truth parking representation or anchor box. In block 1416, a target data structure (1307) is generated for each of the plurality of BEV images. Each target data structure (1307) contains at least a center coordinate and a center-relative coordinate pair (i.e., a corner displacement) of each identified available parking space (102) in the associated BEV image. In other words, each target data structure (1307) contains a ground-truth parking representation for each identified available parking space (102) in the associated BEV image.The target data structure (1307) is constructed relative to the grid cells and anchor boxes of the BEV image. In box 1418, a training dataset is generated. The training dataset includes multiple BEV images and their associated target data structures (1307). Finally, in box 1420, at least the training dataset is used to train the machine learning model (608). In one or more embodiments, a portion of the training dataset may be retained (not used for training) as a validation dataset and / or a test dataset. The machine learning model (608) is trained using a backpropagation process, wherein the parameters of the machine learning model (608) are updated using guidance from a customized loss function. The resulting trained machine learning model (608) can directly receive the BEV image and output parking space prediction data (610). The parking space prediction data (610) includes at least the predicted center coordinates and corner displacements of the predicted parking space (102). In one or more embodiments, the parking space prediction data (610) is post-processed using scaling and Canny filtering methods to further refine and locate the predicted parking space (102). Because the multiple BEV images contain parking spaces (102) of many configurations, the trained machine learning model is robust and can accurately detect parking spaces (102) regardless of their orientation, surface material, and other factors.

[0129] Figure 15 A flow chart outlining the use of a trained machine learning model (608) according to one or more embodiments is depicted. In block 1502, a first image is received from a first camera disposed on a vehicle (301). In block 1504, a BEV image is generated from the first image. In one or more embodiments, the BEV image is generated from the first image using an inverse perspective mapping (IPM) method. In block 1506, the BEV image (608) is processed using the machine learning model. Again, for Figure 15 For the purpose of Figure 14The machine learning model (608) is trained according to the flowchart of FIG. . Thus, the machine learning model (608) is configured to generate or output parking space prediction data (610) when processing the BEV image. The parking space prediction data includes a first center coordinate and first corner displacement data of a first available parking space, wherein the first available parking space is visible within the BEV image. The first corner displacement data includes a relative coordinate pair for each corner of the parking representation of the first available parking space. That is, the first corner displacement data includes at least: a first relative coordinate pair locating the first corner relative to the first center coordinate and a second relative coordinate pair locating the second corner relative to the first center coordinate. In one or more embodiments, the parking representation may be defined using four corners, such that the first corner displacement data also includes a third relative coordinate pair locating the third corner relative to the first center coordinate and a fourth relative coordinate pair locating the fourth corner relative to the first center coordinate. The coordinate pairs each carry a mutually exclusive corner designation. For example, in one or more embodiments, the first corner is the left entrance corner (702). In one or more embodiments, the corner designations are provided using indices. For example, the corners may be designated as corner 1, corner 2, and so on, up to the last corner (recall that the number of corners may be two or more). The first center coordinates and the first corner displacement data are sufficient to fully define a first parking representation (i.e., a predicted parking representation) that represents a prediction of a location of the first available parking space. The parking space prediction data (610) also includes a first parking location confidence that indicates the confidence of the machine learning model (610) that the first parking representation is accurately aligned with the available parking space. Continuing, in block 1506, a first location (610) of the first available parking space is determined using the parking space prediction data. Specifically, the first location is determined using the first parking representation that represents the predicted location (i.e., the first location) of the first available parking space. In one or more embodiments, the first location indicates the location of the first available parking space in the physical space of the vehicle (301). Determining the first location may require scaling of the first parking representation, however, one of ordinary skill in the art will recognize that coordinate conversions between model space and physical space are not beyond the scope of this disclosure. In block 1510, given the predicted first location, the vehicle (301) can be parked in the first available parking space. In one or more embodiments, when the first parking position confidence meets or exceeds a predetermined threshold, the vehicle (301) is parked in the first available parking space. For example, in one or more embodiments, the threshold is set to 0.5. In one or more embodiments, the vehicle (301) is parked in the first available parking space without driver assistance. That is, the vehicle (301) is parked automatically. In one or more embodiments, the first available parking space (102) is suggested to a user of the vehicle (e.g., the driver), and the user can confirm or reject the parking space.Finally, it should be noted that when the vehicle (301) is parked, as a first available parking space is detected and a threshold is satisfied, one or more images acquired by one or more cameras disposed on the vehicle (301) may be used to continuously generate BEV images. The BEV images acquired during parking may be processed by the machine learning model (608) to continuously update (in real time) the first center coordinate and first corner displacement data (i.e., the predicted parking representation and the first position) relative to the vehicle (301) and assist in the parking process (e.g., determining and monitoring a proposed trajectory for the vehicle).

[0130] Embodiments of the parking detection method and system disclosed herein have at least the following advantages. The embodiments disclosed herein utilize a parking representation of an area surrounding a parking space (102) to identify an available parking space (102). The parking representation is defined using two or more corners. Therefore, the representation of the parking space (102) is not limited to a rectangular form. In addition, by representing the available parking space (102) with the parking representation, the embodiments disclosed herein can detect the available parking space (102) without rotation or affine operations, regardless of the relative viewpoint of the vehicle. In addition, the embodiments disclosed herein specify a corner name for each corner of the parking representation of the parking space (102). Therefore, one or more entrance lines of the available parking space (102) can be easily determined directly from the parking representation without the need for a dedicated and separate entrance line detector. In addition, the embodiments disclosed herein output parking space prediction data (610) directly from the machine learning model (608) without the need for multiple neural network heads. That is, the machine learning model (608) does not need to be divided into separate prediction tasks. Furthermore, in one or more embodiments, the parking space prediction data (610) classifies the visibility of each corner of the parking representation, and the parking representation can be used to represent the parking space (102) even when one or more of the corners of the parking space (102) are obscured. Another major advantage is that the BEV images can be acquired and processed by the machine learning model (608) throughout the parking process to continuously monitor the relative position of the parking space (102) to the vehicle (301) and ensure successful vehicle parking. That is, in one or more embodiments, the onboard driver assistance system of the vehicle 301 uses the parking space prediction data 610 when actively parking or "driving" the vehicle into the detected available parking space 102. Finally, the parking representation can be used to easily determine the area enclosed by the parking space (102), regardless of whether the parking space (102) is obscured.

[0131] The embodiments of the present invention can be implemented on almost any type of computing system, regardless of the platform used. For example, the parking space detection method and system discussed in the embodiments of this article can be implemented as follows Figure 16The computing system shown may include one or more computing systems. Such computing systems may be one or more mobile devices (e.g., laptops, smartphones, personal digital assistants, tablet computers, or other mobile devices), desktop computers, servers, blades in a server chassis, one or more ECUs in a vehicle, or any other type of computing device or devices that include at least minimal processing power, memory, and input and output devices to perform one or more embodiments of the present invention. For example, Figure 16 As shown, the computing system (1600) may include one or more computer processors (1602), associated memory (1604) (e.g., random access memory (RAM), cache memory, flash memory, etc.), one or more storage devices (1606) (e.g., a hard disk, an optical drive (such as a compact disc (CD) drive or a digital versatile disc (DVD) drive), a flash memory stick, etc.), and many other elements and functions. The computer processor(s) (1602) may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores or microcores of a processor. The computing system 1600 may also include one or more input devices 1610, such as a touch screen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, or any other type of input device. In addition, the computing system (1600) may include one or more output devices (1608), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input device. The computing system (1600) can be connected to a network (1612) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) via a network interface connection (not shown). Input and output devices can be connected locally or remotely (e.g., via the network (1612)) to the computer processor (1602), memory (1604), and storage device (1606). Many different types of computing systems exist, and the aforementioned input and output devices can take other forms.

[0132] The software instructions in the form of computer-readable program code for performing embodiments of the present invention may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium, such as a CD, DVD, storage device, floppy disk, tape, flash memory, physical memory, or any other computer-readable storage medium. Specifically, the software instructions may correspond to a computer-readable program code that, when executed by a processor, is configured to perform embodiments of the present invention.

[0133] In addition, one or more elements of the aforementioned computing system 1600 can be located at a remote location and connected to the other elements via a network 1612. In addition, one or more embodiments of the present invention can be implemented on a distributed system having multiple nodes, wherein each part of the present invention can be located on a different node within the distributed system. In one embodiment of the present invention, the nodes correspond to different computing devices. Alternatively, the nodes can correspond to computer processors with associated physical memory. Alternatively, the nodes can correspond to computer processors or micro-cores of computer processors with shared memory and / or resources.

[0134] While many of the embodiments described herein utilize machine learning models based on convolutional neural networks, those skilled in the art will recognize that the parking space detection methods and systems disclosed herein can readily be used with other types of machine learning models. Therefore, the embodiments disclosed herein are not limited to the use of machine learning models based on convolutional neural networks. Machine learning models such as the Visual Transformer (ViT) can readily be plugged into this framework without departing from the scope of this disclosure.

[0135] Furthermore, while the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the claims appended hereto.

Claims

1. A method comprising: obtaining a first image from a first camera disposed on the vehicle; generating a bird's eye view (BEV) image using the first image; The BEV image is processed using a machine learning model to generate parking space prediction data, wherein the parking space prediction data includes: a first center coordinate of a first available parking space; The first corner displacement data includes: a first relative coordinate pair locating a first corner relative to said first center coordinate; and a second relative coordinate pair locating a second corner relative to the first center coordinate; First parking position confidence; determining a first location of a first available parking space using the parking space prediction data; and When the first parking position confidence satisfies a threshold, the vehicle is parked in the first available parking space.

2. The method according to claim 1, wherein The parking space prediction data also includes: a second center coordinate of a second available parking space; The second corner displacement data includes: a third relative coordinate pair locating a third corner relative to the second center coordinate; a fourth relative coordinate pair locating a fourth corner relative to the second center coordinate; and The method also includes determining a second location of a second available parking space using the parking space prediction data.

3. The method according to claim 1, further comprising: obtaining a second image from a second camera disposed on the vehicle; The BEV image is generated using the first image and the second image.

4. The method according to claim 1, wherein The BEV image is generated by inverse perspective mapping.

5. The method according to claim 1, wherein While the vehicle is parked, the first center coordinate and the first corner displacement data are continuously updated based on processing newly generated BEV images based on a machine learning model.

6. The method according to claim 1, further comprising: representing the first available parking space with a first parking representation, wherein the first parking representation is completely specified by the first center coordinate and the first corner displacement data; determining one or more entry lines to the first available parking space; and An area enclosed by the first parking representation is determined.

7. The method according to claim 1, wherein The parking space prediction data also includes: first corner visibility data, wherein the first corner visibility data indicates whether each corner in the first corner displacement data is visible or occluded; and A first parking space surface material, wherein the first parking space surface material specifies a material category of the first available parking space.

8. The method according to claim 1, further comprising: A parking space orientation of the first available parking space is determined, wherein the parking space orientation is one of vertical, parallel, and fishbone.

9. A computer-implemented method for training a machine learning model, comprising: Acquire multiple bird's eye view (BEV) images; For each BEV image in the plurality of BEV images: Identify available parking spaces; For each available parking space identified: Determining a ground truth parking representation of the identified available parking spaces, including: Center coordinates; and Corner displacement; calculating an envelope width and an envelope height using the center coordinates and the corner displacements; and matching the identified available parking space to one or more anchor boxes using the envelope height and the envelope width; and generating a target data structure comprising center coordinates and corner displacements for each identified available parking space; generating a training dataset comprising a plurality of BEV images and their associated target data structures; and A machine learning model is trained using the training dataset, wherein the machine learning model is configured to directly receive one or more BEV images from the plurality of BEV images.

10. The method according to claim 9, wherein: The target data structure for each BEV image of the plurality of BEV images further includes parking space surface material data and corner visibility data for each identified available parking space.

11. The method according to claim 9, wherein The machine learning model is trained using a loss function comprising: Aggregate corner intersection-over-union loss; Corner distance loss; and The vehicle position confidence is lost.

12. The method according to claim 9, further comprising: The training dataset is enhanced by applying at least one of the following techniques to one or more BEV images of the plurality of BEV images: vertical flipping, horizontal flipping, rotation, and color adjustment.

13. The method according to claim 9, wherein: For each identified parking space, also include: scaling the center coordinates relative to the grid cells; and The corner displacements are transformed into the scale of the one or more matched anchor boxes.

14. The method according to claim 11, wherein for a given predicted parking representation and an associated ground truth parking representation, the aggregated corner intersection-over-union loss is an average of intersection-over-union functions calculated for each corner of the given predicted parking representation and the associated ground truth parking representation; and Wherein, for the given predicted parking representation and the associated ground truth parking representation, the corner distance loss is an average Euclidean norm calculated for each corner of the given predicted parking representation and the associated ground truth parking representation.

15. A system comprising: vehicle; a first camera disposed on the vehicle; Bird's-Eye View (BEV) imagery; Machine learning models; and A computer comprising one or more computer processors, the computer being configured to: obtaining a first image from the first camera; constructing the BEV image based on the first image; Processing the BEV image using the machine learning model to generate parking space prediction data, wherein the parking space prediction data includes: a first center coordinate of a first available parking space; The first corner displacement data includes: a first relative coordinate pair locating a first corner relative to said first center coordinate; and a second relative coordinate pair locating a second corner relative to said first center coordinate; and First parking position confidence; determining a first location of a first available parking space using the parking space prediction data; and When the first parking position confidence satisfies a threshold, the vehicle is parked in the first available parking space without driver assistance.

16. The system according to claim 15, wherein: The parking space prediction data also includes: a second center coordinate of a second available parking space; The second corner displacement data includes: a third relative coordinate pair, the third relative coordinate pair locating a third corner relative to the second center coordinate; a fourth relative coordinate pair locating a fourth corner relative to the second center coordinate; and Second parking position confidence; The computer is further configured to use the parking space prediction data to determine a second position of the second available parking space.

17. The system of claim 15, further comprising: a second camera disposed on the vehicle, Wherein, the computer is further configured to: obtaining a second image from the second camera; and The BEV image is constructed using the first image and the second image.

18. The system according to claim 15, wherein: While parking the vehicle, the machine learning model is used to process newly acquired BEV images to continuously update the first center coordinates and the first corner displacement data.

19. The system of claim 15, wherein: The parking space prediction data also includes: first corner visibility data, wherein the first corner visibility data indicates whether each corner in the first corner displacement data is visible or occluded; and A first parking space surface material, wherein the first parking space surface material specifies a material category of the first available parking space.

20. The system of claim 15, wherein: The computer is further configured to: A parking space orientation of the first available parking space is determined, wherein the parking space orientation is one of vertical, parallel, and fishbone.

Images