A parking space detection method and tracking method thereof, a parking space detection device, a parking space detection apparatus, and a computer readable storage medium

Abstract

The application relates to a parking space detection method and tracking method thereof, a parking space detection device, a parking space detection equipment and a computer readable storage medium. The parking space detection method and tracking method thereof comprise the following steps: S1, a surround view picture of a parking space is acquired, low-level semantic features and high-level semantic features of the surround view picture are extracted, the low-level semantic features and the high-level semantic features are fused, and fused features are acquired; S2, a decoding operation is performed on the fused features, and multiple groups of branches are acquired; and S3, reasoning is performed according to the branches to acquire all detected parking spaces. The application provides a parking space detection method and tracking method thereof, a parking space detection device, a parking space detection equipment and a computer readable storage medium, and the accuracy and robustness of parking space detection and tracking can be improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a parking space detection method and tracking method, a parking space detection device, a parking space detection equipment, and a computer-readable storage medium. Background Technology

[0002] With rapid societal development, the number of private cars in large cities is increasing, making it difficult to find available parking spaces. A study shows that over 50% of drivers cannot find free parking in heavily trafficked areas. Furthermore, 23% of all car collisions occur in parking lots.

[0003] In this context, the Park Assist System (PAS) was developed, which consists of three parts: target location identification, path planning, and parking guidance or path tracking. Target location identification, as a crucial component of the PAS system, is tasked with accurately detecting parking spaces, tracking them, and transmitting the parking information to the control system in real time.

[0004] Based on parking space detection methods, PAS (Parking Space Detection System) technology can be broadly divided into two categories: one uses ultrasonic sensors, laser scanners, stereo cameras, short-range radar, structured light, etc. This type of method requires obstacles (vehicles) to be parked in front of and behind the target parking space, which has significant limitations. The other type determines parking spaces by recognizing features in parking space images. This method is applied to low-speed parking scenarios and can refine the recognition results through multi-frame detection, representing the current development trend in parking space detection technology. Furthermore, with increasing emphasis on vehicle safety, countless vehicles are equipped with around-view monitors (AVMs), which can monitor the area around the vehicle in 360 degrees. Therefore, vacant parking space detection in surrounding view images can fully utilize existing equipment on the vehicle without incurring additional equipment costs.

[0005] Parking space detection in panoramic images needs to meet both real-time performance and robustness. A series of marker-based parking space detection methods have been proposed, but these methods rely on designed features to detect markers that are easily affected by changes in lighting. To achieve more robust marker detection, methods based on deep convolutional neural networks (DCNNs) have gradually gained attention in recent years. Due to their powerful feature extraction capabilities, DCNNs significantly improve the accuracy of parking space detection compared to traditional methods. However, they cannot determine parking space occupancy, and the subsequent steps of inferring the complete parking space are cumbersome and unstable. Methods such as DeepPS and DMPR-PS require combining corner distance information to obtain parking space location information, but cannot determine whether a parking space is occupied. VPS-Net divides vacant parking space detection into two steps: object detection and classification. First, object detection determines the location of the parking space entrance line, and then a classification network determines whether the parking space is occupied. However, in complex real-world environments, classification networks often perform poorly. In addition, some semantic segmentation-based methods, such as VH-HFCN and DFNet, have also been proposed in recent years. While these methods perform well in ground marking and segmentation, they require post-processing to generate parking spaces, which is time-consuming and inaccurate. Therefore, improving the accuracy and robustness of parking space detection and occupancy status has become an urgent problem to be solved.

[0006] Furthermore, if parking spaces are to be tracked, the DBT (Detection-Based Tracking) method is the preferred method. However, current DBT tracking frameworks mostly use the parameters of the target rectangle or the image features corresponding to the rectangle as input, and the output of the tracking framework is also related to the parameters of the rectangle. Due to the perspective of the panoramic view, the parking space will more or less change and become a general parallelogram. The traditional rectangle tracking framework is no longer applicable and cannot complete the tracking. It is also impossible to calculate the data association. The existing technology needs to be improved. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, this invention proposes a parking space detection method and its tracking method, a parking space detection device, a parking space detection equipment, and a computer-readable storage medium, which can improve the accuracy and robustness of parking space detection and tracking.

[0008] Specifically, this invention proposes a parking space detection method, including the following steps:

[0009] S1, acquire a surround view image of the parking space, extract low-level semantic features and high-level semantic features from the surround view image, fuse the low-level semantic features and high-level semantic features to obtain fused features;

[0010] S2, Decode the fused features to obtain multiple branches, including an angle regression branch, a target localization branch, a target category prediction branch, and a target detection confidence branch. The angle regression branch is used to predict the tilt angle of the parking space's entrance line. The target localization branch is used to predict the position of the parking space's entrance line and corner points. The target category prediction branch is used to predict the target category, which includes the entrance line and corner points. The target detection confidence branch is used to determine whether there are currently entrance lines and corner points.

[0011] S3, perform reasoning based on the branch to obtain all detected parking spaces.

[0012] According to an embodiment of the present invention, step S3 includes the following steps:

[0013] S31, merge data, merge the data of multiple branches into multiple sets of data, the multiple sets of data include entry line data and corner point data, the entry line data includes the entry line position obtained based on the target positioning branch and the inclination angle of the entry line obtained based on the angle regression branch, and the corner point data includes the corner point position obtained based on the target positioning branch;

[0014] S32, Data matching: Match each entry line data with all corner point data separately to obtain all matching results. This matching process includes determining whether the corner point belongs to the entry line based on the coordinate position relationship between the entry line and the corner point. If so, determine that the entry line data and the corner point data match, and add the corner point data to the entry line data.

[0015] S33, Data Reasoning: Based on the matching results, all detected parking spaces are inferred. The reasoning process includes inferring the location information of the remaining corner points of the detected parking space containing the entrance line based on the entrance line data and corner point data.

[0016] According to one embodiment of the present invention, the multiple branches obtained in step S2 further include occupancy branches, which are used to predict the occupancy status of parking spaces;

[0017] The entrance line data and corner point data mentioned in step S31 include the parking space occupancy status obtained based on the occupancy branch;

[0018] Step S3 also includes step S34, which determines whether a parking space is available based on the parking space occupancy status of the entrance line data.

[0019] According to an embodiment of the present invention, in step S32, the intersection area of ​​the entrance line and the corner point is calculated. If the intersection area is greater than the area of ​​the corner point than a preset value, then the corner point is considered to belong to the entrance line, and the two are matched.

[0020] According to an embodiment of the present invention, step S33 includes the following steps:

[0021] S331, if the entrance line data contains only one corner point data, then the position of another corner point belonging to the entrance line is inferred based on the texture information of the parking space; if the entrance line data contains multiple corner point data, then the two corner points that are farthest apart are selected as the two corner points belonging to the entrance line.

[0022] S332, based on the inclination angle of the entrance line and the positions of the two corner points belonging to the entrance line, infer the position information of the other two corner points of the detected parking space.

[0023] According to an embodiment of the present invention, in step S31, each of the multiple sets of data includes data features {x, y, w, h, confidence, class, occupied, angle};

[0024] Where x, y, w, and h are obtained based on the target positioning branch and are used to represent the position of the entrance line or corner point of the parking space. x and y represent the center coordinates of the entrance line or corner point, and w and h represent the width and length of the area corresponding to the entrance line or corner point, respectively.

[0025] The class indicates that the target category is determined based on the predicted branch. If the target category is the entry line, then the data in this group is the entry line data; if the target category is the corner point, then the data in this group is the corner point data.

[0026] Confidence represents the probability of an entry line or corner point existing at the current location, based on the target detection confidence branch and the target category.

[0027] "occupancy" indicates the parking space occupancy status obtained based on the occupancy branch.

[0028] angle represents the tilt angle of the entry line obtained based on the angle branch.

[0029] According to an embodiment of the present invention, the step of obtaining the class includes: obtaining the scores of the two target categories corresponding to the target category prediction branch, and taking the larger score as the target category;

[0030] And / or, the steps of obtaining conf include: multiplying the value output by the target detection confidence branch by the scores of the two target categories output by the target category prediction branch, taking the higher value of the product and comparing it with a set threshold, and judging the possibility of an entry line or corner point at the current position based on the comparison result;

[0031] And / or, the steps to obtain occupy include: reading the scores of the occupied and idle states in the occupied branch, taking the larger value as the parking space occupancy state, and assigning a value to occupy according to the parking space occupancy state.

[0032] This invention also provides a parking space tracking method, which uses the detection results obtained by the aforementioned parking space detection method, and includes the following steps:

[0033] T1, obtain the parking space set O of the current frame based on the parking space detection method according to any one of claims 1 to 8, predict the parking space of the current frame based on the trajectory set T updated in the previous frame, and obtain the predicted trajectory set P of the current frame.

[0034] T2, obtain the IOU matching results between the parking space set O and the predicted trajectory set P, including the following steps:

[0035] T21, Based on the parking space set O obtained from the detection result of the current frame and the predicted trajectory set P of the current frame, establish the IOU matrix between the parking space set O and the predicted trajectory set P, including the following steps:

[0036] T211, find the intersection point of the overlapping area between each parking space Oi in the parking space set O and each predicted trajectory Pn in the predicted trajectory set P, where parking space Oi∈parking space set O; predicted trajectory Pn∈P;

[0037] The overlapping area is a polygon. The area of ​​the overlapping area is calculated based on the intersection points of the polygons. The area of ​​the parking space Oi and the area of ​​the predicted trajectory Pn corresponding to the overlapping area are also calculated. The IOU value is calculated based on the area of ​​the overlapping area, the area of ​​the parking space Oi, and the area of ​​the predicted trajectory Pn.

[0038] T212, Based on the correspondence between the IOU value and the parking space Oi and the predicted trajectory Pn, establish the IOU matrix;

[0039] T22, Matching judgment: Obtain a set of IOU values ​​that meet the optimal match in the IOU matrix, and output the parking space and predicted trajectory corresponding to the set of IOU values ​​as the matching result;

[0040] T3, update the trajectory set T, update the trajectory Tm corresponding to the predicted trajectory Pn according to the parking space Oi in the matching result, obtain the updated trajectory set P and go to step T1, where Tm∈trajectory set T, the update of trajectory Tm includes updating the actual position information TL of trajectory Tm in the current frame, updating the occupancy status information TC of trajectory occupancy status, updating the motion model, and updating the lifetime of trajectory validity.

[0041] The present invention also provides a parking space detection device for implementing the aforementioned parking space detection method, comprising:

[0042] The image module is used to obtain surround view images of the parking space;

[0043] The extraction module is used to extract the low-level semantic features and high-level semantic features of the panoramic image;

[0044] The fusion module is used to fuse the low-level semantic features and the high-level semantic features to obtain fused features;

[0045] The decoding module is used to perform decoding operations on the fused features to obtain multiple branches, including an angle regression branch, a target localization branch, a target category prediction branch, and a target detection confidence branch. The angle regression branch is used to predict the tilt angle of the entrance line of the parking space. The target localization branch is used to predict the position of the entrance line and corner point of the parking space. The target category prediction branch is used to predict the target category, which includes the entrance line and corner point. The target detection confidence branch is used to determine whether there is an entrance line and corner point at present.

[0046] The reasoning module is used to perform reasoning based on the branches to obtain all detected parking spaces.

[0047] The present invention also provides a parking space detection device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the preceding parking space detection methods.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the parking space detection method described in any of the preceding claims.

[0049] This invention provides a parking space detection method and its tracking method, parking space detection device, parking space detection equipment, and computer-readable storage medium. It is an end-to-end detection method based on deep learning. It obtains multiple branches by extracting features from the surrounding images, fusing features, and decoding. Based on the inference of multiple branches, it obtains all detected parking spaces, which can improve the accuracy and robustness of parking space detection. On the basis of the above parking space detection method, it also utilizes the IOU matching method in step T2 to make the parking space tracking results more accurate and stable.

[0050] It should be understood that the above general description and the following detailed description of the invention are exemplary and illustrative, and are intended to provide further explanation of the invention as described in the claims. Attached Figure Description

[0051] The accompanying drawings are included to provide further explanation of the invention; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of the invention and, together with this specification, serve to explain the principles of the invention. In the drawings:

[0052] Figure 1 A flowchart of a parking space detection method according to an embodiment of the present invention is shown.

[0053] Figure 2 An example diagram of a parking space detection method according to an embodiment of the present invention is shown.

[0054] Figure 3 yes Figure 2 Line drawing diagram.

[0055] Figure 4 A flowchart of step S3 of a parking space detection method according to an embodiment of the present invention is shown.

[0056] Figure 5 A schematic diagram of the detection results obtained according to the parking space detection method is shown.

[0057] Figure 6 A flowchart of a parking space tracking method according to an embodiment of the present invention is shown.

[0058] Figure 7 A schematic diagram of a polygon representing overlapping parking spaces according to an embodiment of the present invention is shown.

[0059] Figure 8 A schematic diagram of a parking space detection device according to an embodiment of the present invention is shown. Detailed Implementation

[0060] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0062] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0063] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0064] In the description of this application, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is usually based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0065] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.

[0066] Figure 1A flowchart of a parking space detection method according to an embodiment of the present invention is shown. As shown, the present invention provides a parking space detection method. The parking space detection method includes the following steps:

[0067] S1. Obtain a surround view image of the parking space, extract low-level semantic features and high-level semantic features from the surround view image, and fuse the low-level semantic features and high-level semantic features to obtain fused features.

[0068] In step S1, surround-view images of the vehicle's surroundings can be acquired using an AVM (Around View Monitor). Low-level and high-level semantic features of the surround-view images are extracted via a backbone network. This process involves inputting the acquired surround-view images into the backbone network and extracting their low-level and high-level semantic features for subsequent network use. Low-level semantic features have higher resolution and contain more location and detail information; these features represent the location and detail information of the surround-view images. High-level semantic features have stronger semantic information but lower resolution and poorer detail perception; these features represent semantic information features. Next, the low-level and high-level semantic features are fused to obtain fused features. The process of obtaining fused features includes using methods such as convolution, addition, and upsampling to combine deep and shallow information, constructing feature pyramid structures of different sizes. This not only aggregates features from different layers but also ensures the integrity and diversity of the features.

[0069] Generally, the backbone network in step S1 uses CSPDarknet (Cross-Stage-Partial-connections Darknet), and the feature fusion method employs one or more combinations of FPN (Feature Pyramid Network), PANet (Path Aggregation Network), NAS-FPN (Neural Architecture Search Feature Pyramid Network), and BiFPN (Bi-directional Feature Pyramid Network). It's easy to understand that in many deep learning applications (e.g., object detection, image segmentation), fusing features at different scales is a crucial method for improving performance. Low-level semantic features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantic validity is lower and they contain more noise. High-level semantic features have stronger semantic information, but their resolution is low, and their ability to perceive details is poor. How to efficiently fuse the two, taking advantage of their strengths and discarding their weaknesses, is key to improving the segmentation model. Currently, commonly used feature fusion methods include FPN, PANet, NAS-FPN, and BiFPN. These methods use convolution, addition, upsampling, etc., to combine deep and shallow information to construct feature pyramid structures of different sizes.

[0070] S2, decode the fused features to obtain multiple branches. These branches include an angle regression branch, a location branch, an object class prediction branch, and an object detection confidence branch. The angle regression branch predicts the tilt angle of the parking space's entrance line; the location branch predicts the positions of the parking space's entrance line and corner points; the object class prediction branch predicts the object class, which includes the entrance line and corner points; and the object detection confidence branch determines whether an entrance line and corner points currently exist.

[0071] Preferably, step S2 includes using convolutional layers to perform a secondary channel transformation on the fused features and extracting the features corresponding to each branch. The parameters in each convolutional layer are generated through multiple training iterations.

[0072] Preferably, step S2 includes the following steps:

[0073] S21, use convolutional layers to perform channel transformation on the fused features to unify the number of channels;

[0074] S22, Input the convolutional layer to extract the features corresponding to each branch;

[0075] S23 uses a convolutional layer to perform channel transformation in order to obtain the number of channels corresponding to each branch.

[0076] Figure 2 An example diagram of a parking space detection method according to an embodiment of the present invention is shown. Figure 3 yes Figure 2 A line drawing. (For example...) Figure 3 As shown, the black arrow indicates the tilt direction of the parking space entrance line, which has a tilt angle. This tilt angle can be determined based on any direction in the panoramic view image coordinate system, using a custom clockwise or counterclockwise direction as the reference axis. The target localization branch predicts the positions of the parking space entrance line and corner points, which typically include the coordinate information of the entrance line and corner points. The target category prediction branch predicts the target category, which includes entrance lines and corner points. Points P1, P2, P3, and P4 are the four corner points of the parking space. The dashed box 301 represents the position of the entrance line, and the solid boxes 302 and 303 represent the positions of corner points P1 and P2, respectively. The target detection confidence branch determines whether there are currently entrance lines and corner points.

[0077] S3, perform reasoning based on the branches to obtain all detected parking spaces. Detected parking spaces include at least their location information.

[0078] Figure 4 A flowchart of step S3 of a parking space detection method according to an embodiment of the present invention is shown. As shown, preferably, step S3 includes the following steps:

[0079] S31, Merge data: Combine data from multiple branches into multiple sets of data. These multiple sets of data include entry line data and corner point data. The entry line data includes the entry line position obtained from the target positioning branch and the inclination angle of the entry line obtained from the angle regression branch. The corner point data includes the corner point positions obtained from the target positioning branch.

[0080] S32, Data matching: Match each entry line data with all corner point data separately to obtain all matching results. This matching process includes determining whether a corner point belongs to the entry line based on the coordinate position relationship between the entry line and the corner point. If so, determine that the entry line data and the corner point data match and add the corner point data to the entry line data.

[0081] S33, Data Reasoning: Based on the matching results, all detected parking spaces are inferred. The reasoning process includes inferring the location information of the remaining corner points of the detected parking space containing the entrance line based on the entrance line data and corner point data.

[0082] Based on the above steps, the location detection of vacant parking spaces can be realized. That is, after training based on the input entrance line data and corner point data, the trained model can output the vacant parking spaces around the current vehicle based on the obtained surround view images of the parking spaces. Figure 5 The diagram illustrates the detection results obtained using the parking space detection method. As shown, the parking space detection method outputs four detected parking spaces ① to ④ in the diagram for subsequent parking space tracking and other applications.

[0083] Furthermore, the multiple branches obtained in step S2 also include an occupancy branch, which is used to predict the occupancy status of parking spaces. In step S31, the entrance line data and corner data contain the parking space occupancy status obtained based on the occupancy branch. Step S3 also includes step S34, which determines whether a parking space is available based on its occupancy status. This setup further combines object detection, angle regression, and vacancy judgment, and achieves end-to-end training, enabling accurate and real-time detection of the location, direction, and occupancy status of parking spaces. That is, after training based on the input entrance line data, corner data, and occupancy status, the trained model, based on the obtained surround-view images of parking spaces, can output the parking spaces around the current vehicle and their occupancy status. Figure 5 The output of the 6 parking spaces is as follows: 4 vacant parking spaces ①~④ and 2 occupied parking spaces ⑤⑥.

[0084] The following example, which includes multiple branches and also occupies branches, will be used to explain the process of merging data in step S31.

[0085] Preferably, in step S31, the data of each branch is merged into multiple sets of data, each set of data including data features {x, y, w, h, confidence, class, occupy, angle}.

[0086] Where x, y, w, and h are obtained based on the target positioning branch and are used to represent the position of the entrance line or corner point of the parking space. x and y represent the center coordinates of the entrance line or corner point, and w and h represent the width and length of the area corresponding to the entrance line or corner point, respectively.

[0087] The class indicates that the target category is determined based on the predicted branch. If the target category is the entry line, then the data in this group is the entry line data; if the target category is the corner point, then the data in this group is the corner point data.

[0088] Confidence represents the probability of an entry line or corner point existing at the current location, based on the target detection confidence branch and the target category.

[0089] "occupancy" indicates the parking space occupancy status obtained based on the occupancy branch.

[0090] angle represents the tilt angle of the entry line obtained based on the angle branch.

[0091] Preferably, the class is obtained by scoring and comparing the targets in the target category prediction branch. Specifically, the scores of the two target categories corresponding to the target category prediction branch are obtained, the scores of the two target categories are compared, and the category with the higher score is taken as the target category. For example, the target category prediction branch outputs scores for the current target as either an entry line or a corner point. If the current output score for the entry line is 0.2 and the score for the corner point is 0.8, the scores are compared, and the category corresponding to the higher score is taken as the target category. Since the corner point score is greater than the entry line score, the current target category is determined to be a corner point.

[0092] Confidence is obtained through the object detection confidence branch and the object category prediction branch. Specifically, the value output by the object detection confidence branch is multiplied by the scores of the two categories output by the object category prediction branch, and the higher product value is output. This output is then compared with a pre-set threshold to determine whether there is a parking space. For example, if the threshold is set to 0.5 and the value output by the object detection confidence branch is 0.8, 0.8 is multiplied by the scores of the two target categories: the entry line score (0.2) and the corner score (0.8). The larger value, 0.64, is output. This is then compared with the threshold of 0.5. Since 0.64 is greater than 0.5, it is ultimately determined that there is a target, meaning there is a corner or entry line at the current location.

[0093] `occupancy` is determined through the occupancy branch. `occupancy` has two states: 0 (empty) and 1 (occupied). Correspondingly, the `occupy` statement determines the parking space's occupancy status by: reading the score from the occupancy branch (which has two categories: occupied and empty); comparing the two scores; and taking the larger value as the parking space's occupancy status. For example, if the occupancy branch outputs an occupancy score of 0.2 and an empty score of 0.8, comparing the empty and occupied scores and taking the larger empty score of 0.8 as the parking space's status (empty), then `occupy` will have a value of 0.

[0094] The angle is obtained through the angle regression branch, that is, by directly reading the data from the angle regression branch.

[0095] Preferably, in step S32, the intersection area of ​​the entry line and the corner point is calculated. If the intersection area is greater than the area of ​​the corner point by a preset value, then the corner point is considered to belong to the entry line, and the two are matched. (Reference) Figure 3As mentioned earlier, points P1 and P2 are the two corner points of the detected parking spaces. The dashed box 301 represents the position of the entrance line, and the solid boxes 302 and 303 represent the positions of corner points P1 and P2, respectively. The preset value is set to 0.8. The area of ​​the solid box 302 of corner point P1 intersects with the area of ​​the dashed box 301 of the entrance line. If the ratio of this intersecting area to the area of ​​the solid box 302 of corner point P1 is greater than the preset value of 0.8, then corner point P1 is considered to belong to that entrance line. The area of ​​the solid box 303 of corner point P2 completely overlaps with the area of ​​the dashed box 301 of the entrance line, and the ratio of their intersecting area to the area of ​​the solid box 303 of corner point P2 is equal to 1, which is greater than the preset value of 0.8. Therefore, corner point P2 and intersection point P1 belong to the same entrance line.

[0096] Preferably, step S33 includes the following steps:

[0097] S331, if the entrance line data contains only one corner point, then infer the location of the other corner point belonging to that entrance line based on the texture information of the parking space; still refer to Figure 3 If the entry line data only matches one intersection point, meaning the entry line only contains corner point P1, then the location of the other corner point P2 belonging to that entry line is inferred based on the parking space's texture information. If the location of the other corner point cannot be inferred, it cannot be used as the detected parking space output.

[0098] If the entry line data contains multiple corner points, then select the two corner points that are furthest apart as the two corner points belonging to that entry line; refer to Figure 3 If the entrance line contains three corner points P1, P2 and P5, then the two corner points P1 and P2 that are farthest apart are selected as the two corner points belonging to the entrance line.

[0099] S332, based on the inclination angle of the entrance line and the positions of two corner points belonging to that entrance line, infers the position information of the other two corner points of the detected parking space. (Reference) Figure 3 Based on the inclination angle of the entrance line and the corner points P1 and P2 belonging to the entrance line, the position information of the other two corner points P3 and P4 is inferred, thereby obtaining the complete position information of the detected parking space.

[0100] Figure 6 A flowchart of a parking space tracking method according to an embodiment of the present invention is shown. As shown in the figure, the present invention also provides a parking space tracking method. This tracking method uses the detection results obtained by the aforementioned parking space detection method, and includes the following steps:

[0101] T1, based on the aforementioned parking space detection method, obtain the parking space set O of the current frame, and predict the parking space of the current frame based on the trajectory set T updated in the previous frame, and obtain the predicted trajectory set P of the current frame.

[0102] T2, obtain the IOU matching results between the parking space set O and the predicted trajectory set P, including the following steps:

[0103] T21. Based on the parking space set O obtained from the detection results of the current frame and the predicted trajectory set P of the current frame, establish the IOU matrix between the parking space set O and the predicted trajectory set P, including the following steps:

[0104] T211, find the intersection point of the overlapping area between each parking space Oi in the parking space set O and each predicted trajectory Pn in the predicted trajectory set P, where parking space Oi∈parking space set O; predicted trajectory Pn∈P;

[0105] The overlapping area is a polygon. The area of ​​the overlapping area, the area of ​​the parking space Oi, and the area of ​​the predicted trajectory Pn are calculated based on the intersection points of the polygons. The IOU value is then calculated based on the area of ​​the overlapping area, the area of ​​the parking space Oi, and the area of ​​the predicted trajectory Pn.

[0106] T212, based on the correspondence between IOU values ​​and parking space Oi and predicted trajectory Pn, establish an IOU matrix;

[0107] T22, Matching judgment, obtain a set of IOU values ​​that meet the optimal match in the IOU matrix, and output the parking space and predicted trajectory corresponding to the set of IOU values ​​as the matching result;

[0108] T3, update the trajectory set T, update the trajectory Tm corresponding to the predicted trajectory Pn according to the parking space Oi in the matching result, obtain the updated trajectory set P and go to step T1, where Tm∈trajectory set T, the update of trajectory Tm includes the update of the actual position information TL of the trajectory Tm in the current frame, the update of the occupancy status information TC of the trajectory, the update of the motion model, and the update of the lifecycle of the trajectory validity.

[0109] This invention provides a parking space tracking method. First, it obtains the parking space set O of the current frame based on the aforementioned parking space detection method. Then, it utilizes the IOU matching method in step T2, which uses polygons instead of rectangles to calculate the IOU value. Based on the established IOU matrix, it obtains the optimal matching result between the parking space set O and the predicted trajectory set P. The trajectory Tm corresponding to the predicted trajectory Pn is updated using the matching result. Regardless of how the shape of the parking space changes, its position information does not change instantaneously, making the parking space tracking prediction result more accurate and stable.

[0110] Figure 7This diagram illustrates a polygon representing overlapping parking spaces according to an embodiment of the present invention. As shown, let the four vertices of polygon S1, representing the parking space Oi in the current frame's parking space set O, be t1, t2, m3, and t4. Let the four vertices of polygon S2, representing a predicted trajectory Pn, be m1, n2, n3, and n4. Polygons S1 and S2 overlap, forming intersection points m2 and m4. The four vertices of polygon S3, formed by the overlap of polygons S1 and S2, are m1, m2, m3, and m4.

[0111] In step T211, the intersection points m2 and m4 of polygons S1 and S2 are found, and the four vertices of the overlapping polygon S3 are obtained as m1, m2, m3, and m4. The areas of polygon S1, S2, and S3 are calculated as area1, area2, and area3. The IOU is calculated as area3 / (area1 + area2 - area3).

[0112] The present invention also provides a parking space detection device for implementing the aforementioned parking space detection method. Figure 8 A schematic diagram of a parking space detection device according to an embodiment of the present invention is shown. As shown in the figure, the parking space detection device 800 mainly includes an image module 801, an extraction module 802, a fusion module 803, a decoding module 804, and an inference module 805.

[0113] Among them, the image module 801 is used to acquire surround view images of the parking space.

[0114] The extraction module 802 is used to extract low-level semantic features and high-level semantic features from the panoramic image.

[0115] The fusion module 803 is used to fuse low-level semantic features and high-level semantic features to obtain fused features.

[0116] The decoding module 804 is used to perform decoding operations on the fused features and obtain multiple branches, including an angle regression branch, a target localization branch, a target category prediction branch, and a target detection confidence branch. The angle regression branch is used to predict the tilt angle of the entrance line of the parking space. The target localization branch is used to predict the position of the entrance line and corner of the parking space. The target category prediction branch is used to predict the target category, which includes the entrance line and corner. The target detection confidence branch is used to determine whether there is an entrance line and corner.

[0117] The reasoning module 805 is used to reason based on the branches to obtain all detected parking spaces.

[0118] The present invention also provides a parking space detection device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of any of the aforementioned parking space detection methods.

[0119] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the aforementioned parking space detection methods.

[0120] The specific implementation methods and technical effects of the parking space detection device, parking space detection equipment, and computer-readable storage medium can be found in the embodiments of the parking space detection method provided by the present invention, and will not be repeated here.

[0121] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0122] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0123] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0124] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0125] It will be apparent to those skilled in the art that various modifications and variations can be made to the exemplary embodiments described above without departing from the spirit and scope of the invention. Therefore, it is intended that this invention cover modifications and variations falling within the scope of the appended claims and their equivalents.

Claims

1. A parking space detection method, comprising the following steps: S1, acquire a surround view image of the parking space, extract low-level semantic features and high-level semantic features from the surround view image, fuse the low-level semantic features and high-level semantic features to obtain fused features; S2, Decode the fused features to obtain multiple branches, including an angle regression branch, a target localization branch, a target category prediction branch, and a target detection confidence branch. The angle regression branch is used to predict the tilt angle of the parking space's entrance line. The target localization branch is used to predict the position of the parking space's entrance line and corner points. The target category prediction branch is used to predict the target category, which includes the entrance line and corner points. The target detection confidence branch is used to determine whether there are currently entrance lines and corner points. S3, perform reasoning based on the branch to obtain all detected parking spaces; Step S3 includes the following steps: S31, merge data, merge the data of multiple branches into multiple sets of data, the multiple sets of data include entry line data and corner point data, the entry line data includes the entry line position obtained based on the target positioning branch and the inclination angle of the entry line obtained based on the angle regression branch, and the corner point data includes the corner point position obtained based on the target positioning branch; S32, Data matching: Match each entry line data with all corner point data separately to obtain all matching results. This matching process includes determining whether the corner point belongs to the entry line based on the coordinate position relationship between the entry line and the corner point. If so, determine that the entry line data and the corner point data match, and add the corner point data to the entry line data. S33, Data Reasoning: Based on the matching results, all detected parking spaces are inferred. The reasoning process includes inferring the location information of the remaining corner points of the detected parking space containing the entrance line based on the entrance line data and corner point data. Step S33 includes the following steps: S331, if the entrance line data contains only one corner point data, then the position of another corner point belonging to the entrance line is inferred based on the texture information of the parking space; if the entrance line data contains multiple corner point data, then the two corner points that are farthest apart are selected as the two corner points belonging to the entrance line. S332, based on the inclination angle of the entrance line and the positions of the two corner points belonging to the entrance line, infer the position information of the other two corner points of the detected parking space.

2. The parking space detection method as described in claim 1, characterized in that, The multiple branches obtained in step S2 also include occupancy branches, which are used to predict the occupancy status of parking spaces; The entrance line data and corner point data mentioned in step S31 include the parking space occupancy status obtained based on the occupancy branch; Step S3 also includes step S34, which determines whether the detected parking space is available based on the parking space occupancy status of the entrance line data.

3. The parking space detection method as described in claim 1, characterized in that, In step S32, the intersection area of ​​the entrance line and the corner point is calculated. If the intersection area is greater than the area of ​​the corner point than a preset value, then the corner point is considered to belong to the entrance line, and the two are matched.

4. The parking space detection method as described in claim 3, characterized in that, In step S31, each set of data in the multiple sets of data includes data features {x, y, w, h, confidence, class, occupy, angle}; Where x, y, w, and h are obtained based on the target positioning branch and are used to represent the position of the entrance line or corner point of the parking space. x and y represent the center coordinates of the entrance line or corner point, and w and h represent the width and length of the area corresponding to the entrance line or corner point, respectively. The class indicates that the target category is determined based on the predicted branch. If the target category is the entry line, then the data in this group is the entry line data; if the target category is the corner point, then the data in this group is the corner point data. conf indicates the probability of an entry line or corner point existing at the current location based on the target detection confidence branch and the target category; "occupancy" indicates the parking space occupancy status obtained based on the occupancy branch. angle represents the tilt angle of the entry line obtained based on the angle branch.

5. The parking space detection method as described in claim 4, characterized in that, The steps for obtaining the class include: obtaining the scores of the two target categories corresponding to the target category prediction branch, and taking the larger score as the target category; And / or, the steps to obtain confidence include: multiplying the value output by the target detection confidence branch by the scores of the two target categories output by the target category prediction branch, taking the higher value of the product and comparing it with a set threshold, and judging the possibility of an entry line or corner point at the current position based on the comparison result; And / or, the steps to obtain occupy include: reading the scores of the occupied and idle states in the occupied branch, taking the larger value as the parking space occupancy state, and assigning a value to occupy according to the parking space occupancy state.

6. A parking space tracking method, using the detection results obtained by the parking space detection method as described in claims 1 to 5, characterized in that, Including the following steps: T1, based on the detection results, obtain the parking space set O of the current frame, and based on the trajectory set T updated in the previous frame, predict the parking space of the current frame and obtain the predicted trajectory set P of the current frame. T2, obtain the IOU matching results between the parking space set O and the predicted trajectory set P, including the following steps: T21, Based on the parking space set O obtained from the detection result of the current frame and the predicted trajectory set P of the current frame, establish the IOU matrix between the parking space set O and the predicted trajectory set P, including the following steps: T211, find the intersection point of the overlapping area between each parking space Oi in the parking space set O and each predicted trajectory Pn in the predicted trajectory set P, where parking space Oi∈parking space set O; predicted trajectory Pn∈P; The overlapping area is a polygon. The area of ​​the overlapping area is calculated based on the intersection points of the polygons. The area of ​​the parking space Oi and the area of ​​the predicted trajectory Pn corresponding to the overlapping area are also calculated. The IOU value is calculated based on the area of ​​the overlapping area, the area of ​​the parking space Oi, and the area of ​​the predicted trajectory Pn. T212, Based on the correspondence between the IOU value and the parking space Oi and the predicted trajectory Pn, establish the IOU matrix; T22, Matching judgment: Obtain a set of IOU values ​​that meet the optimal match in the IOU matrix, and output the parking space and predicted trajectory corresponding to the set of IOU values ​​as the matching result; T3, update the trajectory set T, update the trajectory Tm corresponding to the predicted trajectory Pn according to the parking space Oi in the matching result, obtain the updated trajectory set P and go to step T1, where Tm∈trajectory set T, the update of trajectory Tm includes updating the actual position information TL of trajectory Tm in the current frame, updating the occupancy status information TC of trajectory occupancy status, updating the motion model, and updating the lifetime of trajectory validity.

7. A parking space detection device for implementing the parking space detection method of claim 1, characterized in that, include: The image module is used to obtain surround view images of the parking space; The extraction module is used to extract the low-level semantic features and high-level semantic features of the panoramic image; The fusion module is used to fuse the low-level semantic features and the high-level semantic features to obtain fused features; The decoding module is used to perform decoding operations on the fused features to obtain multiple branches, including an angle regression branch, a target localization branch, a target category prediction branch, and a target detection confidence branch. The angle regression branch is used to predict the tilt angle of the entrance line of the parking space. The target localization branch is used to predict the position of the entrance line and corner point of the parking space. The target category prediction branch is used to predict the target category, which includes the entrance line and corner point. The target detection confidence branch is used to determine whether there is an entrance line and corner point at present. The reasoning module is used to perform reasoning based on the branches to obtain all detected parking spaces.

8. A parking space detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the parking space detection method as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the parking space detection method as described in any one of claims 1-5.

Images