Parking space detection method, device, equipment and storage medium

Abstract

The present application relates to a parking space detection method, device, equipment and storage medium. The method includes: obtaining a continuous multi-frame detection map of the area where the vehicle is located; identifying the identified parking space and the parking space corner points of the identified parking space in each frame detection map; based on the parking space corner points, performing parking space verification to determine the verified parking space from the identified parking space; tracking the verified parking space in the continuous multi-frame detection map to record the number of continuous visible frames identified in the continuous multi-frame detection map and the number of continuous lost frames not identified in the continuous multi-frame detection map for each verified parking space in the parking space tracking list, and deleting the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold; based on the parking space corner points of the verified parking space, determining and outputting the parking space semantic information of each verified parking space in the parking space tracking list whose number of continuous visible frames reaches a second frame number threshold. The present method can detect parking spaces at low cost and with high accuracy.

Description

Technical Field

[0001] The present application relates to the field of vehicle detection technology, and in particular to a parking space detection method, device, equipment and storage medium. Background Art

[0002] In the field of intelligent driving technology, vehicles can use various sensors installed in the vehicle to perceive information inside and outside the vehicle to assist in driving the vehicle.

[0003] For example, in a vehicle parking scenario, the vehicle can use sensors such as cameras and lidar to collect environmental information around the vehicle, and use the collected environmental information to identify the parking scenario in which the vehicle is located, and detect and output information related to the parking space in the parking scenario to assist in parking the vehicle.

[0004] However, existing parking space detection methods in parking scenarios can generally only detect the general area of ​​the parking space, and may have the defect of insufficient parking space detection accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a parking space detection method, device, equipment and storage medium that can detect parking spaces at low cost and with high precision in order to address the above technical problems.

[0006] A parking space detection method, comprising:

[0007] Obtain continuous multi-frame detection images of the area where the vehicle is located;

[0008] Identify a recognized parking space in each detection frame of the continuous multiple detection frames and a parking space corner point of the recognized parking space;

[0009] Based on the parking space corner points, performing parking space verification to determine a verified parking space from the identified parking spaces;

[0010] Tracking the verified parking spaces in the continuous multi-frame detection images by using the parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection images and the number of continuous lost frames not identified in the continuous multi-frame detection images of each verified parking space in the parking space tracking list, and deleting the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold;

[0011] For each of the verified parking spaces in the parking space tracking list whose number of continuous visible frames reaches a second frame number threshold, the parking space semantic information of the verified parking space is determined and output based on the parking space corner points of the verified parking space.

[0012] A parking space detection device, comprising:

[0013] A detection image acquisition module is used to obtain continuous multi-frame detection images of the area where the vehicle is located;

[0014] A parking space recognition module, used to recognize a recognized parking space in each detection frame of the continuous multiple detection frames and a parking space corner point of the recognized parking space;

[0015] A parking space verification module, configured to perform parking space verification based on the parking space corner points to determine a verified parking space from the identified parking spaces;

[0016] a parking space tracking module, configured to track the verified parking spaces in the continuous multi-frame detection images by using a parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection images and the number of continuous lost frames not identified in the continuous multi-frame detection images of each verified parking space in the parking space tracking list, and to delete the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold;

[0017] The parking space semantic output module is used to determine and output the parking space semantic information of each verified parking space in the parking space tracking list whose continuous visible frame number reaches a second frame number threshold based on the parking space corner point of the verified parking space.

[0018] A parking space detection device is installed in a vehicle, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the parking space detection method described in the above embodiment when executing the computer program.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the parking space detection method described in the above embodiment.

[0020] The parking space detection method, device, equipment and storage medium described above identify the parking space corner points of each identified parking space, and perform parking space verification to screen out verified parking spaces, and can pre-delete parking spaces that do not meet the requirements; use the parking space tracking list to track and record the verified parking spaces, delete the verified parking spaces whose continuous lost frames reach a first frame number threshold from the parking space tracking list, and output parking space semantic information for each verified parking space whose continuous visible frames reach a second frame number threshold, so that the verified parking spaces in the detection image of the required frame can be efficiently tracked and the parking space semantic information can be output, without the need to output the parking space semantic information of the verified parking spaces whose continuous lost frames reach a first frame number threshold, thereby saving computing resources and improving computing efficiency; because the parking space semantic information and parking space verification are based on more refined parking space corner points, rather than only on rough parking space areas, the parking space verification and the analysis of parking space semantic information can be made more accurate and detailed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is an application environment diagram of a parking space detection method in an embodiment;

[0022] Figure 2 A schematic diagram of a flow chart of a parking space detection method in one embodiment;

[0023] Figure 3 A schematic diagram of determining whether a quadrilateral formed by four corner points of an identified parking space is a convex quadrilateral in one embodiment;

[0024] Figure 4 A schematic diagram of parking space status tracking in one embodiment;

[0025] Figure 5 A schematic diagram of calculating the intersection-and-joint ratio between a first checked parking space A and a second checked parking space B in one embodiment;

[0026] Figure 6 A schematic diagram of a process for determining the direction of a trunk road in one embodiment;

[0027] Figure 7 is a schematic diagram of a centroid direction vector between a first parking space and a second parking space in one embodiment;

[0028] Figure 8 A schematic diagram of determining a parking space entrance side, a parking space depth, a parking space width, and a parking space orientation of a verified parking space in one embodiment;

[0029] Fig. 9 is a schematic diagram of an oblique parking space, a vertical parking space, and a parallel parking space in one embodiment;

[0030] Fig.10 is a schematic diagram of possible example intersection postures of a detection frame representing a non-fixed obstacle and a quadrilateral of a verified parking space in one embodiment;

[0031] Fig.11 An example of a bird's-eye view of detecting a parking space in the prior art and an example of a bird's-eye view of detecting a parking space using the parking space detection method of the present application;

[0032] Fig.12 is a structural block diagram of a parking space detection device in one embodiment;

[0033] Fig.13 FIG. 4 is an internal structure diagram of a parking space detection device in one embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] The parking space detection method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the parking space detection device 102 is loaded in the vehicle 10, and the parking space detection device 102 is connected to the environmental sensor 104 installed on the vehicle 10 for detecting the environmental information of the vehicle. The parking space detection device 102 receives the environmental information detected by the environmental sensor 104, and obtains the continuous multi-frame detection image based on the environmental information frame by frame in real time. The environmental sensor 104 in the present application can be implemented using a purely visual sensor, such as a camera, so that compared with the solution that requires the use of ultrasonic radar, laser radar, etc. to detect the parking space, the detection of the parking space can be achieved at a lower cost. While the parking space detection device 102 obtains multiple frames of detection images frame by frame in real time, it executes the parking space detection method of the present application in real time to detect and output the parking space semantic information in each frame of the detection image. The vehicle 10 can also be equipped with a display device, such as a display screen, and the parking space detection device 102 outputs the parking space semantic information to the display device, so as to display the parking space semantic information using the display device for the convenience of the user to view.

[0036] In one embodiment, Figure 2 As shown, a parking space detection method is provided, which is applied to Figure 1 Taking the parking space detection device 102 in the example as an example, the method includes the following steps S210-S250:

[0037] Step S210, obtaining a continuous multi-frame detection image of the area where the vehicle is located.

[0038] In this step, the parking space detection device 102 may utilize the environmental sensor 104 to detect frame by frame and obtain a continuous multi-frame detection image of the area where the vehicle is located.

[0039] For example, the environmental sensor 104 may include four fisheye cameras respectively fixed at the midpoint of the front bumper, the midpoint of the rear bumper, below the left rearview mirror, and below the right rearview mirror of the vehicle 10. The parking space detection device 102 receives four environmental images from the four fisheye cameras in real time, and synthesizes the four environmental images into a detection image, which can be, for example, a bird's eye view (Bird's Eye View, BEV), so that multiple frames of detection images can be obtained continuously.

[0040] Step S220, identifying the identified parking space and the parking space corner points of the identified parking space in each detection frame of the continuous multiple detection frames.

[0041] In this step, the parking space detection device 102 can use a pre-trained neural network model to identify the identified parking space in each frame of the detection image, and multiple parking space corner points of each identified parking space. Generally speaking, the parking space is rectangular or parallelogram, so each identified parking space should usually have 4 parking space corner points. In this step, one or more identified parking spaces in each frame of the detection image, and the 4 coordinate values ​​(x, y) of the 4 parking space corner points of each identified parking space can be identified. It can be understood that the coordinate values ​​(x, y) in the detection image have a one-to-one correspondence with the real geographic coordinates of the environment in which the vehicle is located, so based on the coordinate values ​​(x, y) in the detection image, the real geographic coordinates of the location in the environment corresponding to the coordinate values ​​(x, y) can be determined.

[0042] Step S230 , performing parking space verification based on the parking space corner points to determine a verified parking space from the identified parking spaces.

[0043] In this step, by using parking space verification, the misidentified parking spaces that do not meet the requirements in the identified parking spaces in the previous steps can be deleted, thereby avoiding the waste of resources caused by processing the misidentified parking spaces in the subsequent steps and improving the accuracy of parking space identification.

[0044] In one embodiment, step S230 may include steps S231 to S234:

[0045] Step S231: for each identified parking space, determine whether the parking space corner point of the identified parking space meets the parking space self-verification condition; if not, determine that the identified parking space is a verification failed parking space.

[0046] In one embodiment, the parking space self-checking condition includes one or more combinations of the following i-iv:

[0047] i. The number of parking corner points of the identified parking space meets the number of predetermined parking corner points;

[0048] For example, it can be determined whether an identified parking space contains four corner points. If an identified parking space does not contain four corner points, the verification of the identified parking space fails.

[0049] ii. The corner points of the identified parking spaces are all within the predetermined area of ​​interest;

[0050] For example, it can be determined whether the coordinates of each corner point of the identified parking space are within a predetermined region of interest (ROI). If at least one corner point of the identified parking space is not within the ROI, the identified parking space verification fails. The ROI can be defined by the field of view that can be captured by the vehicle's environmental sensor, for example, a 20m*20m square area formed by extending 10m forward, backward, left and right from the geometric centroid of the vehicle as the center point.

[0051] iii. The quadrilateral formed by the corner points of the identified parking space is a convex quadrilateral.

[0052] Generally speaking, a normal parking space should be a convex quadrilateral. If the quadrilateral formed by the corner points of the identified parking space is a concave quadrilateral, it can be determined that the verification of the identified parking space has failed.

[0053] For example, see Figure 3 As shown, the following steps a to f can be used to determine whether the quadrilateral formed by the four corner points of the identified parking space is a convex quadrilateral:

[0054] Step a. From the four parking corner points of the currently identified parking space, select a parking corner point with the own vehicle as a reference and mark it as the first corner point Corner1 (for example, the parking corner point closest to the "front bumper center ground projection point" of the own vehicle can be selected as the starting point, recorded as the first corner point Corner1), and randomly mark the remaining three parking corner points as the second corner point Corner2, the third corner point Corner3, and the fourth corner point Corner4.

[0055] Step b. Calculate the first direction vector Vec01 from the first corner point Corner1 to the second corner point Corner2, the second direction vector Vec02 from the first corner point Corner1 to the third corner point Corner3, and the third direction vector Vec03 from the first corner point Corner1 to the fourth corner point Corner4.

[0056] Step c. respectively determine a first rotation direction angle Angle_A from the first direction vector Vec01 to the second direction vector Vec02 and a second rotation direction angle Angle_B from the second direction vector Vec02 to the third direction vector Vec03.

[0057] Step d. When the first rotation direction angle Angle_A and the second rotation direction angle Angle_B have the same sign and are both positive, that is, when the rotation directions are the same and are counterclockwise, for example, see Figure 3 In the example case shown in (a), the marked corner points are ordered counterclockwise. When the first rotation direction angle Angle_A and the second rotation direction angle Angle_B have the same sign and are both negative, that is, the rotation directions are the same and are clockwise, for example, see Figure 3 In the example case shown in (b), the marked corner points are sorted clockwise;

[0058] Step e. When the first rotation direction angle Angle_A and the second rotation direction angle Angle_B have different signs, for example, see Figure 3In the example case shown in (c), respectively, try to swap the marks of the second corner point Corner2 and the third corner point Corner3 and perform step bd again, and try to swap the marks of the third corner point Corner3 and the fourth corner point Corner4 and perform step bd again, until the corner points are successfully marked and sorted in one of the attempts;

[0059] Step f. After the marked corner points are sorted, four corner points are connected in sequence according to the marked corner points to form a quadrilateral, and the four internal angles of the quadrilateral are calculated. When any of the four internal angles is greater than 180 degrees, for example, see Figure 3 In the example case shown in (d), the quadrilateral formed by the four corner points of the parking space is identified as a concave quadrilateral, and when the four internal angles are all less than 180 degrees, the quadrilateral formed by the four corner points of the parking space is identified as a convex quadrilateral.

[0060] iv. The geographic area of ​​the quadrilateral formed by the corner points of the identified parking space in the current frame detection image to which the identified parking space belongs falls within a predetermined geographic area range.

[0061] The quadrilateral formed by the corner points of the parking space defines the boundary of the parking space. The geographical area of ​​the quadrilateral can represent the size of the parking space in the real three-dimensional space. If the parking space is identified as being too large or too small, the parking space identification is incorrect. Therefore, the predetermined geographical area range can be set according to the size that the parking space should normally have. When the geographical area of ​​the identified parking space is found to exceed the predetermined geographical area range, it can be determined that the identification of the parking space has failed.

[0062] Step S232, for multiple identified parking spaces in the same frame detection image, determine whether each of the multiple identified parking spaces meets the parking space mutual verification condition based on the mutual relationship between the multiple identified parking spaces, and determine the identified parking spaces that do not meet the parking space mutual verification condition as verification failed parking spaces.

[0063] In one embodiment, the parking space mutual verification condition includes one or more combinations of the following i and ii:

[0064] i. The parking space ID of the identified parking space is not repeated with the parking space IDs of other identified parking spaces in the same frame detection image;

[0065] This condition can be used to perform a parking space ID check to ensure the uniqueness of the parking spaces in the same frame detection image while avoiding duplication, which can be used to check whether the parking space ID allocation is abnormal.

[0066] ii. The difference between the average geographical area of ​​the quadrilateral formed by the parking corner points of the identified parking space in the frame detection image and the average geographical area of ​​the quadrilateral formed by the parking corner points of other identified parking spaces in the same frame detection image in the frame detection image does not exceed the predetermined difference threshold.

[0067] For example, in the same frame detection image, there may be multiple recognized parking spaces, and the sizes of the multiple recognized parking spaces with similar sizes can be used as templates. If the size of another recognized parking space is larger than twice the template or smaller than half the template, it can be considered that the size of the recognized parking space is too different from the sizes of the other recognized parking spaces, and the recognized parking space can be determined as a parking space that failed verification. This conditional judgment can be performed only when there are more than three recognized parking spaces in the same frame detection image.

[0068] Step S233, for the identified parking spaces in the two different detection frames, determining whether the identified parking spaces in the two different detection frames meet the inter-frame verification condition, and determining the identified parking spaces that do not meet the inter-frame verification condition as verification-failed parking spaces;

[0069] In one embodiment, the inter-frame check condition includes one or more combinations of the following i and ii:

[0070] i. The timestamp of the current frame detection image to which the parking space belongs is not the same as the timestamp of any historical frame detection image before the current frame detection image;

[0071] The timestamp of the current observation data is self-checked. If the current timestamp is the same as the historical timestamp, it will be returned immediately to avoid repeated operations on the same observation data.

[0072] ii. The parking space semantic information of the identified parking space in the current frame detection map to which it belongs is consistent with the parking space semantic information of the identified parking space that is the same parking space in the historical frame detection map before the current frame detection map.

[0073] For parking spaces with the same ID between frames, some inherent properties of the parking spaces, such as the parking space entrance edge, parking space depth, width, etc., should not change. This information can be used to filter invalid parking spaces. Step S234, the identified parking spaces that meet the parking space self-verification condition, the parking space mutual verification condition, and the inter-frame verification condition are determined as verified parking spaces.

[0074] In this embodiment, the parking space self-verification condition, the parking space mutual verification condition and the inter-frame verification condition are used to perform intra-frame verification and inter-frame verification on the parking space in each frame detection image, thereby improving the detection accuracy of the parking space and avoiding misdetection of the parking space.

[0075] Step S240, using the parking space tracking list, track the verified parking spaces in the continuous multi-frame detection map, so as to record the number of continuous visible frames identified in the continuous multi-frame detection map and the number of continuous lost frames not identified in the continuous multi-frame detection map for each verified parking space in the parking space tracking list, and delete the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches the first frame number threshold M.

[0076] Further, in one embodiment, when the parking space tracking list is used to track the verified parking spaces in the continuous multiple-frame detection images in step S240, the method further includes step S241 of recording the parking space status of each verified parking space in the parking space tracking list, see Figure 4 As shown in , the parking space status may include a new status (New), a predicted status (Predict), an updated status (Updated) and an invalid status (Invalid). Step S241 includes:

[0077] For each checked parking space determined from the continuous multi-frame detection images, execute steps S2411-S2414:

[0078] Step S2411, when the verified parking space is first identified in a certain frame detection image, the verified parking space is recorded in the parking space tracking list and the parking space status of the verified parking space is marked as a new status;

[0079] Step S2412, when the verified parking space is recognized again in any frame detection image after a certain frame detection image, the parking space status of the verified parking space is marked as an updated status;

[0080] Step S2413, when the checked parking space is not identified in any frame detection image after a certain frame detection image, marking the parking space state of the checked parking space as a predicted state;

[0081] Step S2414, when the number of consecutive lost frames that are not identified in the continuous multiple-frame detection images after a certain frame detection image of the verified parking space in the prediction state or the update state reaches the first frame number threshold M, the parking space state of the verified parking space is changed to an invalid state.

[0082] For example, the parking space status of the verified parking space can be divided into four states: new state, predicted state, updated state, and invalid state, and the parking space status of each verified parking space is recorded according to the rules of steps S2411-S2414. The parking space tracking list can record the parking space ID of the verified parking space and the parking space attributes such as parking space corner points of the latest frame detection image of the verified parking space. The parking space matching method of determining whether the verified parking spaces of the two frames of detection images are the same parking space by the intersection and union ratio between the verified parking spaces of the two frames of detection images described below can match the verified parking spaces in the current frame detection image with the verified parking spaces in the previous frame detection image, thereby tracking the number of visible frames and the number of lost frames of each verified parking space, and counting the number of continuous visible frames and the number of continuous lost frames of each verified parking space.

[0083] The parking space tracking list tracks and records the parking space ID of each verified parking space, and ensures the uniqueness of the parking space ID. Available parking space IDs are used cyclically. If the status of a verified parking space in the parking space tracking list becomes Invalid, the parking space ID occupied by the verified parking space will be removed from the parking space tracking list and become an idle parking space ID. When a new verified parking space appears later, an idle parking space ID will be allocated to the new verified parking space according to the size of the parking space ID. In the parking space tracking list, you can also set the maximum number of parking space IDs that can be maintained.

[0084] For example, assuming that five verified parking spaces are identified in the first frame detection image, and all five verified parking spaces appear for the first time, the five verified parking spaces are recorded as parking space IDs 1, 2, 3, 4, and 5 and added to the parking space tracking list, and the parking status of the five verified parking spaces are all marked as New. When it comes to the second frame detection image, five verified parking spaces are identified in the second frame detection image, then the parking space status of the five verified parking spaces in the parking space tracking list are first marked as Predict, and then the five verified parking spaces in the second frame detection image are matched with the five verified parking spaces recorded in the parking space tracking list (the five verified parking spaces in the first frame detection image). For example, if the first four verified parking spaces in the second frame detection image are successfully matched with the four verified parking spaces in the parking space tracking list with parking space IDs 1, 2, 3, and 4, the four verified parking spaces in the parking space tracking list with parking space IDs 1, 2, 3, and 4 are updated to the corresponding first four verified parking spaces in the second frame detection image, and the parking space status of the four verified parking spaces in the parking space tracking list with parking space IDs 1, 2, 3, and 4 are marked as Updated, and the fifth verified parking space in the second frame detection image is If the parking space matching fails (considered to be the first occurrence), it will be added to the parking space tracking list and assigned a new parking space ID number 6, and its parking space status will be marked as New, that is, the parking space status of the verified parking spaces with parking space IDs 1, 2, 3, and 4 in the parking space tracking list is Updated, the parking space status of the verified parking space with parking space ID 5 is Predict, and the parking space status of the verified parking space with parking space ID 6 is New... When any verified parking space is visible in two consecutive frames (N=2) of the detection image, the parking space semantic information of the verified parking space is determined and output in each subsequent frame until the verified parking space is deleted from the parking space tracking list; and when any verified parking space is not visible (lost) for 3 consecutive frames (M=3), the parking space status of the verified parking space is set to Invalid (that is, the verified parking space is deleted from the parking space tracking list).

[0085] When the speed of the vehicle 10 is slightly faster, the position of the same parking space in different frame detection images may change greatly. In the above embodiment, when tracking the verified parking spaces in the continuous multi-frame detection images, it is necessary to match the verified parking spaces in the two previous and next frame detection images (the current frame detection image and the historical frame detection image recorded in the parking space tracking list) to confirm whether a verified parking space in the latter frame detection image is the same as a verified parking space in the previous frame detection image. In one embodiment, when tracking the verified parking spaces in the continuous multi-frame detection images using the parking space tracking list, step S242 is also included, which uses the intersection-and-union ratio between the verified parking spaces in the two frame detection images to determine whether the verified parking spaces in the two frame detection images are the same parking spaces.

[0086] Specifically, in one embodiment, step S242 may include steps S2421-S2423:

[0087] Step S2421, calculating the intersection over union (IOU) between each checked parking space in the previous detection frame of the two detection frames and each checked parking space in the next detection frame of the two detection frames;

[0088] In one embodiment, step S2421 includes steps S24211-S24212:

[0089] Step S24211, superimposing two frames of detection images to obtain a superimposed detection image;

[0090] See also Figure 5 As shown in Figure 5 The previous frame detection image shown in (a) is Figure 5 The next frame detection image shown in (b) is superimposed to obtain Figure 5 The superimposed detection image shown in (c).

[0091] Step S24212: for any first checked parking space A in the previous detection frame of the two detection frames and any second checked parking space B in the next detection frame of the two detection frames, perform the following steps ad to calculate the intersection-and-union ratio between the first checked parking space A and the second checked parking space B:

[0092] a. Determine the circumscribed rectangular frame of the parking corner points of the first verified parking space A and the second verified parking space B in the superimposed detection image, using the maximum and minimum values ​​of the parking corner points of the first verified parking space A and the second verified parking space B in the vertical direction as the upper and lower boundaries of the circumscribed rectangular frame, and using the maximum and minimum values ​​of the parking corner points of the first verified parking space A and the second verified parking space B in the horizontal direction as the left and right boundaries of the circumscribed rectangular frame.

[0093] For example, the first checked parking space A includes four parking space corner points, and the second checked parking space B includes four parking space corner points. Then, among the eight corner points of the two parking spaces, search for the maximum and minimum values ​​of the vertical coordinates in the superimposed detection image, as well as the maximum and minimum values ​​of the horizontal coordinates; form an upper boundary at the maximum value in the vertical direction, a lower boundary at the minimum value in the vertical direction, a right boundary at the maximum value in the horizontal direction, and a left boundary at the minimum value in the horizontal direction, so that the four boundaries enclose the two parking spaces to generate the circumscribed rectangular frame. The generated circumscribed rectangular frame can be found in Figure 5 As shown in (c), Figure 5 The outermost black frame in (c) represents the circumscribed rectangular frame of the first verified parking space A and the second verified parking space B.

[0094] b. when the length of at least one side of the circumscribed rectangular frame is greater than a predetermined side length threshold, determining that the intersection-and-joint ratio between the first checked parking space and the second checked parking space is zero;

[0095] For example, the predetermined side length threshold may be the sum of the longest side of the first checked parking space and the longest side of the second checked parking space. When any side of the circumscribed rectangular boxes of two parking spaces exceeds the predetermined side length threshold, it can be considered that the two parking spaces do not overlap or have very little overlap, and their intersection-and-union ratio can be directly returned as zero, eliminating the need for subsequent calculations.

[0096] c. When the lengths of all sides of the circumscribed rectangular frame are less than or equal to a predetermined side length threshold, a grid map is generated within the circumscribed rectangular frame as a boundary, and the number of grids occupied by the first verified parking space in the grid map and the number of grids occupied by the second verified parking space in the grid map are counted respectively;

[0097] When determining the number of occupied grids, when the covered area of ​​a certain grid is greater than a certain area threshold, the grid is considered to be occupied. The area threshold may be, for example, the area of ​​2 / 3 of the grid.

[0098] d. Calculate the intersection-and-union ratio between the first verified parking space and the second verified parking space based on the number of grids occupied by the first verified parking space in the grid map and the number of grids occupied by the second verified parking space in the grid map.

[0099] For example, the intersection-and-joint ratio between the first checked parking space A and the second checked parking space B may be calculated by the following equation:

[0100]

[0101] Among them, A∪B represents the union of the first verified parking space A and the second verified parking space B.

[0102] Step S2422, determining a checked parking space in the subsequent frame detection image whose intersection-and-joint ratio with any checked parking space in the previous frame detection image is greater than or equal to a predetermined intersection-and-joint ratio threshold as the same parking space as the any checked parking space in the previous frame detection image;

[0103] Step S2423, the verified parking spaces in the subsequent frame detection image whose intersection-and-joint ratios with all verified parking spaces in the previous frame detection image are all less than a predetermined intersection-and-joint ratio threshold are determined as different parking spaces from all verified parking spaces in the previous frame detection image.

[0104] For example, the predetermined intersection-and-join ratio threshold may be 60%. Then, when the intersection-and-join ratio is greater than or equal to 60%, it can be determined that the first checked parking space and the second checked parking space are the same parking space; when the intersection-and-join ratio is less than 60%, it can be determined that the first checked parking space and the second checked parking space are not the same parking space.

[0105] Step S250 , for each verified parking space in the parking space tracking list whose number of continuous visible frames reaches a second frame number threshold N, determine and output parking space semantic information of the verified parking space based on the parking space corner points of the verified parking space.

[0106] In one embodiment, the parking space semantic information may include one or more of the parking space corner point position, parking space corner point order, main road direction, parking space entrance edge, parking space depth, parking space width, parking space orientation, parking space direction type, and parking space available area.

[0107] In one embodiment, when the parking space semantic information includes the parking space corner point position, the method further includes:

[0108] Based on the parking corner point position of the verified parking space in the previous frame detection image, a Kalman filter is used to perform smoothing processing on the parking corner point position of the verified parking space in the current frame detection image to determine the parking corner point position after smoothing processing as the parking corner point position of the verified parking space in the current frame detection image and output it.

[0109] In this embodiment, a Kalman filter is used to smooth the position (coordinates) of the corner points of the parking spaces. The parking spaces added to the parking space tracking list will be used to update the parameters of the Kalman filter. The parking corner point position information output by each current frame detection map will be predicted and output using the parameters of the Kalman filter of the historical frame detection map. By adjusting the parameter ratio of the predicted value and the observed value, the jitter of the corner point position after passing through the Kalman filter can be greatly reduced, ensuring that the output of the parking corner point position is smoother.

[0110] The Kalman filter performs the following smoothing process of steps ac on the parking space corner point coordinates of each current frame detection image to obtain the smoothed parking space corner point coordinates:

[0111] a. Use the posterior estimated value of the parking space corner point coordinates of the previous frame detection image to predict the predicted value of the corresponding parking space corner point coordinates of the current frame detection image as the prior estimated value of the current frame detection image, and use the posterior error value of the parking space corner point coordinates of the previous frame detection image to predict the predicted error value of the corresponding parking space corner point coordinates of the current frame detection image as the prior error value of the current frame detection image;

[0112] b. Calculate the Kalman gain based on the prior error value of the current frame detection map;

[0113] c. Based on the Kalman gain calculated in step b, the observed value of the parking space corner point coordinates of the current frame detection image, and the prior estimate of the current frame detection image, calculate the posterior estimate of the current frame detection image as the parking space corner point coordinates after smoothing of the current frame detection image.

[0114] In one embodiment, when the parking space semantic information includes the order of parking space corner points, determining and outputting the parking space semantic information of the verified parking space based on the parking space corner points of the verified parking space in step S250 may include: step S251, determining and outputting the order of parking space corner points of the verified parking space based on the parking space corner points of the verified parking space. Among them, the order of parking space corner points of the four corner points of each verified parking space may be determined by using steps ae in the aforementioned method of determining whether the quadrilateral formed by the four corner points of the identified parking space is a convex quadrilateral. That is, for each verified parking space for which the determination of whether it is a convex quadrilateral has been performed in S230, the order of parking space corner points of its four corner points is determined, and the order of parking space corner points of the determined four corner points may be directly obtained in S250.

[0115] In the aforementioned implementation, when the parking space semantic information includes the order of parking space corner points, the order of parking space corner points of the verified parking space in each frame detection image can be determined, however, the correspondence between the order of parking space corner points of the same parking space in different frame detection images is unclear. In one embodiment, when the parking space semantic information includes the order of parking space corner points, the method may further include: matching the order of parking space corner points of the verified parking space in the current frame detection image with the order of parking space corner points of the corresponding verified parking space in the previous frame detection image, so that the order of parking space corner points of the current frame detection image is consistent with the order of parking space corner points of the previous frame detection image.

[0116] After each verified parking space in the latter detection image of the two detection images is successfully matched with a verified parking space in the former detection image of the two detection images by the parking space matching method, it is also necessary to make the parking space corner point sequence of each verified parking space in the latter detection image consistent with the parking space corner point sequence of the corresponding same verified parking space in the former detection image. In this embodiment, by matching the parking space corner point sequence between frames, it can be ensured that the parking space corner point sequence of the same parking space between the two detection images is consistent.

[0117] For example, taking each parking space including four corner points as an example, when it is determined that the first checked parking space A in the previous frame detection image and the second checked parking space B in the next frame detection image are the same parking space, the parking space corner point sequence matching method of steps ac below can be used to make the parking space corner point sequence of the second checked parking space B consistent with the parking space corner point sequence of the first checked parking space A:

[0118] a. Calculate the Euclidean distance between each pair of the four parking corner points Corner1-4 of the first checked parking space A and the four parking corner points Corner1'-4' of the second checked parking space B, and obtain a total of sixteen Euclidean distance values ​​between sixteen pairs of parking corner points as shown below:

[0119] (Corner 1 of parking space A - Corner 1' of parking space B), (Corner 2 of parking space A - Corner 1' of parking space B),

[0120] (Corner1 of parking space A - Corner2' of parking space B), (Corner2 of parking space A - Corner2' of parking space B),

[0121] (Corner 1 of parking space A - Corner 3' of parking space B), (Corner 2 of parking space A - Corner 3' of parking space B),

[0122] (Corner 1 of parking space A - Corner 4' of parking space B), (Corner 2 of parking space A - Corner 4' of parking space B),

[0123] (Corner 3 of parking space A - Corner 1' of parking space B), (Corner 4 of parking space A - Corner 1' of parking space B),

[0124] (Corner 3 of parking space A - Corner 2' of parking space B), (Corner 4 of parking space A - Corner 2' of parking space B),

[0125] (Corner 3 of parking space A - Corner 3' of parking space B), (Corner 4 of parking space A - Corner 3' of parking space B),

[0126] (Corner 3 of parking space A - Corner 4' of parking space B), (Corner 4 of parking space A - Corner 4' of parking space B).

[0127] b. Use the Hungarian matching algorithm to match the corresponding parking space corner points;

[0128] By using the Hungarian matching algorithm, it can be determined which parking space corner point of each of the four parking space corner points corner1'-4' of the second verified parking space B is the same as one of the four parking space corner points corner1-4 of the first verified parking space A. The Hungarian matching algorithm is a prior art and is therefore not described in detail in this application.

[0129] c. According to the matching result, the parking corner points of the second checked parking space B are arranged in the order of the parking corner points of the first checked parking space A.

[0130] For example, if the Hungarian matching algorithm determines that the parking corner points corner1', corner2', corner3', and corner4' of the second verified parking space B are the same parking corner points corner4, corner1, corner2, and corner3 of the first verified parking space A, respectively, then the parking corner points corner1', corner2', corner3', and corner4' of the second verified parking space B will be replaced by parking corner points corner4, corner1, corner2, and corner3, respectively, so that the order of the parking corner points of the second verified parking space B remains consistent with the order of the parking corner points of the first verified parking space A.

[0131] Further, in one embodiment, when the parking space semantic information includes the order of parking space corner points, determining and outputting the parking space semantic information of the verified parking space based on the verified parking space corner points in step S250 also includes: configuring the order of parking space corner points according to user input.

[0132] In this embodiment, the order of parking space corner points can be configured according to user needs. For example, after the parking space entrance edge is determined, the parking space corner points at both ends of the parking space entrance edge can be marked as the first corner point and the second corner point respectively, consisting of 0-1 corner points. The order of parking space corner points on the left side of the vehicle is clockwise, and the order of parking space corner points on the right side of the vehicle is counterclockwise.

[0133] In one embodiment, when the parking space semantic information includes the main road direction, determining the parking space semantic information of the verified parking space based on the parking space corner point of the verified parking space in step S250 may include:

[0134] Step S252, see Figure 6 As shown in , for any current frame detection image, the following steps ad are performed to determine the main road direction in the current frame detection image:

[0135] a. Identify one or more pairs of adjacent parking spaces in the current frame detection image;

[0136] For example, the Euclidean distance between the corner points of each two checked parking spaces in the current frame detection image can be calculated. For example, for any two checked parking spaces B1 and B2 in the current frame detection image, each checked parking space has four corner points, and the four corner points of the checked parking space B1 and the four corner points of the checked parking space B2 are paired, and sixteen Euclidean distance values ​​between the sixteen pairs of corner points can be calculated. Assuming that the coordinates of the two corner points in each pair of corner points are (x1, y1) and (x2, y2), the Euclidean distance d between each pair of corner points can be calculated by the following formula:

[0137]

[0138] Then, the calculated sixteen Euclidean distance values ​​can be arranged in descending order according to the corner point distance;

[0139] It is determined whether the two smallest Euclidean distances are less than the distance threshold. If any one of the two smallest Euclidean distances is greater than the distance threshold, the two currently checked parking spaces B1 and B2 are non-adjacent parking spaces; if both of the two smallest Euclidean distances are less than the distance threshold, the two currently checked parking spaces B1 and B2 are adjacent parking spaces. For example, the distance threshold may be 20 cm.

[0140] In this way, it can be determined whether every two checked parking spaces in the current frame detection image are adjacent parking spaces, thereby determining one or more pairs of adjacent parking spaces in the current frame detection image.

[0141] b. determining, in each pair of adjacent parking spaces, a centroid direction vector pointing from the centroid of the first parking space in the pair of adjacent parking spaces to the centroid of the second parking space in the pair of adjacent parking spaces, thereby obtaining one or more centroid direction vectors;

[0142] The centroid direction vector between the first parking space and the second parking space can be found in Figure 7 When determining the centroid direction vector in this step, the adjacent edges between the first parking space and the second parking space can also be recorded for use when needed.

[0143] c. classifying one or more centroid direction vectors to determine one or more centroid direction classes;

[0144] In this step, each centroid direction vector can be classified one by one. For example, see Figure 6As shown in , a list of centroid direction vectors can be created, and then each of the one or more centroid direction vectors is matched with the centroid direction vectors in the list one by one. If the parallelism between the current centroid direction vector and a centroid direction vector in the list is greater than or equal to the set parallelism threshold (the angle is less than or equal to the set angle threshold), then the current centroid direction vector can be considered to belong to the centroid direction class represented by the centroid direction vector in the list, that is, the number of votes for the centroid direction vector in the list is increased by one. When the current centroid direction vector cannot have a parallelism greater than or equal to the set parallelism threshold with the centroid direction vector in the list, that is, the parallelism between the current centroid direction vector and all the existing centroid direction vectors in the list is less than the set parallelism threshold (the angles are all greater than the set angle threshold) or there is no centroid direction vector in the list (when matching the first centroid direction vector), then the current centroid direction vector is added to the list as a new centroid direction vector until all centroid direction vectors are matched. For example, the set angle threshold can be 15°. The number of votes for each centroid direction vector in the list indicates the centroid direction vector of the centroid direction class represented by each centroid direction vector.

[0145] d. Determine the direction of the main road based on the centroid direction vector corresponding to the centroid direction class with the largest total number of centroid direction vectors.

[0146] For example, after completing the matching of all centroid direction vectors, the centroid direction vectors in the list can be sorted by votes to determine the centroid direction vector with the highest number of votes. When there is only one centroid direction vector with the highest number of votes, the centroid direction vector with the highest number of votes is output as the road direction vector, and the direction indicated by the road direction vector is the main road direction. When there is a tie among multiple centroid direction vectors with the highest number of votes, the centroid direction vector with the highest parallelism to the direction of travel of the vehicle (i.e., the smallest angle with the direction of travel of the vehicle) among the multiple centroid direction vectors with the highest number of votes is output as the road direction vector, and the direction indicated by the road direction vector is the main road direction.

[0147] In one embodiment, after determining the direction of the main road, when the parking space semantic information includes the parking space entrance edge, determining the parking space semantic information of the verified parking space based on the parking space corner point of the verified parking space in step S250 includes:

[0148] Step S253a, connecting the four parking space corner points of the verified parking space in sequence according to the order of the determined parking space corner points to determine the four sides of the verified parking space.

[0149] Step S253b, based on the direction of the main road, select the two edges with the highest parallelism to the main road direction from the determined four edges as the two candidate edges for the parking space entrance edge, and then select the edge with the shortest Euclidean distance to the geometric centroid of the vehicle among the two candidate edges as the parking space entrance edge.

[0150] like Figure 8 As shown in , the edge represented by the dotted line and the edge represented by the short dash line with the highest parallelism to the main road direction are two candidate edges, and the edge represented by the dotted line that is closer to the geometric centroid of the vehicle is selected as the parking space entrance edge. The edge represented by the short dash line opposite to the parking space entrance edge is the parking space bottom edge.

[0151] In one embodiment, see Figure 8 As shown, after the parking space entrance edge is determined, when the parking space semantic information includes the parking space depth, the parking space semantic information of the verified parking space is determined based on the parking space corner points of the verified parking space in step S250, including: taking the line segment distance between two parallel lines formed by the parking space entrance edge of the verified parking space and the parking space bottom edge opposite to the parking space entrance edge as the parking space depth of the verified parking space.

[0152] In one embodiment, see Figure 8 As shown, after the parking space entrance edge is determined, when the parking space semantic information includes the parking space width, the parking space semantic information of the verified parking space is determined based on the parking space corner point of the verified parking space in step S250, including: taking the line segment distance between two parallel lines formed by two adjacent side edges of the parking space entrance edge of the verified parking space as the parking space width of the verified parking space.

[0153] In one embodiment, see Figure 8 As shown, after the parking space entrance edge is determined, when the parking space semantic information includes the parking space orientation, the parking space semantic information of the verified parking space is determined based on the parking space corner points of the verified parking space in step S250, including: taking the direction perpendicular to the parking space entrance edge and facing outside the verified parking space (the direction from the bottom edge to the parking space entrance edge) as the parking space orientation of the verified parking space.

[0154] In one embodiment, see Fig. 9 As shown, after the parking space entrance edge is determined, when the parking space semantic information includes the parking space direction type, determining the parking space semantic information of the verified parking space based on the parking space corner point of the verified parking space in step S250 may include steps S254a-S254b:

[0155] Step S254a, based on the four inner angles of the quadrilateral formed by the four parking corner points of the checked parking space, determine whether the checked parking space is an oblique parking space; wherein, when the four inner angles are all within the predetermined inner angle range, it is determined that the checked parking space is not an oblique parking space, and when at least one of the four inner angles exceeds the predetermined inner angle range, it is determined that the checked parking space is an oblique parking space. The predetermined inner angle threshold may be, for example, greater than or equal to 75° and less than or equal to 105°.

[0156] Step S254b, when it is determined in step S254a that the verified parking space is not a slanted parking space, when the parking space entrance edge of the verified parking space is one of the two shorter sides of the quadrilateral, the verified parking space is determined to be a vertical parking space, and when the parking space entrance edge of the verified parking space is one of the two longer sides of the quadrilateral, the verified parking space is determined to be a parallel parking space.

[0157] The parking space direction types can include oblique parking spaces, vertical parking spaces, and parallel parking spaces. Among them, oblique parking spaces refer to parking spaces with a certain inclination angle between the parking space direction and the main road direction, for example Fig. 9 A reference example of an oblique parking space is given in (a) of FIG. A vertical parking space refers to a parking space whose direction is substantially perpendicular to the direction of the main road, for example Fig. 9 A reference example of a vertical parking space is given in (b) of FIG. A parallel parking space refers to a parking space that is substantially parallel to the direction of the main road, for example Fig. 9 A reference example of parallel parking spaces is given in (c).

[0158] In some cases, there may be obstacles such as wheel blocking rods or pedestrians in the parking space. When the vehicle needs to park, it may be necessary to output the parking area of ​​the parking space to ensure that the parking process of the vehicle is safer. In one embodiment, when the parking space semantic information includes the parking area of ​​the parking space, determining the parking space semantic information of the verified parking space based on the parking space corner points of the verified parking space in step S250 may include steps S255a-S255d:

[0159] Step S255a, determining the total parking area of ​​the verified parking space based on the parking space corner points of the verified parking space.

[0160] The total parking area is the parking area of ​​the verified parking space when there is no obstacle. The total parking area may be, for example, a quadrilateral area formed by four parking corner points of the verified parking space in sequence.

[0161] Step S255b, detecting whether there is an obstacle on the checked parking space.

[0162] In one embodiment, the obstacles may include fixed obstacles and / or non-fixed obstacles.

[0163] Among them, obstacles fixed on the verified parking space, such as limit blocks / wheel stoppers, etc., are fixed obstacles. Obstacles not fixed on the verified parking space, such as pedestrians, animals, garbage, vehicles and other objects that interfere with parking, are non-fixed obstacles.

[0164] Step S255c: when there is no obstacle on the verified parking space, the total parking area is used as the parking area of ​​the verified parking space.

[0165] Step S255d: when there is an obstacle on the verified parking space, the non-parking area occupied by the obstacle is subtracted from the total parking area to obtain the parking area of ​​the verified parking space.

[0166] Where obstacles may include fixed obstacles and / or non-fixed obstacles:

[0167] When a fixed obstacle is detected on the verified parking space, the non-parking area of ​​the fixed obstacle can be calculated according to the predetermined calculation rules corresponding to the fixed obstacle; for example, for a wheel stopper, the space from the straight line parallel to the bottom edge where the wheel stopper is located to the bottom edge of the total parking area can be treated as a non-parking area and removed from the total parking area.

[0168] When a non-fixed obstacle is detected on the verified parking space, the intersection edges between the non-fixed obstacle and the boundary of the polygon defined by the parking corner points of the verified parking space and the inner corner points of the corner points of the non-fixed obstacle are determined, and the intersection edges and the inner corner points are used to query and determine the target calculation rules corresponding to the intersection edges and the inner corner points in the no-parking area calculation table, and the no-parking area of ​​the non-fixed obstacle is calculated using the target calculation rules.

[0169] It can be understood that the non-parking area that needs to be subtracted from the total parking area is the total area obtained by superimposing the non-parking areas of all detected fixed obstacles and / or non-fixed obstacles (ie, the union of the non-parking areas of all obstacles).

[0170] The non-fixed obstacle may be in various postures, and thus the relative position relationship between the non-fixed obstacle and the checked parking space may be variable. For example, a detection frame (two-dimensional bounding box) defining the area where the non-fixed obstacle is located may be detected by image recognition, and the relative position relationship between the non-fixed obstacle and the checked parking space may be determined by using the detection frame to represent the non-fixed obstacle.

[0171] Thus, it is possible to determine which of the four sides of the quadrilateral defined by the four parking corner points of the checked parking space intersect with the detection box, and which of the four corner points of the detection box are inner corner points within the quadrilateral defined by the four parking corner points of the checked parking space.

[0172] Fig.10 , which shows possible example intersection postures of the detection frame representing the non-fixed obstacle and the quadrilateral defined by the four corner points of the verified parking space. It can be understood that Fig.10 The intersection postures shown in are only examples and are not exhaustive. The white filled box represents the verified parking space, the edge represented by the dotted line represents the parking space entrance edge of the verified parking space, and the gray filled box represents the detection box where the non-fixed obstacle is located. For example, the calculation table of the non-parking area can record the following calculation rules:

[0173] (1) When any of the following conditions ac is met, the non-fixed obstacle non-parking area is calculated as equal to the total parking area (i.e., the parking area is considered to be zero. In this case, the parking area can be ignored and the verified parking space can be directly determined as non-parking area).

[0174] a. The number of intersecting edges is greater than or equal to three;

[0175] b. There is a parking space entrance edge among the intersecting edges;

[0176] c. The distance between any of the inner corner points and any side of the quadrilateral exceeds the side distance threshold (for example, 1 / 5 of the parking space width) or the distance between any of the inner corner points and the bottom of the quadrilateral exceeds the bottom distance threshold (for example, 1 / 5 of the parking space depth);

[0177] (2) When none of the above conditions ac are satisfied, the non-parking area is calculated according to a corresponding calculation rule in the following calculation rules dk:

[0178] d. When the number of intersecting edges = 0 and the number of interior corner points = 4, calculate the distance between each interior corner point and the base of the quadrilateral, and determine the area from the straight line parallel to the base where the interior corner point with the largest distance is located to the base as the non-parking area;

[0179] e. When the number of intersecting edges = 1, the number of interior corners = 0, and the non-parking area is counted as zero;

[0180] f. When the number of intersecting edges = 1 and the number of inner corner points = 1, the area in the total parking area from the straight line parallel to the intersecting edge where the inner corner point is located to the intersecting edge is determined as a non-parking area;

[0181] g. When the number of intersecting edges = 1 and the number of inner corner points = 2, calculate the distance between each inner corner point and the intersecting edge, and determine the area from the straight line parallel to the intersecting edge where the inner corner point with the largest distance is located to the intersecting edge as the non-parking area;

[0182] h. When the number of intersecting edges = 1 and the number of inner corner points = 3, calculate the distance between each inner corner point and the intersecting edge, and determine the area from the straight line parallel to the intersecting edge where the inner corner point with the largest distance is located to the intersecting edge as the non-parking area;

[0183] i. When the number of intersecting edges = 2 and the number of inner corner points = 0, determine the intersection point of the detection frame where the non-fixed obstacle is located and the side of the quadrilateral, and determine the area from the straight line parallel to the bottom edge where the intersection point is located to the bottom edge in the total parking area as the non-parking area;

[0184] j. When the number of intersecting edges = 2 and the number of inner corner points = 1, determine the intersection points of the detection frame where the non-fixed obstacle is located and the side of the quadrilateral, calculate the distances from the inner corner points and each intersection point to the bottom edge, and determine the area from the straight line parallel to the bottom edge where the point with the largest distance from the bottom edge among the intersection points and the inner corner points in the total parking area is located to the bottom edge as the non-parking area;

[0185] k. When the number of intersecting edges = 2 and the number of inner corner points = 2, determine the intersection points of the detection frame where the non-fixed obstacle is located with the side of the quadrilateral, calculate the distances from each inner corner point and each intersection point to the bottom edge respectively, and determine the area in the total parking area from the straight line parallel to the bottom edge where the point with the largest distance from the bottom edge among the intersection points and the inner corner points is located to the bottom edge as the non-parking area.

[0186] Further, in one embodiment, after obtaining the parking area of ​​the verified parking space in step S254d, step S250 may also include: step S254e, when the parking area of ​​the verified parking space is greater than or equal to the parking area threshold, outputting the parking area of ​​the verified parking space; when the parking area of ​​the verified parking space is less than the parking area threshold, determining that the verified parking space is not available for parking, and outputting a prompt message that the verified parking space is not available for parking.

[0187] In the above parking space detection method, the parking space corner points of each identified parking space are identified, and the parking space verification is performed to screen out the verified parking spaces, so that the parking spaces that do not meet the requirements can be deleted in advance; the verified parking spaces are tracked and recorded using the parking space tracking list, and the verified parking spaces whose continuous lost frames reach a first frame number threshold are deleted from the parking space tracking list, and the parking space semantic information is output for each verified parking space whose continuous visible frames reach a second frame number threshold, so that the verified parking spaces in the detection image of the required frame can be efficiently tracked and the parking space semantic information can be output, without outputting the parking space semantic information of the verified parking spaces whose continuous lost frames reach a first frame number threshold, thereby saving computing resources and improving computing efficiency; because the parking space semantic information and parking space verification are based on more refined parking space corner points, rather than only on rough parking space areas, the parking space verification and the analysis of parking space semantic information can be made more accurate and detailed.

[0188] Fig.11 (a) shows an example of a bird's-eye view of a parking space detected in the prior art. It can be seen that the parking space detected in the prior art is only an approximate range and does not contain accurate parking space corner point information. Fig.11 (b) shows an example of a bird's-eye view of a parking space detected using the parking space detection method of the present application. Compared with the prior art, the method of the present application can detect the position of the parking space more accurately, and the corner points of each parking space border displayed can accurately correspond to the actual parking space corner points.

[0189] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0190] In one embodiment, Fig.12 As shown, a parking space detection device 1200 is provided, comprising: a detection map acquisition module 1210, a parking space recognition module 1220, a parking space verification module 1230, a parking space tracking module 1240 and a parking space semantic output module 1250, wherein:

[0191] The detection image acquisition module 1210 is used to acquire a continuous multi-frame detection image of the area where the vehicle is located;

[0192] The parking space recognition module 1220 is used to recognize the parking space in each detection frame of the continuous multiple detection frames and the parking space corner points of the parking space;

[0193] A parking space verification module 1230, configured to perform parking space verification based on parking space corner points to determine a verified parking space from the identified parking spaces;

[0194] The parking space tracking module 1240 is used to track the verified parking spaces in the continuous multi-frame detection image by using the parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection image and the number of continuous lost frames not identified in the continuous multi-frame detection image for each verified parking space in the parking space tracking list, and delete the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold;

[0195] The parking space semantic output module 1250 is used to determine and output parking space semantic information of each verified parking space in the parking space tracking list for which the number of continuous visible frames reaches a second frame number threshold based on the parking space corner points of the verified parking space.

[0196] For the specific definition of the parking space detection device 1200, please refer to the definition of the parking space detection method above, which will not be repeated here. The various modules in the above-mentioned parking space detection device 1200 can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the parking space detection device in the form of hardware, or can be stored in the memory in the parking space detection device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0197] In one embodiment, a parking space detection device is provided, and its internal structure diagram can be as follows: Fig.13 As shown. The parking space detection device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the parking space detection device is used to provide computing and control capabilities. The memory of the parking space detection device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the parking space detection device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a parking space detection method is implemented. The display screen of the parking space detection device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the parking space detection device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the shell of the parking space detection device, or an external keyboard, touchpad or mouse, etc.

[0198] Those skilled in the art will understand that Fig.13 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the parking space detection device to which the scheme of the present application is applied. The specific parking space detection device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0199] In one embodiment, a parking space detection device is provided. The parking space detection device is installed in a vehicle and is communicatively connected to an environmental sensor installed on the vehicle. The parking space detection device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0200] Obtain continuous multi-frame detection images of the area where the vehicle is located;

[0201] Identify the identified parking spaces and the parking corner points of the identified parking spaces in each of the continuous multiple-frame detection images;

[0202] Based on the parking space corner points, performing parking space verification to determine a verified parking space from the identified parking spaces;

[0203] Track the verified parking spaces in the continuous multi-frame detection images by using the parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection images and the number of continuous lost frames not identified in the continuous multi-frame detection images for each verified parking space in the parking space tracking list, and delete the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold;

[0204] For each verified parking space in the parking space tracking list whose number of continuous visible frames reaches a second frame number threshold, based on the parking space corner points of the verified parking space, parking space semantic information of the verified parking space is determined and output.

[0205] In other embodiments, when the processor executes the computer program, it also implements the steps of the vehicle detection method in any of the above embodiments.

[0206] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0207] Obtain continuous multi-frame detection images of the area where the vehicle is located;

[0208] Identify the identified parking spaces and the parking corner points of the identified parking spaces in each of the continuous multiple-frame detection images;

[0209] Based on the parking space corner points, performing parking space verification to determine a verified parking space from the identified parking spaces;

[0210] Track the verified parking spaces in the continuous multi-frame detection images by using the parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection images and the number of continuous lost frames not identified in the continuous multi-frame detection images for each verified parking space in the parking space tracking list, and delete the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold;

[0211] For each verified parking space in the parking space tracking list whose number of continuous visible frames reaches a second frame number threshold, based on the parking space corner points of the verified parking space, parking space semantic information of the verified parking space is determined and output.

[0212] In other embodiments, when the computer program is executed by a processor, the steps of the vehicle detection method in any of the above embodiments are also implemented.

[0213] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0214] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0215] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A parking space detection method, the method comprising: Obtain continuous multi-frame detection images of the area where the vehicle is located; Identify a recognized parking space in each detection frame of the continuous multiple detection frames and a parking space corner point of the recognized parking space; Based on the parking space corner points, performing parking space verification to determine a verified parking space from the identified parking spaces; Tracking the verified parking spaces in the continuous multi-frame detection images by using the parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection images and the number of continuous lost frames not identified in the continuous multi-frame detection images of each verified parking space in the parking space tracking list, and deleting the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold; For each of the verified parking spaces in the parking space tracking list whose number of continuous visible frames reaches a second frame number threshold, the parking space semantic information of the verified parking space is determined and output based on the parking space corner points of the verified parking space.

2. The parking space detection method according to claim 1, characterized in that: The performing parking space verification based on the parking space corner point to determine a verified parking space from the identified parking space comprises: For each of the identified parking spaces, determining whether the parking space corner point of the identified parking space meets the parking space self-verification condition, and if not, determining that the identified parking space is a verification failed parking space; For a plurality of the identified parking spaces in the same frame detection image, judging whether each of the plurality of identified parking spaces satisfies a parking space mutual verification condition according to the mutual relationship between the plurality of identified parking spaces, and determining the identified parking spaces that do not satisfy the parking space mutual verification condition as verification-failed parking spaces; For the identified parking spaces in two different detection frames, determining whether the identified parking spaces in the two different detection frames meet an inter-frame verification condition, and determining the identified parking spaces that do not meet the inter-frame verification condition as verification-failed parking spaces; The identified parking space that meets the parking space self-verification condition, the parking space mutual verification condition and the inter-frame verification condition is determined as a verified parking space.

3. The parking space detection method according to claim 1, characterized in that: The parking space self-verification conditions include one or more of the following combinations: The number of parking space corner points of the identified parking space meets the predetermined number of parking space corner points; The corner points of the identified parking spaces are all within a predetermined area of ​​interest; The quadrilateral formed by the corner points of the identified parking space is a convex quadrilateral; The geographical area of ​​the quadrilateral formed by the parking space corner points of the identified parking space in the current frame detection image to which the identified parking space belongs falls within a predetermined geographical area range.

4. The parking space detection method according to claim 1, characterized in that: The parking space mutual verification conditions include one or more of the following combinations: The parking space ID of the identified parking space is not repeated with the parking space IDs of other identified parking spaces in the same frame detection image; The difference between the geographical area of ​​the quadrilateral formed by the parking corner points of the identified parking space in the frame detection image and the average geographical area of ​​the quadrilateral formed by the parking corner points of other identified parking spaces in the same frame detection image in the frame detection image does not exceed a predetermined difference threshold.

5. The parking space detection method according to claim 1, characterized in that: The inter-frame check condition includes one or more of the following combinations: The timestamp of the current frame detection image to which the identified parking space belongs is not the same as the timestamp of any historical frame detection image before the current frame detection image; The parking space semantic information of the identified parking space in the current frame detection image to which it belongs is consistent with the parking space semantic information of the identified parking space that is the same parking space as it in the historical frame detection image before the current frame detection image.

6. The parking space detection method according to claim 1, characterized in that: When the parking space tracking list is used to track the verified parking spaces in the continuous multiple-frame detection images, the method further includes: The parking space status of each of the verified parking spaces is recorded in the parking space tracking list, wherein the parking space status includes a new status, a predicted status, an updated status, and an invalid status, including: For each of the verified parking spaces determined from the continuous multi-frame detection images: When the verified parking space is first identified in a certain frame detection image, the verified parking space is recorded in a parking space tracking list and the parking space state of the verified parking space is marked as a new state; When the verified parking space is recognized again in any frame detection image after the certain frame detection image, marking the parking space status of the verified parking space as an updated status; When the checked parking space is not identified in any frame detection image after the certain frame detection image, marking the parking space state of the checked parking space as a predicted state; When the number of consecutive lost frames that are not identified in the continuous multiple-frame detection images after the certain frame detection image of the verified parking space in the prediction state or the update state reaches the first frame number threshold, the parking space state of the verified parking space is marked as an invalid state.

7. The parking space detection method according to claim 1, characterized in that: When the parking space tracking list is used to track the verified parking spaces in the continuous detection images, the intersection-and-union ratio between the verified parking spaces in the two detection images is used to determine whether the verified parking spaces in the two detection images are the same parking space.

8. The parking space detection method according to claim 7, characterized in that: The method of using the intersection-and-union ratio between the checked parking spaces of the two detection images to determine whether the checked parking spaces of the two detection images are the same includes: Calculate the intersection-and-union ratio between each checked parking space in the previous detection frame of the two detection frames and each checked parking space in the next detection frame of the two detection frames; Determine a checked parking space in the subsequent frame detection image whose intersection-and-joint ratio with any checked parking space in the previous frame detection image is greater than or equal to a predetermined intersection-and-joint ratio threshold as the same parking space as the any checked parking space in the previous frame detection image; The verified parking spaces in the subsequent frame detection image whose intersection-and-union ratios with all verified parking spaces in the previous frame detection image are all less than a predetermined intersection-and-union ratio threshold are determined to be different parking spaces from all verified parking spaces in the previous frame detection image.

9. The parking space detection method according to claim 8, characterized in that: The calculating of the intersection-and-combination ratio between each checked parking space in the previous detection frame of the two detection frames and each checked parking space in the next detection frame of the two detection frames comprises: Superimposing two frames of detection images to obtain a superimposed detection image; For any first checked parking space in the first detection frame of the two detection frames and any second checked parking space in the second detection frame of the two detection frames: Determine a circumscribed rectangular frame of the parking corner points of the first checked parking space and the second checked parking space in the superimposed detection image by taking the maximum and minimum values ​​of the parking corner points of the first checked parking space and the second checked parking space in the vertical direction in the superimposed detection image as the upper and lower boundaries of the circumscribed rectangular frame, and taking the maximum and minimum values ​​of the parking corner points of the first checked parking space and the second checked parking space in the horizontal direction in the superimposed detection image as the left and right boundaries of the circumscribed rectangular frame; When the length of at least one side of the circumscribed rectangular frame is greater than a predetermined side length threshold, determining that an intersection-over-combination ratio between the first checked parking space and the second checked parking space is zero; When the lengths of all sides of the circumscribed rectangular frame are less than or equal to a predetermined side length threshold, a grid map is generated with the circumscribed rectangular frame as a boundary, and the number of grids occupied by the first verified parking space in the grid map and the number of grids occupied by the second verified parking space in the grid map are counted respectively; Based on the number of grids occupied by the first verified parking space in the grid map and the number of grids occupied by the second verified parking space in the grid map, an intersection-and-union ratio between the first verified parking space and the second verified parking space is calculated.

10. The parking space detection method according to any one of claims 1 to 9, characterized in that: The parking space semantic information includes one or more of the parking space corner point position, parking space corner point order, main road direction, parking space entrance edge, parking space depth, parking space width, parking space orientation, parking space direction type, and parking space available area.

11. The parking space detection method according to claim 10, characterized in that: When the parking space semantic information includes the parking space corner point position, the method further includes: Based on the parking corner point position of the verified parking space in the previous frame detection image, a Kalman filter is used to perform smoothing processing on the parking corner point position of the verified parking space in the current frame detection image to determine the parking corner point position after smoothing processing as the parking corner point position of the verified parking space in the current frame detection image and output it.

12. The parking space detection method according to claim 10, characterized in that: When the parking space semantic information includes a sequence of parking space corner points, the method further includes: The order of parking corner points of the verified parking spaces in the current frame detection image is matched with the order of parking corner points of the verified parking spaces in the previous frame detection image, so that the order of parking corner points of the current frame detection image is consistent with the order of parking corner points of the previous frame detection image.

13. The parking space detection method according to claim 10, characterized in that: When the parking space semantic information includes the main road direction, determining the parking space semantic information of the verified parking space based on the parking space corner point of the verified parking space includes: For any current frame detection image: Identifying one or more pairs of adjacent parking spaces in the current frame detection image; Determine, in each pair of adjacent parking spaces, a centroid direction vector pointing from the centroid of a first parking space in the pair of adjacent parking spaces to the centroid of a second parking space in the pair of adjacent parking spaces, thereby obtaining one or more centroid direction vectors; classifying the one or more centroid direction vectors to determine one or more centroid direction classes; The main road direction is determined based on the centroid direction vector corresponding to the centroid direction class having the largest total number of centroid direction vectors.

14. The parking space detection method according to claim 10, characterized in that: When the parking space semantic information includes a parking area of ​​the parking space, determining the parking space semantic information of the verified parking space based on the parking space corner point of the verified parking space includes: Determining a total parking area of ​​the verified parking space based on the parking space corner point of the verified parking space; Detecting whether there is an obstacle on the verified parking space; When there is an obstacle on the verified parking space, the non-parkable area occupied by the obstacle is subtracted from the total parking area to obtain the parkingable area of ​​the verified parking space; When there is no obstacle on the verified parking space, the total parking area is used as the parkingable area of ​​the verified parking space.

15. The parking space detection method according to claim 14, characterized in that: The obstacles include fixed obstacles and / or non-fixed obstacles; When a fixed obstacle is detected on the verified parking space, the non-parking area of ​​the fixed obstacle is calculated according to a predetermined calculation rule corresponding to the fixed obstacle; When it is detected that there is a non-fixed obstacle on the verified parking space, the intersection edge between the non-fixed obstacle and the boundary of the polygon defined by the parking corner points of the verified parking space and the inner corner points of the corner points of the non-fixed obstacle are determined, and the target calculation rules corresponding to the intersection edge and the inner corner points are queried in a no-parking area calculation table using the intersection edge and the inner corner points, and the no-parking area of ​​the non-fixed obstacle is calculated using the target calculation rule.

16. A parking space detection device, characterized in that: The device comprises: A detection image acquisition module is used to obtain continuous multi-frame detection images of the area where the vehicle is located; A parking space recognition module, used to recognize a recognized parking space in each detection frame of the continuous multiple detection frames and a parking space corner point of the recognized parking space; A parking space verification module, configured to perform parking space verification based on the parking space corner points to determine a verified parking space from the identified parking spaces; a parking space tracking module, configured to track the verified parking spaces in the continuous multi-frame detection images by using a parking space tracking list, so as to record the number of continuous visible frames identified in the continuous multi-frame detection images and the number of continuous lost frames not identified in the continuous multi-frame detection images of each verified parking space in the parking space tracking list, and to delete the verified parking space from the parking space tracking list when the number of continuous lost frames of any verified parking space reaches a first frame number threshold; The parking space semantic output module is used to determine and output the parking space semantic information of each verified parking space in the parking space tracking list whose continuous visible frame number reaches a second frame number threshold based on the parking space corner point of the verified parking space.

17. A parking space detection device, installed in a vehicle, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the parking space detection method according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the parking space detection method according to any one of claims 1 to 15 are implemented.

Images