Parking space state detection method and device based on 3D perspective calculation

Abstract

The invention provides a parking space state detection method based on 3D perspective calculation. The method comprises the following steps: acquiring low-view-angle vacancy and occupation images; adding a pose key point detection branch based on the target detection network, inputting the marked vehicle sample image, and obtaining a target detection model; constructing a 3D cube in the low-view-angle vacancy image; acquiring a vehicle body detection frame, a license plate detection frame, a tire detection frame and a vehicle chassis from the low-view-angle occupation image by adopting the target detection model; obtaining a license plate width threshold, a license plate height threshold and a height ratio threshold; a current frame image is collected, and a 3D cube, a vehicle body detection frame, a license plate detection frame, a tire detection frame and a vehicle chassis of the current frame image are obtained; and judging the parking space state of the current frame image and outputting the parking space state. Compared with the prior art, the accuracy of parking space state detection can be effectively improved.

Description

Technical Field

[0001] This invention relates to image processing, smart parking, and parking space detection, and particularly to a method and apparatus for detecting parking space status based on 3D perspective calculation. Background Technology

[0002] With the popularization of smart parking lots, the detection of the positional relationship between vehicles and parking spaces (such as occupancy status and occupancy percentage) has become a core technical support for parking fee collection and parking space scheduling. Currently, the mainstream method in the industry relies on the intersection-union ratio (IOU) calculation of 2D images (referred to as the 2D IOU method): by acquiring parking space images through cameras, using object detection algorithms to identify the vehicle chassis area, and calculating the pixel overlap ratio (IOU) between the chassis area and the parking space marking area, the vehicle's occupancy status of the parking space is determined.

[0003] However, parking spaces exhibit severe perspective compression in images from low angles, leading to two major defects in the traditional 2D IOU method: pixel scale distortion: the 2D IOU method only calculates planar pixel overlap and cannot reflect the "pixel correspondence" of the compressed area; and spatial dimension loss: the 2D IOU method cannot distinguish the situation where "vehicle pixels are within the parking space image area but actually exceed the visual boundary of the parking space".

[0004] In summary, there is an urgent need to propose a parking space status detection method based on 3D perspective calculation to solve the above problems and address the issue of parking space status from a low perspective. Summary of the Invention

[0005] In view of this, the main objective of the present invention is to provide a parking space status detection method based on 3D perspective calculation, which can improve the accuracy of parking space status detection from a low perspective.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a parking space status detection method based on 3D perspective calculation is provided, the method comprising: The first step is to use low-angle cameras to collect images of the parking spaces when they are not occupied as low-angle empty space images, and to collect images of the parking spaces when they are occupied by vehicles as low-angle occupied space images. The second step involves adding a pose keypoint detection branch to the object detection network, inputting N2 labeled vehicle sample images, and obtaining the trained object detection model. The third step involves constructing a 3D cube on the low-view empty space image for each parking space area using a 3D cube construction method; and using an object detection model to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point in the low-view occupied space image; and using the triangular area formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis. The third step is to count the width and height of all license plate detection frames within the N1 parking space areas, and use the minimum width and minimum height of the license plate as the license plate width threshold and license plate height threshold, respectively. The fourth step is to calculate the maximum height of the tire detection frame for each parking space area, and then calculate the ratio of the maximum height of the tire detection frame to the second height of the 3D cube, which is recorded as the height ratio of the parking space area. The minimum value of the height ratio of the N1 parking space areas is used as the height ratio threshold. The sixth step involves using a low-angle camera to acquire the current frame image; using a 3D cube construction method to construct a 3D cube in the current frame image; and using an object detection model to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point of the current frame image; and using the triangular region formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis of the current frame image. The seventh step is to calculate the intersection-union ratio of the vehicle chassis and the bottom face of the 3D cube in the current frame image, obtain the license plate assistance ratio and the tire assistance ratio, and determine and output the parking space status of the current frame image.

[0007] Wherein, N1 represents the number of parking space areas collected, and N2 represents the number of labeled vehicle sample images.

[0008] The labeled vehicle sample image refers to a vehicle image with a labeled body detection frame, license plate detection frame, tire detection frame, and four tire key points. The four tire key points refer to the contact points between the outer sides of the four tires and the ground in the vehicle sample image, including: a first tire key point, a second tire key point, a third tire key point, and a fourth tire key point. When labeling the vehicle sample image, the tire key point closest to the camera is designated as the first tire key point, the tire key point diagonally opposite the first tire key point is designated as the third tire key point, and the remaining two tire key points are designated as the second and fourth tire key points, respectively.

[0009] The 3D cube construction method includes: The three corner point marking steps involve manually marking three corner points of the parking space area in the image: left rear corner point P1, right rear corner point P2, and right front corner point P3. The steps for obtaining the height of the 3D cube are as follows: Calculate the distances d1, d2, and d3 from the left rear corner point P1, right rear corner point P2, and right front corner point P3 to the upper boundary of the image, respectively; calculate the first height H1 = d1 × λ1, the second height H2 = d2 × λ2, and the third height H3 = d3 × λ3 of the 3D cube, respectively, where λ1, λ2, and λ3 represent the first coefficient, second coefficient, and third coefficient of the 3D cube height, respectively. The steps for constructing a 3D cube for an empty berth are as follows: The triangular region formed by the left rear corner point P1, right rear corner point P2, and right front corner point P3 is used as the bottom face C_bottom of the 3D cube; the left rear corner point P1, right rear corner point P2, and right front corner point P3 are extended upwards to a first height H1, a second height H2, and a third height H3, respectively, to obtain the first vertex Q1, the second vertex Q2, and the third vertex Q3 of the top face of the 3D cube; the quadrilateral region formed by the left rear corner point P1, right rear corner point P2, second vertex Q2, and first vertex Q1 is used as the left side face C_left_side of the 3D cube; the quadrilateral region formed by the right rear corner point P2, right front corner point P3, third vertex Q3, and second vertex Q2 is used as the right side face C_right_side of the 3D cube.

[0010] Furthermore, the seventh step includes: The intersection-over-union (IoU) calculation steps are as follows: For the current frame image, calculate the IoU between the vehicle chassis and the bottom face of the 3D cube, denoted as bottom_iou; calculate the IoU between the vehicle body detection box and the left side face of the 3D cube, denoted as left_side_iou; and calculate the IoU between the vehicle body detection box and the right side face of the 3D cube, denoted as right_side_iou. The license plate auxiliary ratio acquisition step is as follows: For the current frame image, obtain the horizontal coordinate of the center point of the vehicle's license plate detection box. If the horizontal coordinate of the center point of the license plate detection box is between the horizontal coordinates of the left rear corner point P1 and the right rear corner point P2, and simultaneously satisfies the conditions that the license plate width and license plate height of the vehicle's license plate detection box are greater than the license plate width threshold and the license plate height threshold, then set the license plate auxiliary ratio plate_R to 1; otherwise, set the license plate auxiliary ratio plate_R to 0. The tire assist ratio calculation steps are as follows: For the current frame image, the maximum height of the tire detection box is counted, and the ratio of the maximum height of the tire detection box to the second height of the 3D cube is calculated and denoted as the height ratio of the current frame image, ratio_tire_h. If ratio_tire_h is less than the height ratio threshold, the tire assist ratio tire_R is set to 0; otherwise, it is set to 1. Steps for calculating the berth occupancy ratio: Calculate the berth occupancy ratio R, where R = bottom_iou * 0.2 + left_side_iou * 0.35 + right_side_iou * 0.35 + plate_R * 0.1 + tire_R * 0.1; Steps for determining the parking space status: If the berth occupancy ratio R ≤ λ4, it is determined that the parking space in the current frame image is unoccupied; if λ4 < R < λ5, it is determined that the parking space in the current frame image is partially occupied; if the berth occupancy ratio R ≥ λ5, it is determined that the parking space in the current frame image is fully occupied; output the parking space status of the current frame image.

[0011] The λ4 and λ5 respectively represent the first occupancy ratio threshold and the second occupancy ratio threshold.

[0012] According to another aspect of the present invention, a parking space status detection device based on 3D perspective calculation is provided. The device includes: A low-angle empty and occupied image acquisition module, which is used to respectively use a low-angle camera for N1 parking space areas to collect images of the parking space when it is unoccupied as low-angle empty images, and collect images of the parking space when the parking space is occupied by a vehicle as low-angle occupied images; A target detection model acquisition module, which is used to add a pose key point detection branch to the target detection network based on the target detection network, input N2 labeled vehicle sample images, and obtain a trained target detection model; A 3D cube construction and vehicle feature acquisition module for the parking space area, which is used for each parking space area, using a 3D cube construction module to construct a 3D cube on the low-angle empty image; using the target detection model, in the low-angle occupied image, obtain the vehicle body detection frame, license plate detection frame, tire detection frame, first tire key point, second tire key point, third tire key point, and fourth tire key point; form a triangular area with the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis; A license plate width and height threshold acquisition module, which is used to count the width and height of all license plate detection frames in N1 parking space areas, and respectively use the minimum width and minimum height of the license plate as the license plate width threshold and license plate height threshold; A height ratio threshold acquisition module, which is used for each parking space area, to count the maximum height of the tire detection frame, calculate the ratio of the maximum height of the tire detection frame to the second height of the 3D cube, and record it as the height ratio of the parking space area; use the minimum value of the height ratios in N1 parking space areas as the height ratio threshold; The 3D cube construction and vehicle feature acquisition module for the current frame image is used to acquire the current frame image using a low-angle camera; construct a 3D cube in the current frame image using a 3D cube construction method; and acquire the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point of the current frame image using an object detection model; and use the triangular region formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis of the current frame image. The current frame image parking space status acquisition module is used to calculate the intersection-union ratio of the vehicle chassis and the bottom face of the 3D cube in the current frame image, obtain the license plate assistance ratio and tire assistance ratio, determine the parking space status of the current frame image and output it.

[0013] Wherein, N1 represents the number of parking space areas collected, and N2 represents the number of labeled vehicle sample images.

[0014] The 3D cube building module includes: The three corner point marking modules are used to manually mark three corner points of the parking space area in the image, including: left rear corner point P1, right rear corner point P2, and right front corner point P3; The 3D cube height acquisition module is used to calculate the distances d1, d2, and d3 from the left rear corner point P1, right rear corner point P2, and right front corner point P3 to the upper boundary of the image, respectively; and to calculate the first height H1=d1×λ1, the second height H2=d2×λ2, and the third height H3=d3×λ3 of the 3D cube, respectively, where λ1, λ2, and λ3 represent the first coefficient, the second coefficient, and the third coefficient of the 3D cube height, respectively. The 3D cube construction module for the vacant berth is used to construct the bottom face C_bottom of the 3D cube, which is formed by the left rear corner point P1, right rear corner point P2, and right front corner point P3. The left rear corner point P1, right rear corner point P2, and right front corner point P3 are extended upwards to a first height H1, a second height H2, and a third height H3, respectively, to obtain the first vertex Q1, the second vertex Q2, and the third vertex Q3 of the top face of the 3D cube. The quadrilateral region formed by the left rear corner point P1, right rear corner point P2, second vertex Q2, and first vertex Q1 is used as the left side face C_left_side of the 3D cube. The quadrilateral region formed by the right rear corner point P2, right front corner point P3, third vertex Q3, and second vertex Q2 is used as the right side face C_right_side of the 3D cube.

[0015] Furthermore, the current frame image parking space status acquisition module includes: The Intersection over Union (IoU) calculation module is used to calculate the IoU between the vehicle chassis and the bottom surface of the 3D cube for the current frame image, denoted as bottom_iou, calculate the IoU between the vehicle body detection box and the left side of the 3D cube, denoted as left_side_iou; calculate the IoU between the body detection box and the right side of the 3D cube, denoted as right_side_iou; The license plate auxiliary ratio acquisition module is used to obtain the abscissa of the center point of the license plate detection box of the vehicle for the current frame image. If the abscissa of the center point of the license plate detection box is between the abscissa of the left rear corner point P1 and the abscissa of the right rear corner point P2, and at the same time, the license plate width and license plate height of the vehicle's license plate detection box are greater than the license plate width threshold and the license plate height threshold, then set the license plate auxiliary ratio plate_R to 1, otherwise set the license plate auxiliary ratio plate_R to 0; The tire auxiliary ratio calculation module is used to count the maximum height of the tire detection box for the current frame image, calculate the ratio of the maximum height of the tire detection box to the second height of the 3D cube, denoted as the height ratio ratio_tire_h of the current frame image. If ratio_tire_h is less than the height ratio threshold, then set the tire auxiliary ratio tire_R to 0, otherwise set it to 1; The parking space ratio calculation module is used to calculate the parking space ratio R, where R = bottom_iou * 0.2 + left_side_iou * 0.35 + right_side_iou * 0.35 + plate_R * 0. + tire_R * 0.1; The parking space status determination module is used to determine that the parking space of the current frame image is in an unoccupied state if the parking space ratio R ≤ λ4; determine that the parking space of the current frame image is in a partially occupied state if λ4 < R < λ5; determine that the parking space of the current frame image is in a fully occupied state if the parking space ratio R ≥ λ5; output the parking space status of the current frame image.

[0016] The λ4 and λ5 respectively represent the first ratio threshold and the second ratio threshold.

[0017] Compared with the existing parking space status detection technology, a parking space status detection method based on 3D perspective calculation of the present invention is aimed at low-angle images, based on the construction of a 3D cube, combined with multi-dimensional vehicle feature detection, effectively improving the accuracy of parking space status detection. Brief Description of the Drawings

[0018] Figure 1 Shows a flowchart of a parking space status detection method based on 3D perspective calculation according to the present invention.

[0019] Figure 2A framework diagram of a parking space status detection device based on 3D perspective calculation according to the present invention is shown. Detailed Implementation

[0020] To enable those skilled in the art to further understand the structure, features and other objectives of the present invention, the following detailed description is provided in conjunction with the accompanying preferred embodiments. The preferred embodiments described are only for illustrating the technical solutions of the present invention and are not intended to limit the present invention.

[0021] Figure 1 A flowchart of a parking space status detection method based on 3D perspective calculation according to the present invention is provided. Figure 1 As shown, a parking space status detection method based on 3D perspective calculation according to the present invention includes: In the first step S1, for N1 parking space areas, low-angle cameras are used to collect images of the parking spaces when they are not occupied as low-angle empty space images, and to collect images of the parking spaces when they are occupied by vehicles as low-angle occupied space images. The second step, S2, involves adding a pose keypoint detection branch to the object detection network, inputting N2 labeled vehicle sample images, and obtaining the trained object detection model. In the third step S3, for each parking space area, a 3D cube construction method is used to construct a 3D cube on the low-view empty space image; an object detection model is used to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point in the low-view occupied image; the triangular area formed by the first tire key point, second tire key point, and fourth tire key point is taken as the vehicle chassis; In the third step S4, the width and height of all license plate detection frames within the N1 parking space areas are calculated, and the minimum width and minimum height of the license plate are used as the license plate width threshold and license plate height threshold, respectively. In the fourth step S5, for each parking space area, the maximum height of the tire detection frame is calculated, and the ratio of the maximum height of the tire detection frame to the second height of the 3D cube is recorded as the height ratio of the parking space area; the minimum value of the height ratio of the N1 parking space areas is used as the height ratio threshold. Step S6: Use a low-angle camera to acquire the current frame image; use a 3D cube construction method to construct a 3D cube in the current frame image; use an object detection model to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point of the current frame image; use the triangular region formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis of the current frame image. Step S7: Calculate the intersection-union ratio of the vehicle chassis and the bottom face of the 3D cube in the current frame image, obtain the license plate assistance ratio and the tire assistance ratio; determine the parking space status of the current frame image and output it.

[0022] Wherein, N1 represents the number of parking space areas collected. The value of N1 ranges from 10 to 100,000.

[0023] Furthermore, the target detection network includes: YOLO series target detection networks, SSD target detection network, RetinaNet target detection network, and EfficientDet target detection network.

[0024] The labeled vehicle sample image refers to a vehicle image with a labeled body detection frame, license plate detection frame, tire detection frame, and four tire key points. The four tire key points refer to the contact points between the outer edges of the four tires and the ground in the vehicle sample image, including: a first tire key point, a second tire key point, a third tire key point, and a fourth tire key point. When labeling the vehicle sample image, the tire key point closest to the camera is designated as the first tire key point, the tire key point diagonally opposite the first tire key point is designated as the third tire key point, and the remaining two tire key points are designated as the second and fourth tire key points, respectively.

[0025] N2 represents the number of labeled vehicle sample images, and the value of N2 ranges from 500 to 100,000.

[0026] For example, a pose keypoint detection branch is added to the YOLOv8 object detection network, and 10,000 labeled sample images are input for training to obtain a trained object detection model.

[0027] The 3D cube construction method includes: Step S31: Mark the three corner points in the image. Manually mark the three corner points of the parking space area, including: left rear corner point P1, right rear corner point P2, and right front corner point P3. In step S32 of obtaining the height of the 3D cube, the distances d1, d2, and d3 from the left rear corner point P1, the right rear corner point P2, and the right front corner point P3 to the upper boundary of the image are calculated respectively; the first height H1=d1×λ1, the second height H2=d2×λ2, and the third height H3=d3×λ3 of the 3D cube are calculated respectively, where λ1, λ2, and λ3 represent the first coefficient, the second coefficient, and the third coefficient of the 3D cube height, respectively; In step S33 of constructing the 3D cube for the empty berth, the triangular region formed by the left rear corner point P1, the right rear corner point P2, and the right front corner point P3 is used as the bottom face C_bottom of the 3D cube; the left rear corner point P1, the right rear corner point P2, and the right front corner point P3 are extended upwards to a first height H1, a second height H2, and a third height H3, respectively, to obtain the first vertex Q1, the second vertex Q2, and the third vertex Q3 of the top face of the 3D cube; the quadrilateral region formed by the left rear corner point P1, the right rear corner point P2, the second vertex Q2, and the first vertex Q1 is used as the left side face C_left_side of the 3D cube; the quadrilateral region formed by the right rear corner point P2, the right front corner point P3, the third vertex Q3, and the second vertex Q2 is used as the right side face C_right_side of the 3D cube.

[0028] In the image, of the four corner points of the parking area, the leftmost corner point is marked as the left rear corner point P1, the rightmost corner point is marked as the right front corner point P3, and a line segment P1P3 is formed by connecting the left rear corner point P1 and the right front corner point P3. The corner point located below line segment P1P3 is marked as the right rear corner point P2. In a real scene, the corner point located above P1P3 is not visible in the image because it is at the farthest point of the camera.

[0029] Furthermore, the first coefficient λ1 of the 3D cube height has a value range of [0.58, 0.62], the second coefficient λ2 of the 3D cube height has a value range of [0.88, 0.92], and the third coefficient λ3 of the 3D cube height has a value range of [0.48, 0.52].

[0030] In step S33 of constructing the 3D cube of the empty berth, let the x-coordinate and y-coordinate of the left rear corner point P1 be u1 and v1 respectively, the x-coordinate and y-coordinate of the right rear corner point P2 be u2 and v2 respectively, the x-coordinate and y-coordinate of the right front corner point P3 be u3 and v3 respectively, the x-coordinate and y-coordinate of the first vertex Q1 be u1 and v1-H1 respectively, the x-coordinate and y-coordinate of the second vertex Q2 be u2 and v2-H2 respectively, and the x-coordinate and y-coordinate of the third vertex Q3 be u3 and v3-H3 respectively.

[0031] Further, the seventh step S7 includes: In the intersection-over-union (IoU) calculation step S71, for the current frame image, the IoU of the vehicle chassis and the bottom face of the 3D cube is calculated, denoted as bottom_iou; the IoU of the vehicle body detection box and the left side face of the 3D cube is calculated, denoted as left_side_iou; and the IoU of the vehicle body detection box and the right side face of the 3D cube is calculated, denoted as right_side_iou. License plate auxiliary ratio acquisition step S72: For the current frame image, obtain the abscissa of the center point of the license plate detection box of the vehicle. If the abscissa of the center point of the license plate detection box is between the abscissa of the left rear corner point P1 and the abscissa of the right rear corner point P2, and at the same time, it satisfies that the license plate width and license plate height of the license plate detection box of the vehicle are greater than the license plate width threshold and the license plate height threshold, then set the license plate auxiliary ratio plate_R to 1; otherwise, set the license plate auxiliary ratio plate_R to 0; Tire auxiliary ratio calculation step S73: For the current frame image, count the maximum height of the tire detection box, calculate the ratio of the maximum height of the tire detection box to the second height of the 3D cube, denoted as the height ratio ratio_tire_h of the current frame image. If ratio_tire_h is less than the height ratio threshold, then set the tire auxiliary ratio tire_R to 0; otherwise, set it to 1; Berth ratio calculation step S74: Calculate the berth ratio R, R = bottom_iou * 0.2 + left_side_iou * 0.35 + right_side_iou * 0.35 + plate_R * 0.1 + tire_R * 0.1; Parking space status determination step S75: If the berth ratio R ≤ λ4, then determine that the parking space of the current frame image is in the unoccupied state; if λ4 < R < λ5, then determine that the parking space of the current frame image is in the partially occupied state; if the berth ratio R ≥ λ5, then determine that the parking space of the current frame image is in the fully occupied state; output the parking space status of the current frame image.

[0032] The λ4 and λ5 respectively represent the first ratio threshold and the second ratio threshold. The value range of λ4 is [0.26, 0.33], and the value range of λ5 is [0.57, 0.63].

[0033] Figure 2 The framework diagram of a parking space status detection device based on 3D perspective calculation according to the present invention is given. As Figure 2 shown, a parking space status detection device based on 3D perspective calculation according to the present invention includes: Low - perspective empty and occupied image acquisition module 1, which is used to respectively use a low - perspective camera for N1 parking space areas to collect the image when the parking space is unoccupied as the low - perspective empty image, and collect the image of the parking space when the parking space is occupied by a vehicle as the low - perspective occupied image; Target detection model acquisition module 2, which is used to, based on the target detection network, add a pose key point detection branch to the target detection network, input N2 labeled vehicle sample images, and obtain a trained target detection model; The 3D cube construction and vehicle feature acquisition module 3 for parking space areas is used to construct a 3D cube on the low-view empty space image for each parking space area using the 3D cube construction module; and to acquire the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point in the low-view occupied space image using the target detection model; and to take the triangular area formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis. The license plate width and height threshold acquisition module 4 is used to count the width and height of all license plate detection boxes within the N1 parking space areas, and use the minimum width and minimum height of the license plate as the license plate width threshold and license plate height threshold, respectively. The height ratio threshold acquisition module 5 is used to calculate the maximum height of the tire detection frame for each parking space area, and to calculate the ratio of the maximum height of the tire detection frame to the second height of the 3D cube, which is recorded as the height ratio of the parking space area; the minimum value of the height ratio of the N1 parking space areas is used as the height ratio threshold. The 3D cube construction and vehicle feature acquisition module 6 for the current frame image is used to acquire the current frame image using a low-angle camera; construct a 3D cube in the current frame image using a 3D cube construction method; and acquire the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point of the current frame image using an object detection model; and take the triangular region formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis of the current frame image. The current frame image parking space status acquisition module 7 is used to calculate the intersection-union ratio of the vehicle chassis and the bottom face of the 3D cube in the current frame image, obtain the license plate assistance ratio and the tire assistance ratio; determine the parking space status of the current frame image and output it.

[0034] Wherein, N1 represents the number of parking space areas collected. The value of N1 ranges from 10 to 100,000.

[0035] Furthermore, the target detection network includes: YOLO series target detection networks, SSD target detection network, RetinaNet target detection network, and EfficientDet target detection network.

[0036] The labeled vehicle sample image refers to a vehicle image with a labeled body detection frame, license plate detection frame, tire detection frame, and four tire key points. The four tire key points refer to the contact points between the outer edges of the four tires and the ground in the vehicle sample image, including: a first tire key point, a second tire key point, a third tire key point, and a fourth tire key point. When labeling the vehicle sample image, the tire key point closest to the camera is designated as the first tire key point, the tire key point diagonally opposite the first tire key point is designated as the third tire key point, and the remaining two tire key points are designated as the second and fourth tire key points, respectively.

[0037] N2 represents the number of labeled vehicle sample images, and the value of N2 ranges from 500 to 100,000.

[0038] The 3D cube building module includes: The three corner point marking module 31 is used to manually mark three corner points of the parking space area in the image, including: left rear corner point P1, right rear corner point P2, and right front corner point P3; The 3D cube height acquisition module 32 is used to calculate the distances d1, d2, and d3 from the left rear corner point P1, right rear corner point P2, and right front corner point P3 to the upper boundary of the image, respectively; and to calculate the first height H1=d1×λ1, the second height H2=d2×λ2, and the third height H3=d3×λ3 of the 3D cube, respectively, where λ1, λ2, and λ3 represent the first coefficient, the second coefficient, and the third coefficient of the 3D cube height, respectively. The 3D cube construction module 33 for the empty berth is used to take the triangular area formed by the left rear corner point P1, the right rear corner point P2, and the right front corner point P3 as the bottom face C_bottom of the 3D cube; extend the left rear corner point P1, the right rear corner point P2, and the right front corner point P3 upwards to a first height H1, a second height H2, and a third height H3 respectively to obtain the first vertex Q1, the second vertex Q2, and the third vertex Q3 of the top face of the 3D cube; take the quadrilateral area formed by the left rear corner point P1, the right rear corner point P2, the second vertex Q2, and the first vertex Q1 as the left side face C_left_side of the 3D cube; and take the quadrilateral area formed by the right rear corner point P2, the right front corner point P3, the third vertex Q3, and the second vertex Q2 as the right side face C_right_side of the 3D cube.

[0039] In the image, of the four corner points of the parking area, the leftmost corner point is marked as the left rear corner point P1, the rightmost corner point is marked as the right front corner point P3, and a line segment P1P3 is formed by connecting the left rear corner point P1 and the right front corner point P3. The corner point located below line segment P1P3 is marked as the right rear corner point P2. In a real scene, the corner point located above P1P3 is not visible in the image because it is at the farthest point of the camera.

[0040] Furthermore, the first coefficient λ1 of the 3D cube height has a value range of [0.58, 0.62], the second coefficient λ2 of the 3D cube height has a value range of [0.88, 0.92], and the third coefficient λ3 of the 3D cube height has a value range of [0.48, 0.52].

[0041] In the 3D cube construction module 33 for the empty berth, the x-coordinate and y-coordinate of the left rear corner point P1 are u1 and v1, the x-coordinate and y-coordinate of the right rear corner point P2 are u2 and v2, the x-coordinate and y-coordinate of the right front corner point P3 are u3 and v3, the x-coordinate and y-coordinate of the first vertex Q1 are u1 and v1-H1, the x-coordinate and y-coordinate of the second vertex Q2 are u2 and v2-H2, and the x-coordinate and y-coordinate of the third vertex Q3 are u3 and v3-H3.

[0042] Furthermore, the current frame image parking space status acquisition module 7 includes: The intersection-over-union (IoU) calculation module 71 is used to calculate the IoU of the vehicle chassis and the bottom face of the 3D cube for the current frame image, denoted as bottom_iou; calculate the IoU of the vehicle body detection box and the left side face of the 3D cube, denoted as left_side_iou; and calculate the IoU of the vehicle body detection box and the right side face of the 3D cube, denoted as right_side_iou. The license plate auxiliary ratio acquisition module 72 is used to obtain the horizontal coordinate of the center point of the license plate detection box of the vehicle for the current frame image. If the horizontal coordinate of the center point of the license plate detection box is between the horizontal coordinate of the left rear corner point P1 and the horizontal coordinate of the right rear corner point P2, and the license plate width and license plate height of the vehicle's license plate detection box are greater than the license plate width threshold and the license plate height threshold, then the license plate auxiliary ratio plate_R is set to 1; otherwise, the license plate auxiliary ratio plate_R is set to 0. The tire assist ratio calculation module 73 is used to calculate the maximum height of the tire detection box for the current frame image, and calculate the ratio of the maximum height of the tire detection box to the second height of the 3D cube, which is denoted as the height ratio ratio_tire_h of the current frame image. If the ratio_tire_h is less than the height ratio threshold, the tire assist ratio tire_R is set to 0, otherwise it is set to 1. The berth ratio calculation module 74 is used to calculate the berth ratio R, R=bottom_iou*0.2+left_side_iou*0.35+right_side_iou*0.35+plate_R*0.1+tire_R*0.1; A parking space status determination module 75 is configured to determine that the parking space in the current frame image is in an unoccupied state if the occupancy ratio R ≤ λ4; determine that the parking space in the current frame image is in a partially occupied state if λ4 < R < λ5; determine that the parking space in the current frame image is in a fully occupied state if the occupancy ratio R ≥ λ5; and output the parking space status of the current frame image.

[0043] The λ4 and λ5 respectively represent a first occupancy ratio threshold and a second occupancy ratio threshold. The value range of λ4 is [0.26, 0.33], and the value range of λ5 is [0.57, 0.63].

[0044] Compared with the existing parking space status detection technologies, a parking space status detection method based on 3D perspective calculation according to the present invention is directed to low-angle images. Based on the construction of a 3D cube and combined with multi-dimensional vehicle feature detection, it effectively improves the accuracy of parking space status detection.

[0045] As described above, it is only a preferred embodiment of the present invention and is not used to limit the protection scope of the present invention. It should be understood that the present invention is not limited to the implementation solutions described here, and the purpose of describing these implementation solutions is to help those skilled in the art practice the present invention. Any person skilled in the art can easily make further improvements and refinements without departing from the spirit and scope of the present invention. Therefore, the present invention is only limited by the content and scope of the claims of the present invention, and it is intended to cover all alternative solutions and equivalent solutions included in the spirit and scope of the present invention defined by the appended claims.

Claims

1. A parking space status detection method based on 3D perspective calculation, characterized in that, The method includes: The first step is to use low-angle cameras to collect images of the parking spaces when they are not occupied as low-angle empty space images, and to collect images of the parking spaces when they are occupied by vehicles as low-angle occupied space images. The second step involves adding a pose keypoint detection branch to the object detection network, inputting N2 labeled vehicle sample images, and obtaining the trained object detection model. The third step involves constructing a 3D cube on the low-view empty space image for each parking space area using a 3D cube construction method; and using an object detection model to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point in the low-view occupied space image; and using the triangular area formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis. The third step is to count the width and height of all license plate detection frames within the N1 parking space areas, and use the minimum width and minimum height of the license plate as the license plate width threshold and license plate height threshold, respectively. The fourth step is to calculate the maximum height of the tire detection frame for each parking space area, and then calculate the ratio of the maximum height of the tire detection frame to the second height of the 3D cube, which is recorded as the height ratio of the parking space area. The minimum value of the height ratio of the N1 parking space areas is used as the height ratio threshold. The sixth step involves using a low-angle camera to acquire the current frame image; using a 3D cube construction method to construct a 3D cube in the current frame image; and using an object detection model to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point of the current frame image; and using the triangular region formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis of the current frame image. The seventh step is to calculate the intersection-union ratio of the vehicle chassis and the bottom face of the 3D cube in the current frame image, obtain the license plate assistance ratio and the tire assistance ratio; determine the parking space status of the current frame image and output it. Wherein, N1 represents the number of parking space areas collected, and N2 represents the number of labeled vehicle sample images; the labeled vehicle sample images refer to vehicle images with labeled body detection frames, license plate detection frames, tire detection frames, and four tire key points; the four tire key points refer to the contact points between the outer sides of the four tires of the vehicle and the ground in the vehicle sample image, including: first tire key point, second tire key point, third tire key point, and fourth tire key point; when labeling vehicle sample images, the tire key point closest to the camera is designated as the first tire key point, the tire key point diagonally opposite the first tire key point is designated as the third tire key point, and the remaining two tire key points are designated as the second tire key point and the fourth tire key point, respectively. The 3D cube construction method includes: The three corner point marking steps involve manually marking three corner points of the parking space area in the image: left rear corner point P1, right rear corner point P2, and right front corner point P3. The steps for obtaining the height of the 3D cube are as follows: Calculate the distances d1, d2, and d3 from the left rear corner point P1, right rear corner point P2, and right front corner point P3 to the upper boundary of the image, respectively; calculate the first height H1 = d1 × λ1, the second height H2 = d2 × λ2, and the third height H3 = d3 × λ3 of the 3D cube, respectively, where λ1, λ2, and λ3 represent the first coefficient, second coefficient, and third coefficient of the 3D cube height, respectively. The steps for constructing a 3D cube for an empty berth are as follows: The triangular area formed by the left rear corner point P1, right rear corner point P2, and right front corner point P3 is used as the bottom face of the 3D cube. The left rear corner point P1, right rear corner point P2, and right front corner point P3 are extended upwards to a first height H1, a second height H2, and a third height H3, respectively, to obtain the first vertex Q1, the second vertex Q2, and the third vertex Q3 of the top face of the 3D cube. The quadrilateral area formed by the left rear corner point P1, right rear corner point P2, second vertex Q2, and first vertex Q1 is used as the left side face of the 3D cube. The quadrilateral area formed by the right rear corner point P2, right front corner point P3, third vertex Q3, and second vertex Q2 is used as the right side face of the 3D cube.

2. The method as described in claim 1, characterized in that, The target detection networks include: YOLO series target detection networks, SSD target detection network, RetinaNet target detection network, and EfficientDet target detection network.

3. The method as described in claim 1, characterized in that, The value of N1 ranges from 10 to 100,000, and the value of N2 ranges from 500 to 100,000. The first coefficient λ1 of the 3D cube height ranges from [0.58, 0.62], the second coefficient λ2 of the 3D cube height ranges from [0.88, 0.92], and the third coefficient λ3 of the 3D cube height ranges from [0.48, 0.52].

4. The method as described in claim 1, characterized in that, The seventh step includes: The intersection-over-union (IoU) calculation steps are as follows: For the current frame image, calculate the IoU between the vehicle chassis and the bottom face of the 3D cube, denoted as bottom_iou; calculate the IoU between the vehicle body detection box and the left side face of the 3D cube, denoted as left_side_iou; and calculate the IoU between the vehicle body detection box and the right side face of the 3D cube, denoted as right_side_iou. The license plate auxiliary ratio acquisition step is as follows: For the current frame image, obtain the horizontal coordinate of the center point of the vehicle's license plate detection box. If the horizontal coordinate of the center point of the license plate detection box is between the horizontal coordinates of the left rear corner point P1 and the right rear corner point P2, and simultaneously satisfies the conditions that the license plate width and license plate height of the vehicle's license plate detection box are greater than the license plate width threshold and the license plate height threshold, then set the license plate auxiliary ratio plate_R to 1; otherwise, set the license plate auxiliary ratio plate_R to 0. Steps for calculating the auxiliary ratio of the tire. For the current frame image, the maximum height of the tire detection box is statistically counted, and the ratio of the maximum height of the tire detection box to the second height of the 3D cube is calculated, denoted as the height ratio ratio_tire_h of the current frame image. If ratio_tire_h is less than the height ratio threshold, the auxiliary ratio of the tire tire_R is set to 0; otherwise, it is set to 1. Steps for calculating the berth ratio. Calculate the berth ratio R, where R = bottom_iou * 0.2 + left_side_iou * 0.35 + right_side_iou * 0.35 + plate_R * 0.1 + tire_R * 0.

1. Steps for determining the parking space status. If the berth ratio R ≤ λ4, it is determined that the parking space in the current frame image is unoccupied; if λ4 < R < λ5, it is determined that the parking space in the current frame image is partially occupied; if the berth ratio R ≥ λ5, it is determined that the parking space in the current frame image is fully occupied; output the parking space status of the current frame image. Among them, the λ4 and λ5 respectively represent the first ratio threshold and the second ratio threshold.

5. The method as described in claim 4, characterized in that, The value range of the λ4 is [0.26, 0.33], and the value range of the λ5 is [0.57, 0.63].

6. A parking space status detection device based on 3D perspective calculation, characterized in that, The device includes: A low-angle vacant and occupied image acquisition module, which is used to respectively use a low-angle camera for N1 parking space areas to acquire the image when the parking space is unoccupied as the low-angle vacant image, and acquire the image of the parking space when the parking space is occupied by a vehicle as the low-angle occupied image. A target detection model acquisition module, which is used to add a pose key point detection branch to the target detection network based on the target detection network, input N2 labeled vehicle sample images, and obtain a trained target detection model. A 3D cube construction and vehicle feature acquisition module for the parking space area, which is used to construct a 3D cube on the low-angle vacant image for each parking space area using the 3D cube construction module; use the target detection model to obtain the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point in the low-angle occupied image; form a triangular area with the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis. A license plate width and height threshold acquisition module, which is used to statistically count the width and height of all license plate detection boxes in N1 parking space areas, and respectively use the minimum width and minimum height of the license plate as the license plate width threshold and license plate height threshold. A height ratio threshold acquisition module, which is used to statistically count the maximum height of the tire detection box for each parking space area, calculate the ratio of the maximum height of the tire detection box to the second height of the 3D cube, denoted as the height ratio of the parking space area; use the minimum value of the height ratios in N1 parking space areas as the height ratio threshold. The 3D cube construction and vehicle feature acquisition module for the current frame image is used to acquire the current frame image using a low-angle camera; construct a 3D cube in the current frame image using a 3D cube construction method; and acquire the vehicle body detection box, license plate detection box, tire detection box, first tire key point, second tire key point, third tire key point, and fourth tire key point of the current frame image using an object detection model; and use the triangular region formed by the first tire key point, second tire key point, and fourth tire key point as the vehicle chassis of the current frame image. The current frame image parking space status acquisition module is used to calculate the intersection-union ratio of the vehicle chassis and the bottom face of the 3D cube in the current frame image, obtain the license plate assistance ratio and tire assistance ratio, determine the parking space status of the current frame image and output it; Wherein, N1 represents the number of parking space areas collected, and N2 represents the number of labeled vehicle sample images; the labeled vehicle sample images refer to vehicle images with labeled body detection frames, license plate detection frames, tire detection frames, and four tire key points; the four tire key points refer to the contact points between the outer sides of the four tires of the vehicle and the ground in the vehicle sample image, including: first tire key point, second tire key point, third tire key point, and fourth tire key point; when labeling vehicle sample images, the tire key point closest to the camera is designated as the first tire key point, the tire key point diagonally opposite the first tire key point is designated as the third tire key point, and the remaining two tire key points are designated as the second tire key point and the fourth tire key point, respectively. The 3D cube building module includes: The three corner point marking modules are used to manually mark three corner points of the parking space area in the image, including: left rear corner point P1, right rear corner point P2, and right front corner point P3; The 3D cube height acquisition module is used to calculate the distances d1, d2, and d3 from the left rear corner point P1, right rear corner point P2, and right front corner point P3 to the upper boundary of the image, respectively; and to calculate the first height H1=d1×λ1, the second height H2=d2×λ2, and the third height H3=d3×λ3 of the 3D cube, respectively, where λ1, λ2, and λ3 represent the first coefficient, the second coefficient, and the third coefficient of the 3D cube height, respectively. The 3D cube construction module for the vacant berth is used to construct the bottom face C_bottom of the 3D cube, which is formed by the left rear corner point P1, right rear corner point P2, and right front corner point P3. The left rear corner point P1, right rear corner point P2, and right front corner point P3 are extended upwards to a first height H1, a second height H2, and a third height H3, respectively, to obtain the first vertex Q1, the second vertex Q2, and the third vertex Q3 of the top face of the 3D cube. The quadrilateral region formed by the left rear corner point P1, right rear corner point P2, second vertex Q2, and first vertex Q1 is used as the left side face C_left_side of the 3D cube. The quadrilateral region formed by the right rear corner point P2, right front corner point P3, third vertex Q3, and second vertex Q2 is used as the right side face C_right_side of the 3D cube.

7. The apparatus as claimed in claim 6, characterized in that, The target detection network includes: the target detection network of the YOLO series, the SSD target detection network, the RetinaNet target detection network, and the EfficientDet target detection network.

8. The apparatus as claimed in claim 6, characterized in that, The value range of N1 is 10 to 100000, and the value range of N2 is 500 to 100000; the value range of the first coefficient λ1 of the 3D cube height is [0.58, 0.62], the value range of the second coefficient λ2 of the 3D cube height is [0.88, 0.92], and the value range of the third coefficient λ3 of the 3D cube height is [0.48, 0.52].

9. The apparatus as claimed in claim 6, characterized in that, The current frame image parking space status acquisition module includes: The intersection over union calculation module is used to calculate the intersection over union of the vehicle chassis and the bottom surface of the 3D cube for the current frame image, denoted as bottom_iou, and calculate the intersection over union of the vehicle body detection frame and the left side surface of the 3D cube, denoted as left_side_iou; calculate the intersection over union of the body detection frame and the right side surface of the 3D cube, denoted as right_side_iou; The license plate auxiliary occupancy ratio acquisition module is used to obtain the abscissa of the center point of the license plate detection frame of the vehicle for the current frame image. If the abscissa of the center point of the license plate detection frame is between the abscissa of the left rear corner point P1 and the abscissa of the right rear corner point P2, and at the same time, the license plate width and license plate height of the vehicle's license plate detection frame are greater than the license plate width threshold and license plate height threshold, then set the license plate auxiliary occupancy ratio plate_R to 1, otherwise set the license plate auxiliary occupancy ratio plate_R to 0; The tire auxiliary occupancy ratio calculation module is used to count the maximum height of the tire detection frame for the current frame image, calculate the ratio of the maximum height of the tire detection frame to the second height of the 3D cube, denoted as the height ratio ratio_tire_h of the current frame image. If ratio_tire_h is less than the height ratio threshold, then set the tire auxiliary occupancy ratio tire_R to 0, otherwise set it to 1; The berth occupancy ratio calculation module is used to calculate the berth occupancy ratio R, R = bottom_iou * 0.2 + left_side_iou * 0.35 + right_side_iou * 0.35 + plate_R * 0.1 + tire_R * 0.1; The parking space status determination module is used to determine that the parking space of the current frame image is in an unoccupied state if the berth occupancy ratio R ≤ λ4; determine that the parking space of the current frame image is in a partially occupied state if λ4 < R < λ5; determine that the parking space of the current frame image is in a fully occupied state if the berth occupancy ratio R ≥ λ5; output the parking space status of the current frame image; The λ4 and λ5 respectively represent the first occupancy ratio threshold and the second occupancy ratio threshold.

10. The apparatus as claimed in claim 9, characterized in that, The value range of λ4 is [0.26, 0.33], and the value range of λ5 is [0.57, 0.63].

Images