Automatic parking method and system for stereo garage based on cooperation of multiple vision sensors

Abstract

The invention provides a stereo garage automatic parking method based on cooperation of multiple visual sensors, which comprises the following steps: a vehicle is driven into a buffer parking space of a stereo garage, ground contact points of front and rear tires are obtained by detecting a top view shot at an angle above the parking space, and a wheelbase is obtained according to the distance between the ground contact points of the tires; the distance between rearview mirrors is obtained by detecting a front view shot at the front angle of a parking space and a side view shot by a side vision sensor and serves as the vehicle width; a side view is detected to obtain a vehicle contour line so as to calculate the highest point to calculate the vehicle height, and a rear view is detected through a multi-category detection model to obtain the vehicle modification condition; vehicle body size measurement results including the vehicle width, the vehicle height, the wheelbase and the refitting condition are formed; and controlling the vehicle carrying device to clamp the vehicle wheels on the parking space according to the vehicle body size measurement result. According to the invention, based on multi-view visual perception, high-precision dynamic measurement of the vehicle size is realized.

Description

Technical Field

[0001] This invention relates to the fields of vehicle technology and automated parking garage technology, and particularly to an automated parking method, device, electronic device, computer-readable storage medium, and computer program product based on multi-visual sensor collaboration in automated parking garages. Background Technology

[0002] Stacking-type automated parking systems, also known as lane-stacking automated parking systems, constitute a key form of modern automated parking solutions. The core of this system lies in utilizing vehicle transport devices to precisely move back and forth within the lanes, thereby achieving automatic vehicle storage, retrieval, and positioning. This highly automated operating mode aims to significantly improve space utilization and parking efficiency.

[0003] In the crucial process of vehicle handling, to completely avoid contact with or even damage to the vehicle chassis during lifting or lateral movement, a clamping vehicle transport device has been developed. This device safely moves the vehicle by directly clamping its tires. Its operation relies on a pre-set vehicle model database. After identifying the vehicle model, the system retrieves key parameters such as wheelbase and track width from the database and calculates the precise clamping position accordingly. However, this method, heavily reliant on pre-set data, has inherent limitations. It cannot effectively adapt to new vehicle models not entered into the database, and it struggles to handle significantly modified vehicles with individual differences, leading to access failures when dealing with such vehicles and limiting the system's versatility and flexibility. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an automated parking method for a multi-visual sensor-based automated parking garage, comprising:

[0005] Step 1: The vehicle drives into the buffer parking space of the stacked multi-level parking garage. The buffer parking space is equipped with visual sensors on its four sides and top.

[0006] Step 2: Using multiple vision sensors, obtain the side view, top view, and front view of the vehicle. Detect the top view taken from above the parking space using a keypoint detection model to obtain the ground contact points of the front and rear tires, and calculate the wheelbase based on the distance between these contact points. Detect the front view and side view using the keypoint detection model to obtain the distance between the rearview mirrors as the vehicle width. Detect the side view using a deep learning semantic segmentation model to obtain the vehicle outline, and calculate the vehicle height based on the high points of the outline. Detect the rear view taken by the camera behind the buffer parking space using a multi-class detection model to obtain information about vehicle modifications. This constitutes the vehicle body size measurement results, including vehicle width, vehicle height, wheelbase, and modification details.

[0007] Step 3: Based on the vehicle body size measurement results and the vehicle storage size limit of the stacked automated parking garage, determine whether the vehicle can be parked in the stacked automated parking garage. If yes, proceed to step 4; otherwise, notify the vehicle to leave the stacked automated parking garage.

[0008] Step 4: Control the vehicle transport device to grip the wheels of the vehicle located on the buffer parking space to transfer the vehicle to the designated parking space in the stacked automated parking garage.

[0009] This invention obtains the ground contact points of the front and rear tires by detecting a top-view image taken from above the parking space, and calculates the wheelbase based on the distance between these contact points. It also obtains the vehicle width by detecting a front view taken from the front of the parking space and a side view taken by a side vision sensor, calculating the distance between the rearview mirrors. Furthermore, it obtains the vehicle outline by detecting the side view to calculate the vehicle height from the highest point, and uses a multi-category detection model to detect the rear view, revealing any vehicle modifications. This results in a comprehensive vehicle body size measurement, including width, height, wheelbase, and modification details. Based on these measurements, a vehicle transport device is used to grip the vehicle's wheels in the parking space. This invention enables high-precision dynamic measurement of vehicle dimensions based on multi-view visual perception. Even if a vehicle is not entered into the database, its dimensions can still be obtained, preventing parking failures due to the lack of the vehicle's dimensions in the database. This improves the parking success rate and enhances the versatility and flexibility of the parking system.

[0010] Furthermore, this invention utilizes multiple visual sensors positioned at specific locations within the buffer parking space to capture images from multiple perspectives during the vehicle's parking process. By combining the correlation between these images and a neural network model, image information from different images can be extracted. Furthermore, by combining the positional relationships between these image information, precise vehicle size information can be obtained. Moreover, this invention employs a purely visual approach, avoiding the problem of laser light reflecting poorly back to the sensor when it strikes the roof glass, resulting in severe point cloud distortion and excessive noise. This application also reduces the technical issue of reduced recognition accuracy due to light reflection from ambient materials.

[0011] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 2, which comprises:

[0012] A vehicle database containing multiple vehicle models and their corresponding body dimensions is constructed. The vehicle model is obtained through the target detection model. Based on the vehicle model, the vehicle database is searched to obtain the estimated values ​​of wheelbase, width and height, which are used as the measurement results of the vehicle body dimensions.

[0013] Therefore, this invention can construct a database from vehicle size information published by the Ministry of Industry and Information Technology or from vehicle size information published on the manufacturer's official website, and then obtain vehicle size information simply by searching, reducing the amount of real-time calculation and obtaining vehicle size information more efficiently.

[0014] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 2, which comprises:

[0015] Step 22: Use a visual sensor to detect the front and rear overhangs as the vehicle enters the buffer parking space, and obtain the front and rear overhang length values ​​of the vehicle.

[0016] Step 23: After the vehicle stops in the buffer parking space and the driver leaves the buffer parking space, the vehicle angle is determined based on the image collected by the vision sensor, and the vehicle posture is automatically adjusted to center the vehicle in the buffer parking space.

[0017] Step 24: After adjusting the vehicle's posture, measure the wheelbase, width, and height of the vehicle, and inspect the vehicle's exterior modifications to obtain the vehicle's dimensions, including wheelbase, width, height, and modifications.

[0018] Therefore, this invention can obtain vehicle body size measurement results including front and rear overhang length values.

[0019] The aforementioned method for automatic parking in a multi-sensor automated parking garage, wherein the vision sensor in step 2 is a camera, and the front and rear overhang detection includes:

[0020] Step 221: Calibrate the camera and obtain the camera intrinsic parameter matrix;

[0021] Step 222: Install cameras on the top left front, right front, right rear and left rear of the buffer parking space respectively to shoot the vehicle body from above, so that the camera optical axis is perpendicular to the ground. The left front and right front cameras are responsible for front suspension detection, and the left rear and right rear cameras are responsible for rear suspension detection.

[0022] Step 223: Establish a world coordinate system on the buffer parking space, obtain the transformation matrix between the camera coordinate system and the world coordinate system, and mark the edge lines on the corrected image. The left front camera marks the front edge line and the left edge line, the right front camera marks the front edge line and the right edge line, the left rear camera marks the rear edge line and the left edge line, and the right rear camera marks the rear edge line and the right edge line.

[0023] Step 224: The camera continuously captures the process of the vehicle entering the buffer parking space. The contact point between the wheel and the ground is point a, the vertical projection point of the rear of the vehicle on the ground is point B, and the camera captures the projection point of the rear of the vehicle on the ground as point b. When the rear of the vehicle is directly below the camera, point B coincides with point b, and step 225 is executed.

[0024] Step 225: Perform target detection on the captured image using a key point detection model to detect the contact points a between the wheels and the ground, including: ground contact point a1 of the left front wheel, ground contact point a2 of the right front wheel, ground contact point a3 of the left rear wheel, and ground contact point a4 of the right rear wheel. Calculate the distance from the contact point to the edge line of the parking space. Obtain the tilt angle theta of the vehicle from the distance between the wheel contact point on the same side of the wheel and the edge line.

[0025] Step 226: Rotate the image according to the tilt angle so that the car body is not tilted in the rotated image. Use the car body target detection model to detect and obtain the detection box of the car body. Record the distance dist of the corner point b of the car body detection box in the rotated image from the center line of the image in the height direction.

[0026] Step 227: Restore corner point b back to point b in the unrotated image according to the theta angle, calculate the distance between point b and wheel ground contact point a in the world coordinate system, and obtain the rear overhang value lb2 detected by the right rear overhang camera and the rear overhang value lb1 detected by the left rear overhang camera.

[0027] Step 228: Restore the upper right corner point b of the vehicle body detection box in the rotated image back to point b in the unrotated image according to the theta angle. Calculate the distance between point b and the wheel ground contact point a in the world coordinate system to obtain the front suspension value lf2 detected by the right front suspension camera and the front suspension value lf1 detected by the left front suspension camera. Record the distance dist calculated in step 226 and the distance between the wheel ground contact point and the left and right edge lines calculated in step 225 with the obtained front suspension values. Record the detection results of the left and right cameras respectively.

[0028] Step 229: Repeat steps 224 to 228 until the distance dist is greater than the set threshold, or the number of calculations exceeds the set number of times.

[0029] Step 2210: Compare the distances (dist) acquired by the four cameras in step 226, and take the front or back overhang value corresponding to the smallest distance (dist) value as the final front and back overhang values.

[0030] Therefore, by limiting the timing of taking photos and the calculation method, this invention can obtain the front and rear overhang lengths of a vehicle more accurately.

[0031] By centering the vehicle, this invention can unify the vehicle's position in various coordinate systems, thereby improving the accuracy of vehicle size measurement. Furthermore, because the vehicle is centered, it is more convenient to clamp the wheels subsequently, improving the clamping accuracy of the vehicle conveying device and reducing the error rate in the wheel clamping process.

[0032] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 2210, which comprises:

[0033] In the front suspension value determination step, if step 2210 obtains the calculated values ​​Lf1 from the left front suspension camera and Lf2 from the right front suspension camera, then the distances corresponding to the two values ​​to the left and right edge lines are determined respectively. If the distance between the wheel ground contact point corresponding to one of Lf1 and Lf2 and the edge line is less than a threshold, then this value is discarded, and the other value is taken as the final front suspension value. If both are greater than the threshold, then the average of the two is taken. If step 210 obtains only one front suspension value, then the previous front suspension value is taken as the final front suspension value.

[0034] In the rear suspension value determination step, if the calculated values ​​Lb1 and Lb2 of the left rear suspension camera are obtained in step 2210, then the distances of the two values ​​to the left and right edge lines are determined respectively. If the distance between the wheel ground contact point corresponding to one of Lb1 and Lb2 and the edge line is less than the threshold, then this value is discarded and the other value is taken as the final rear suspension value; if both are greater than the threshold, then the average value of the two is taken.

[0035] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 23, which comprises:

[0036] Step 231: Install the cameras at the front left, front right, rear left, and rear right of the buffer parking space to obtain four cameras including a front left camera, a front right camera, a rear left camera, and a rear right camera.

[0037] Step 232: Calibrate the four cameras separately and obtain the intrinsic parameters of each camera;

[0038] Step 233: Establish a world coordinate system on the buffer parking space, obtain the transformation matrix between the camera coordinate system and the world coordinate system for each camera, use it as the camera extrinsic parameter, and mark the edge lines of the parking space on the corrected image. The left front camera marks the front edge line and left edge line of the parking space, the right front camera marks the front edge line and right edge line of the parking space, the left rear camera marks the rear edge line and left edge line of the parking space, and the right rear camera marks the rear edge line and right edge line of the parking space.

[0039] Step 234: After the vehicle stops, each wheel is located on the belt of the centering mechanism. The images captured by the four cameras are obtained. The captured images are corrected using the corresponding intrinsic parameters. The corrected images are then used to perform target detection using a key point detection model to obtain the contact point a between each wheel and the ground. The distance between the contact point a and the marked parking space edge line in the world coordinate system is calculated.

[0040] Step 235: Based on the distance of each wheel from the edge of the parking space in the world coordinate system, calculate the distance the front and rear wheels have moved, and control the centering mechanism to move the belt under each wheel to adjust the vehicle's posture and center the vehicle in the buffer parking space.

[0041] The aforementioned automated parking method for a multi-visual sensor collaborative parking garage, wherein step 234, calculating the distance between contact point a and the marked parking space edge line in the world coordinate system, specifically includes:

[0042] Calculate the perpendicular point a' of contact point a on the edge line of the parking space in the image coordinate system, and convert the two points to coordinates in the world coordinate system as follows:

[0043] K is the camera's intrinsic parameter matrix; R is the rotation matrix, representing the rotation from the world coordinate system to the camera coordinate system; T is the translation vector, representing the translation from the world coordinate system to the camera coordinate system; (x, y) are the pixel coordinates on the image; Z is the known depth value, representing the Z-axis coordinate of the point in the world coordinate system; p is the homogeneous pixel coordinate vector [xy1]T; p norm p is the normalized camera coordinate vector; world World coordinate vector;

[0044] Calculate normalized camera coordinates:

[0045] Use the inverse K−1 of the intrinsic parameter matrix to convert the pixel coordinates to normalized camera coordinates;

[0046] The third component of the vector, i.e., the homogeneous coordinate component, is represented here.

[0047] Calculate world coordinates:

[0048] Using the transpose of the rotation matrix The translation vector T converts the normalized camera coordinates to world coordinates.

[0049] Combine the above steps

[0050]

[0051] From p world Extract the X, Y, and Z components of the world coordinate system:

[0052]

[0053] Where a(x1,y1,z1) and a'(x2,y2,z2) are used to calculate the distance between two points in the world coordinate system:

[0054]

[0055] The ground contact points a1 (left front wheel), a2 (right front wheel), a3 (left rear wheel), and a4 (right rear wheel) are obtained. The distances from each contact point to the edge of the parking space are also calculated. Specifically, the distances from the left front wheel contact point a1 to the front edge of the parking space are d1f and d1l; the distances from the right front wheel contact point a2 to the front edge of the parking space are d2f and d2r; the distances from the left rear wheel contact point a3 to the rear edge of the parking space are d3b and d3l; and the distances from the right rear wheel contact point a4 to the rear edge of the parking space are d4b and d4r.

[0056] The distance the front wheels should move when aligned is (d1l-d2r) / 2; the distance the rear wheels should move when aligned is (d3l-d4r) / 2.

[0057] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 24, which comprises:

[0058] Wheelbase measurement steps:

[0059] The length of the buffer parking space is l. The wheelbase is obtained from the distances of the left front wheel and the left rear wheel from the edge line of the parking space.

[0060]

[0061] The wheelbase is calculated from the distances of the right front wheel and the right rear wheel from the edge of the parking space.

[0062]

[0063] Take the average of the two wheelbases as the final wheelbase value axle = (axle1 + axle2) / 2;

[0064] Vehicle width measurement steps:

[0065] The cameras were installed at the left front, right front, and front of the buffer parking space, respectively, resulting in three cameras: a left front camera, a right front camera, and a front camera. The left front camera and the right front camera were used to shoot the vehicle vertically from above, while the front camera was installed in front of the buffer parking space to shoot the front of the vehicle horizontally.

[0066] The three cameras were calibrated to obtain the camera's intrinsic parameters K1, K2, K3 and distortion coefficients D1, D2, D3;

[0067] The distortion coefficients D1, D2, and D3 are used to correct the distortion of the images captured by each camera, and the corrected images are output.

[0068] The rearview mirrors of the vehicle are detected using a key point detection model on the image corrected by the front-end camera, and the pixel coordinates (u1, v1) and (u1', v1') of the outermost point of each rearview mirror in the image are obtained respectively. The rearview mirrors are detected using a key point detection model on the image corrected by the left front camera, and the pixel coordinates (u2, v2) of the outermost point of the rearview mirror in the image are obtained. The rearview mirrors are detected using a key point detection model on the image corrected by the right front camera, and the pixel coordinates (u2', v2') of the outermost point of the rearview mirror in the image are obtained.

[0069] The outermost point P of the left rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1, v1) and (u2, v2). The outermost point P' of the right rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1', v1') and (u2', v2'). Find the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the coordinate system of the front camera. The front camera, the left front camera, and the right front camera are Camera1, Camera2, and Camera3, respectively.

[0070] Find the coordinates (X', Y', Z') of the outermost point P' of the right rearview mirror in Camera1, and calculate the spatial distance d between the two points as the vehicle width:

[0071] .

[0072] Vehicle height measurement steps:

[0073] The vehicle height detection uses a front-end camera and a side camera. The side camera is installed on the side of the buffer parking space to capture the side of the vehicle horizontally.

[0074] Establish a world coordinate system for the parking space and calibrate the extrinsic parameters of the front-end camera, namely the transformation matrix Rt between its camera coordinate system and the world coordinate system;

[0075] After the vehicle comes to a complete stop and is centered, the optical axis of the side camera is aligned with the optical axis of the front camera, and the front camera and the side camera take pictures respectively. The distortion of the images taken by the side camera and the front camera is corrected, and the corrected image is output.

[0076] The vehicle and background are segmented using a deep learning semantic segmentation model in the image corrected by the side camera. The vehicle outline is extracted, and the pixel coordinates (u1, v1) of the highest point on the side of the vehicle in the image are obtained by searching the outline point pairs. Similarly, the vehicle and background are segmented using a deep learning semantic segmentation model in the image corrected by the front camera. The vehicle outline is extracted, and the pixel coordinates (u2, v2) of the highest point on the side of the vehicle in the image are obtained by searching the outline point pairs.

[0077] The same spatial point P has corresponding pixel observation points in both cameras: (u1, v1) and (u2, v2). To find the coordinates (X, Y, Z) of point P in the front-end camera coordinate system, refer to step 5 of the vehicle width detection method. Using the transformation matrix obtained in step 2, the coordinates (Xw, Yw, Zw) of the point in the world coordinate system are obtained as follows:

[0078]

[0079] Where Zw is the height component, which is the desired vehicle height;

[0080] Inspection steps for vehicle exterior modifications:

[0081] Vehicle modifications are detected by a rear camera located above and behind the buffer parking space. A labeled dataset containing multiple exterior modification categories is constructed, and an end-to-end multi-class detection model is trained. After vehicle alignment, the rear camera captures an image of the area above and behind the vehicle. This image is then input into the target detection model for inference. For each detected exterior modification, the spatial geometric relationship between its detection box and the main detection box of the vehicle category is analyzed to determine whether it belongs to the current vehicle. The determination process includes: whether the modification detection box is located within or adjacent to the vehicle detection box, and whether it meets the preset position rationality rules. If so, the exterior modification is determined to belong to the current vehicle; otherwise, it is considered a false detection or a component of a neighboring vehicle and is removed.

[0082] The aforementioned automatic parking method for a multi-vision sensor collaborative automated parking garage, wherein the specific calculation process of the coordinates (X, Y, Z) in the front-end camera coordinate system is as follows:

[0083] Backprojecting the pixel coordinates of each camera into a ray, for Camera 1:

[0084]

[0085] In the Camera 1 coordinate system, this is ray 1 originating from the optical center of Camera 1:

[0086]

[0087] For Camera2:

[0088] Calculate the normalization direction:

[0089]

[0090] Transform it to the coordinate system of Camera1:

[0091]

[0092] It is the rotation matrix from Camera2 to Camera1;

[0093] In the camera1 coordinate system, ray 2 from Camera2 is:

[0094]

[0095] Triangulation to find points: Both rays are in the Camera1 coordinate system:

[0096] Ray 1:

[0097] Ray 2:

[0098] Find the point of shortest distance between these two rays:

[0099]

[0100] Solve Substituting these values ​​into the system will give you the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the Camera1 coordinate system.

[0101] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 2, which comprises:

[0102] Step 241: The multi-view image is analyzed collaboratively by the key point detection model, semantic segmentation model, and object detection model to obtain the vehicle body size measurement results, including wheelbase, width, height, length, and front and rear overhang values, as AI measurement values; a vehicle database including multiple vehicle models and their corresponding body sizes is constructed; the vehicle model is obtained through the object detection model; the vehicle database is searched based on the vehicle model to determine whether the vehicle is located in the vehicle database; if so, the inferred values ​​including wheelbase, width, height, length, and front and rear overhang values ​​are obtained, and step 242 is executed; otherwise, step 244 is executed.

[0103] Step 242: Determine whether the difference between the radar measurement value and the predicted value is within the first preset range. If yes, the verification is successful. Take the median of the radar measurement value and the vehicle's historical measurement value as the new predicted value and use it for this vehicle storage. If the difference exceeds the first preset range, proceed to step 243.

[0104] Step 243: Determine whether the difference between the AI ​​measurement value and the predicted value is within the first preset range. If so, the verification is successful. Take the median of the current AI measurement value and the vehicle's historical measurement value as the new predicted value and use it for this parking. If the difference exceeds the first preset range, proceed to step 244.

[0105] Step 244: Determine whether the difference between the current radar measurement value and the AI ​​measurement value is within the second preset range. If yes, the verification is successful. Take the median of the current radar measurement value and the vehicle's historical measurement values ​​as the new inferred value and use it for this vehicle storage. If the difference exceeds the second preset range, proceed to step 45.

[0106] Step 245: Determine whether the difference between the current radar measurement value and the AI ​​measurement value is within the third preset range. If so, the verification is successful. Take the median of the current radar measurement value and the vehicle's historical measurement value as the new inferred value and use it for this parking, and reduce the parking speed. If the difference exceeds the third preset range, the verification fails and parking is not allowed.

[0107] Therefore, this invention can simultaneously obtain the inferred value obtained from database retrieval and the measured value obtained by AI measurement through multi-vision sensor collaboration through two schemes: a database scheme and a multi-vision sensor collaboration scheme. By verifying the inferred value and the measured value, the accuracy of vehicle body size measurement is further improved, thereby enhancing vehicle storage safety. The two schemes also improve the adaptability of the vehicle storage system.

[0108] The aforementioned method for automatic parking in a multi-sensor automated parking garage includes step 4, which comprises:

[0109] During the vehicle transport process, the vehicle transport device performs real-time anomaly detection. If no anomaly is found, the vehicle is transferred to a designated parking space in the stacked automated parking garage. If the target vehicle frame parking space is abnormal, step 41 is executed. If the vehicle transport device malfunctions, step 42 is executed. If the buffer parking space malfunctions, step 43 is executed. If the vehicle exceeds the RCV service range, step 44 is executed.

[0110] Step 41: Reassign vehicle rack parking spaces. If there is a available rack parking space for the vehicle, the control device will change the vehicle's parking space. If there is no rack parking space, the vehicle will be assigned to an empty buffer parking space. This avoids the situation where the vehicle cannot be parked, which would result in the vehicle being unable to be stored. This invention will temporarily place the vehicle in the buffer parking space, allowing the buffer parking space to serve as a temporary parking space, thereby maximizing the parking rate of the garage.

[0111] Step 42: Set the parking area where the vehicle conveyor is located to be temporarily suspended, and send a departure reminder to users currently queuing in the area, informing them that the area has been temporarily suspended.

[0112] Step 43: Set the currently faulty buffer parking space to suspended service, send a vehicle relocation reminder to the owner of the vehicle waiting to be parked, and guide them to move their vehicle away from the faulty buffer parking space.

[0113] Step 44: Determine whether the buffer parking space where the vehicle was parked is occupied by another vehicle. If so, the vehicle transport device will move the vehicle to the buffer parking space. Otherwise, send a reminder to the owner of the vehicle occupying the buffer parking space to move the vehicle. After the occupying vehicle leaves, the vehicle transport device will move the vehicle to the buffer parking space.

[0114] Therefore, this invention provides a complete solution to parking anomalies, in order to maximize parking efficiency and provide comprehensive parking guidance for car owners.

[0115] This invention also proposes an automated parking system B for a multi-visual sensor-based automated parking garage, comprising:

[0116] The initial module involves vehicles entering buffer parking spaces in a stacked multi-level parking garage, where visual sensors are installed around and on top of the buffer parking spaces.

[0117] The fusion module uses multiple vision sensors to obtain the vehicle's side, top, and front views. It uses a keypoint detection model to detect the top view taken from above the parking space, obtaining the ground contact points of the front and rear tires, and calculating the wheelbase based on the distance between these points. The keypoint detection model also detects the front and side views, obtaining the distance between the rearview mirrors as the vehicle width. A deep learning semantic segmentation model detects the side view, obtaining the vehicle's outline, and calculating the vehicle height based on the high points of the outline. Finally, a multi-class detection model detects the rear view captured by a camera behind the buffer parking space, revealing any modifications to the vehicle. These results constitute a comprehensive measurement of the vehicle's dimensions, including width, height, wheelbase, and modifications.

[0118] The judgment module determines whether the vehicle can be parked in the stacked automated parking garage based on the vehicle size measurement results and the parking size limit of the stacked automated parking garage. If yes, the transfer module is called; otherwise, the vehicle is notified to leave the stacked automated parking garage.

[0119] The transfer module controls the vehicle transport device to grip the wheels of a vehicle located in the buffer parking space, so as to transfer the vehicle to a designated parking space in the stacked automated parking garage.

[0120] The aforementioned method for automatic parking in a multi-sensor automated parking garage, wherein the fusion module includes:

[0121] A vehicle database containing multiple vehicle models and their corresponding body dimensions is constructed. The vehicle model is obtained through the target detection model. Based on the vehicle model, the vehicle database is searched to obtain the estimated values ​​of wheelbase, width and height, which are used as the measurement results of the vehicle body dimensions.

[0122] The aforementioned method for automatic parking in a multi-sensor automated parking garage, wherein the fusion module includes:

[0123] Module 22: Detects the front and rear overhangs of the vehicle as it enters the buffer parking space using a vision sensor, and obtains the front and rear overhang length values ​​of the vehicle.

[0124] Module 23: When the vehicle stops in the buffer parking space and the driver leaves the buffer parking space, the angle of the vehicle is determined based on the image collected by the vision sensor, and the vehicle posture is automatically adjusted to center the vehicle in the buffer parking space.

[0125] Module 24: After adjusting the vehicle's posture, perform wheelbase measurement, vehicle width measurement, vehicle height measurement, and inspection of vehicle exterior modifications to obtain vehicle body size measurement results including wheelbase, vehicle width, vehicle height, and modifications.

[0126] The aforementioned method for automatic parking in a multi-sensor automated parking garage, wherein the visual sensor in the fusion module is a camera, and the front and rear overhang detection includes:

[0127] Module 221: Calibrate the camera and obtain the camera intrinsic parameter matrix;

[0128] Module 222: Install cameras on the top left front, right front, right rear and left rear of the buffer parking space respectively to shoot the vehicle body from above, so that the camera optical axis is perpendicular to the ground. The left front and right front cameras are responsible for front suspension detection, and the left rear and right rear cameras are responsible for rear suspension detection.

[0129] Module 223: Establish a world coordinate system on the buffer parking space, obtain the transformation matrix between the camera coordinate system and the world coordinate system, and mark the edge lines on the corrected image. The left front camera marks the front edge line and the left edge line, the right front camera marks the front edge line and the right edge line, the left rear camera marks the rear edge line and the left edge line, and the right rear camera marks the rear edge line and the right edge line.

[0130] Module 224: The camera continuously captures the process of the vehicle entering the buffer parking space. The contact point between the wheel and the ground is point a, and the vertical projection point of the rear of the vehicle on the ground is point B. The camera captures the projection point of the rear of the vehicle on the ground as point b. When the rear of the vehicle is directly below the camera, point B coincides with point b. Module 225 is executed.

[0131] Module 225: Perform target detection on the captured image using a key point detection model, detect the contact points a between the wheel and the ground, including: ground contact point a1 of the left front wheel, ground contact point a2 of the right front wheel, ground contact point a3 of the left rear wheel, and ground contact point a4 of the right rear wheel, and calculate the distance from the contact point to the edge line of the parking space. The tilt angle theta of the vehicle is obtained from the distance from the wheel contact point on the same side of the wheel to the edge line.

[0132] Module 226: Rotate the image according to the tilt angle so that the vehicle body is not tilted in the rotated image. Use the vehicle body target detection model to detect and obtain the detection box of the vehicle body. Record the distance dist of the corner point b of the vehicle body detection box in the rotated image from the center line of the image in the height direction.

[0133] Module 227: Restore corner point b back to point b in the unrotated image according to the theta angle, calculate the distance between point b and wheel ground contact point a in the world coordinate system, and obtain the rear overhang value lb2 detected by the right rear overhang camera and the rear overhang value lb1 detected by the left rear overhang camera.

[0134] Module 228: Restore the upper right corner point b of the vehicle body detection box in the rotated image back to point b in the unrotated image according to the theta angle. Calculate the distance between point b and the wheel ground contact point a in the world coordinate system to obtain the front suspension value lf2 detected by the right front suspension camera and the front suspension value lf1 detected by the left front suspension camera. Record the distance dist calculated in step 6 and the distance between the wheel ground contact point and the left and right edge lines calculated in step 5 with the obtained front suspension values. Record the detection results of the left and right cameras respectively.

[0135] Module 229, repeat modules 224 to 228 until the distance dist is greater than the set threshold, or the number of calculations exceeds the set number of times;

[0136] Module 2210 compares the distances (dist) acquired by the four cameras in module 226, and takes the front or back overhang value corresponding to the smallest distance (dist) value as the final front and back overhang value.

[0137] The aforementioned automatic parking method for a multi-visual sensor collaborative automated parking garage, wherein module 2210 includes:

[0138] In the front suspension value determination module, if module 2210 obtains the calculated value Lf1 from the left front suspension camera and the calculated value Lf2 from the right front suspension camera, it determines the distances to the left and right edge lines corresponding to the two values ​​respectively. If the distance between the wheel ground contact point corresponding to one of Lf1 and Lf2 and the edge line is less than a threshold, this value is discarded, and the other value is taken as the final front suspension value. If both are greater than the threshold, the average of the two is taken. If module 210 obtains only one front suspension value, the previous front suspension value is taken as the final front suspension value.

[0139] If module 2210 obtains the calculated value Lb1 from the left rear suspension camera and the calculated value Lb2 from the right rear suspension camera, then it determines the distances of the two values ​​to the left and right edge lines respectively. If the distance between the wheel ground contact point corresponding to one of Lb1 and Lb2 and the edge line is less than a threshold, then this value is discarded and the other value is taken as the final rear suspension value; if both are greater than the threshold, then the average of the two is taken.

[0140] The aforementioned automatic parking method for a multi-visual sensor collaborative automated parking garage, wherein module 23 includes:

[0141] Module 231: Install the cameras at the front left, front right, rear left, and rear right of the buffer parking space respectively, to obtain four cameras including a front left camera, a front right camera, a rear left camera, and a rear right camera.

[0142] Module 232: Calibrate the four cameras separately and obtain the intrinsic parameters of each camera;

[0143] Module 233: Establish a world coordinate system on the buffer parking space, obtain the transformation matrix between the camera coordinate system and the world coordinate system of each camera as the camera extrinsic parameter, and mark the edge lines of the parking space on the corrected image. The left front camera marks the front edge line and left edge line of the parking space, the right front camera marks the front edge line and right edge line of the parking space, the left rear camera marks the rear edge line and left edge line of the parking space, and the right rear camera marks the rear edge line and right edge line of the parking space.

[0144] Module 234: After the vehicle stops, each wheel is located on the belt of the centering mechanism. The images captured by the four cameras are acquired, and the captured images are corrected using the corresponding intrinsic parameters. The corrected images are then used to perform target detection using a key point detection model to obtain the contact point a between each wheel and the ground. The distance between the contact point a and the marked edge line of the parking space in the world coordinate system is calculated.

[0145] Module 235: Based on the distance of each wheel from the edge of the parking space in the world coordinate system, calculate the distance the front and rear wheels have moved, and control the centering mechanism to move the belt under each wheel to adjust the vehicle's posture and center the vehicle in the buffer parking space.

[0146] The aforementioned automated parking method for a multi-visual sensor collaborative parking garage includes, in module 234, the calculation of the distance between contact point a and the marked parking space edge line in the world coordinate system, specifically including:

[0147] Calculate the perpendicular point a' of contact point a on the edge line of the parking space in the image coordinate system, and convert the two points to coordinates in the world coordinate system as follows:

[0148] K is the camera's intrinsic parameter matrix; R is the rotation matrix, representing the rotation from the world coordinate system to the camera coordinate system; T is the translation vector, representing the translation from the world coordinate system to the camera coordinate system; (x, y) are the pixel coordinates on the image; Z is the known depth value, representing the Z-axis coordinate of the point in the world coordinate system; p is the homogeneous pixel coordinate vector [xy1]T; p norm p is the normalized camera coordinate vector; world World coordinate vector;

[0149] Calculate normalized camera coordinates:

[0150] Use the inverse K−1 of the intrinsic parameter matrix to convert the pixel coordinates to normalized camera coordinates;

[0151]

[0152] Where (K−1p)3 represents the third component of vector K−1p, that is, the w component of homogeneous coordinates;

[0153] Calculate world coordinates:

[0154] Using the transpose of the rotation matrix The translation vector T converts the normalized camera coordinates to world coordinates.

[0155]

[0156] Merge the above modules:

[0157]

[0158] From p world Extract the X, Y, and Z components of the world coordinate system:

[0159]

[0160] Where a(x1,y1,z1) and a'(x2,y2,z2) are used to calculate the distance between two points in the world coordinate system:

[0161]

[0162] The ground contact points a1 (left front wheel), a2 (right front wheel), a3 (left rear wheel), and a4 (right rear wheel) are obtained. The distances from each contact point to the edge of the parking space are also calculated. Specifically, the distances from the left front wheel contact point a1 to the front edge of the parking space are d1f and d1l; the distances from the right front wheel contact point a2 to the front edge of the parking space are d2f and d2r; the distances from the left rear wheel contact point a3 to the rear edge of the parking space are d3b and d3l; and the distances from the right rear wheel contact point a4 to the rear edge of the parking space are d4b and d4r.

[0163] The distance the front wheels should move when aligned is (d1l-d2r) / 2; the distance the rear wheels should move when aligned is (d3l-d4r) / 2.

[0164] The aforementioned automatic parking method for a multi-visual sensor collaborative automated parking garage, wherein module 24 includes:

[0165] Wheelbase measurement module:

[0166] The length of the buffer parking space is l. The wheelbase is obtained from the distances of the left front wheel and the left rear wheel from the edge line of the parking space.

[0167]

[0168] The wheelbase is calculated from the distances of the right front wheel and the right rear wheel from the edge of the parking space.

[0169]

[0170] Take the average of the two wheelbases as the final wheelbase value axle = (axle1 + axle2) / 2;

[0171] Vehicle width measurement module:

[0172] The cameras were installed at the left front, right front, and front of the buffer parking space, respectively, resulting in three cameras: a left front camera, a right front camera, and a front camera. The left front camera and the right front camera were used to shoot the vehicle vertically from above, while the front camera was installed in front of the buffer parking space to shoot the front of the vehicle horizontally.

[0173] The three cameras were calibrated to obtain the camera's intrinsic parameters K1, K2, K3 and distortion coefficients D1, D2, D3;

[0174] The distortion coefficients D1, D2, and D3 are used to correct the distortion of the images captured by each camera, and the corrected images are output.

[0175] The rearview mirrors of the vehicle are detected using a key point detection model on the image corrected by the front-end camera, and the pixel coordinates (u1, v1) and (u1', v1') of the outermost point of each rearview mirror in the image are obtained respectively. The rearview mirrors are detected using a key point detection model on the image corrected by the left front camera, and the pixel coordinates (u2, v2) of the outermost point of the rearview mirror in the image are obtained. The rearview mirrors are detected using a key point detection model on the image corrected by the right front camera, and the pixel coordinates (u2', v2') of the outermost point of the rearview mirror in the image are obtained.

[0176] The outermost point P of the left rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1, v1) and (u2, v2). The outermost point P' of the right rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1', v1') and (u2', v2'). Find the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the coordinate system of the front camera. The front camera, the left front camera, and the right front camera are Camera1, Camera2, and Camera3, respectively.

[0177] Find the coordinates (X', Y', Z') of the outermost point P' of the right rearview mirror in Camera1, and calculate the spatial distance d between the two points as the vehicle width:

[0178] .

[0179] Vehicle height measurement module:

[0180] The vehicle height detection uses a front-end camera and a side camera. The side camera is installed on the side of the buffer parking space to capture the side of the vehicle horizontally.

[0181] Establish a world coordinate system for the parking space and calibrate the extrinsic parameters of the front-end camera, namely the transformation matrix Rt between its camera coordinate system and the world coordinate system;

[0182] After the vehicle comes to a complete stop and is centered, the optical axis of the side camera is aligned with the optical axis of the front camera, and the front camera and the side camera take pictures respectively. The distortion of the images taken by the side camera and the front camera is corrected, and the corrected image is output.

[0183] The vehicle and background are segmented using a deep learning semantic segmentation model in the image corrected by the side camera. The vehicle outline is extracted, and the pixel coordinates (u1, v1) of the highest point on the side of the vehicle in the image are obtained by searching the outline point pairs. Similarly, the vehicle and background are segmented using a deep learning semantic segmentation model in the image corrected by the front camera. The vehicle outline is extracted, and the pixel coordinates (u2, v2) of the highest point on the side of the vehicle in the image are obtained by searching the outline point pairs.

[0184] A point P in space has corresponding pixel observation points in both cameras: (u1, v1) and (u2, v2). Find the coordinates (X, Y, Z) of point P in the front-end camera coordinate system. Using the transformation matrix obtained from module 2, we can obtain the coordinates (Xw, Yw, Zw) of the point in the world coordinate system.

[0185]

[0186] Where Zw is the height component, which is the desired vehicle height;

[0187] Vehicle exterior modification detection module:

[0188] Vehicle modifications are detected by a rear camera located above and behind the buffer parking space. A labeled dataset containing multiple exterior modification categories is constructed, and an end-to-end multi-class detection model is trained. After vehicle alignment, the rear camera captures an image of the area above and behind the vehicle. This image is then input into the target detection model for inference. For each detected exterior modification, the spatial geometric relationship between its detection box and the main detection box of the vehicle category is analyzed to determine whether it belongs to the current vehicle. The determination process includes: whether the modification detection box is located within or adjacent to the vehicle detection box, and whether it meets the preset position rationality rules. If so, the exterior modification is determined to belong to the current vehicle; otherwise, it is considered a false detection or a component of a neighboring vehicle and is removed.

[0189] The aforementioned automatic parking method for a multi-vision sensor collaborative automated parking garage, wherein the specific calculation process of the coordinates (X, Y, Z) in the front-end camera coordinate system is as follows:

[0190] Backprojecting the pixel coordinates of each camera into a ray, for Camera 1:

[0191]

[0192] In the Camera 1 coordinate system, this is ray 1 originating from the optical center of Camera 1:

[0193]

[0194] For Camera2:

[0195] Calculate the normalization direction:

[0196]

[0197] Transform it to the coordinate system of Camera1:

[0198]

[0199] It is the rotation matrix from Camera2 to Camera1;

[0200] In the camera1 coordinate system, ray 2 from Camera2 is:

[0201]

[0202] Triangulation to find points: Both rays are in the Camera1 coordinate system:

[0203] Ray 1:

[0204] Ray 2:

[0205] Find the point of shortest distance between these two rays:

[0206]

[0207] Solve Substituting these values ​​into the system will give you the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the Camera1 coordinate system.

[0208] The aforementioned method for automatic parking in a multi-sensor automated parking garage, wherein the fusion module includes:

[0209] Module 241 analyzes the multi-view image collaboratively using a keypoint detection model, a semantic segmentation model, and an object detection model to obtain the vehicle's body dimensions, including wheelbase, width, height, length, and front and rear overhangs, as AI measurement values. It constructs a vehicle database containing multiple vehicle models and their corresponding body dimensions, obtains the vehicle model using the object detection model, searches the vehicle database based on the vehicle model, and determines whether the vehicle is located in the vehicle database. If it is, it obtains the inferred values ​​including wheelbase, width, height, length, and front and rear overhangs, and executes module 242; otherwise, it executes module 244.

[0210] Module 242 determines whether the difference between the radar measurement value and the inferred value is within a first preset range. If so, the verification is successful, and the median of the current radar measurement value and the vehicle's historical measurement value is taken as the new inferred value and used for this vehicle storage. If the difference exceeds the first preset range, then module 243 is executed.

[0211] Module 243 determines whether the difference between the AI ​​measurement value and the predicted value is within a first preset range. If so, the verification is successful, and the median of the current AI measurement value and the vehicle's historical measurement value is taken as the new predicted value and used for this vehicle storage. If the difference exceeds the first preset range, then module 244 is executed.

[0212] Module 244 determines whether the difference between the current radar measurement value and the AI ​​measurement value is within the second preset range. If so, the verification is successful, and the median of the current radar measurement value and the vehicle's historical measurement values ​​is taken as the new inferred value and used for this vehicle storage. If the difference exceeds the second preset range, then module 45 is executed.

[0213] Module 245 determines whether the difference between the current radar measurement value and the AI ​​measurement value is within the third preset range. If so, the verification is successful, and the median of the current radar measurement value and the vehicle's historical measurement value is taken as the new inferred value and used for this parking, and the parking speed is reduced. If the difference exceeds the third preset range, the verification fails, and parking is not allowed.

[0214] The aforementioned automated parking method for multi-visual sensor collaborative automated parking garage, wherein the transfer module includes:

[0215] During the vehicle transport process, the vehicle transport device performs real-time anomaly detection. If no anomaly is found, the vehicle is transferred to a designated parking space in the stacked automated parking garage. If the target vehicle frame parking space is abnormal, module 41 is executed. If the vehicle transport device malfunctions, module 42 is executed. If the buffer parking space malfunctions, module 43 is executed. If the vehicle exceeds the RCV service range, module 44 is executed.

[0216] Module 41, reallocate chassis parking spaces. If there is a chassis parking space available for the vehicle, the control device will change the vehicle's parking space. If there is no chassis parking space, the vehicle will be assigned to an empty buffer parking space.

[0217] Module 42 sets the parking area where the vehicle conveyor is located to be out of service and sends a departure reminder to users currently queuing in the area, informing them that the area has been out of service;

[0218] Module 43 sets the currently faulty buffer parking space to suspended service, sends a vehicle relocation reminder to the owner of the waiting vehicle, and guides them to move their vehicle away from the faulty buffer parking space.

[0219] Module 44 determines whether the buffer parking space is occupied by a vehicle. If so, the vehicle transport device will move the vehicle to the buffer parking space. Otherwise, it will send a reminder to the owner of the vehicle occupying the buffer parking space to move the vehicle. After the vehicle occupies the buffer parking space, the vehicle transport device will move the vehicle to the buffer parking space.

[0220] The present invention also proposes a three-dimensional parking garage, including the aforementioned three-dimensional parking garage automatic parking system based on multi-visual sensor collaboration.

[0221] As can be seen from the above solutions, the advantages of the present invention are:

[0222] This invention employs a pure vision-based solution that utilizes multiple vision sensors in collaboration, overcoming the shortcomings of solutions such as lidar or TOF cameras: the high reflectivity of the laser point cloud results in poor imaging quality, significantly impacting accuracy and even making detection impossible.

[0223] Based on multi-view 2D visual perception, this invention achieves high-precision dynamic measurement of vehicle dimensions, automatic centering, multi-source verification and anomaly handling through the deep integration of image processing, deep learning and coordinate transformation algorithms. It solves the technical problems of high vehicle modeling accuracy requirements, large environmental interference and the need to support the detection of modified vehicles in three-dimensional parking garages.

[0224] This invention proposes a dynamic, high-precision vehicle body size measurement method based on multi-2D camera fusion and the collaborative operation of key point detection, target detection, semantic segmentation, and modification recognition models. This method uses a coordinate transformation algorithm to convert the pixel information output by the aforementioned models into actual physical dimensions, ultimately forming a fully automated three-dimensional parking garage retrieval system integrating automatic measurement, attitude adjustment, size verification, and anomaly handling. Its innovation lies in the systematic integration of software algorithms and AI models, achieving high-precision 3D measurement results with low-cost 2D cameras, thus solving the stringent requirements for vehicle modeling accuracy and reliability in stacked three-dimensional parking garages. Attached Figure Description

[0226] Figure 1 This is a flowchart of the vehicle storage process;

[0227] Figure 2 Layout diagram of the parking space cameras;

[0228] Figure 3 Side view of the arrangement of cameras in the buffer parking space;

[0229] Figure 4 Side view of the rear camera;

[0230] Figure 5 To align the top view of the vehicle in the center;

[0231] Figure 6 To align the top view of the vehicle in the center;

[0232] Figure 7 To align the top view of the other vehicle in front of the center;

[0233] Figure 8 To align the top view of the other vehicle after centering it;

[0234] Figure 9 Image of the right front camera;

[0235] Figure 10 This is a schematic diagram showing the position of the rearview mirror in a front view of a vehicle.

[0236] Figure 11 This is a schematic diagram of the rearview mirror position in a top view of another vehicle.

[0237] Figure 12 This is a schematic diagram showing the position of the rearview mirrors in a top-view view of a vehicle.

[0238] Figure 13 This is a schematic diagram showing the position of the highest point of the vehicle in the front view;

[0239] Figure 14 This is a schematic diagram showing the position of the highest point of the vehicle in the side view;

[0240] Figure 15 Flowchart for vehicle body size verification

[0241] Figure 16 Here is a flowchart for wheelbase verification;

[0242] Figure 17 This is a flowchart for checking parking anomalies. Detailed Implementation

[0244] In the operation of automated parking garages, vehicle handling typically relies on a pre-set vehicle model database. After identifying the vehicle model, the system retrieves key parameters such as wheelbase and track width from the database and calculates the accurate vehicle storage and retrieval location accordingly. Particularly in stacked automated parking garages, there are challenges in deploying vehicle measurement solutions, which are costly and whose accuracy is significantly affected by ambient lighting conditions.

[0245] To address this, this invention proposes an automated vehicle parking method for multi-sensor automated parking systems, comprising: 1) The vehicle enters a buffer parking space in the automated parking system; 2) By detecting a top-view image taken from above the parking space, the ground contact points of the front and rear tires are obtained; 3) The wheelbase is calculated based on the distance between the tire ground contact points; 4) By detecting a front view taken from the front of the parking space and a side view taken by a side vision sensor, the distance between the rearview mirrors is obtained as the vehicle width; 5) By detecting the side view, the vehicle outline is obtained to calculate the vehicle height from the highest point; 6) The rear view is detected using a multi-class detection model to obtain information on vehicle modifications; 7) A vehicle body size measurement result is constructed, including vehicle width, vehicle height, wheelbase, and modification details; 8) Based on the vehicle body size measurement result, a vehicle transport device is controlled to grip the vehicle wheels in the parking space. This invention achieves high-precision dynamic measurement of vehicle dimensions based on multi-view visual perception.

[0246] Currently, common methods for vehicle size measurement include: 3D camera measurement, binocular vision, and photoelectric sensor measurement. 3D camera measurement, an active 3D imaging technology, requires actively emitting a laser and receiving the laser signal reflected back from the target. However, the glossy paint and roof glass of passenger vehicles can cause the laser to not reflect back effectively to the camera sensor, resulting in a distorted point cloud with excessive noise, potentially leading to measurement failure or excessive error. Photoelectric sensor measurement uses a receiver to receive a light beam from a transmitter. When the beam is blocked by an object, it produces a changing light signal. When applied to vehicle size measurement, it requires a walking mechanism, has a long measurement time, poor reliability, and is susceptible to interference from passing personnel, resulting in incorrect results; its anti-interference capability is poor.

[0247] To address this issue, this invention further employs 2D camera technology. Multiple 2D cameras are strategically positioned at specific locations within the buffer parking space, capturing images from multiple perspectives during vehicle parking. By combining the correlation between these images and a neural network model, image information from different images can be extracted. Furthermore, by considering the positional relationships between these images, precise vehicle dimensions can be obtained. This solution is convenient to deploy, cost-effective, utilizes mature 2D camera technology, and offers affordable equipment. Installation and maintenance are also easier, effectively reducing overall investment costs. Simultaneously, the 2D camera solution exhibits excellent resistance to environmental interference, effectively avoiding problems commonly encountered with current 3D imaging technologies. It is unaffected by reflections or shadows on the vehicle surface. This high robustness ensures the system's reliability and accuracy, thereby meeting the precision requirements of multi-level parking garages for vehicle modeling and guaranteeing parking safety.

[0248] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0249] In a stacked automated parking garage, the vehicle handling and retrieval operations rely on accurate vehicle dimensions, including wheelbase, length, width, height, front overhang, rear overhang, and vehicle modifications. To meet the safe parking requirements of both enclosed and open parking environments, it is necessary to measure the vehicle's accurate dimensions in real time before it is placed into the stacked rack. Furthermore, the system must be able to automatically handle any abnormalities that occur during parking, ensuring parking efficiency and the reliability of the parking garage operation, unless equipment malfunctions and prevents further operation.

[0250] The measurement and processing procedures for vehicles before they are stored in the stackable chassis are as follows: Figure 1 As shown, the camera layout of the buffer parking space is as follows: Figure 2 , Figure 3 As shown in the diagram. The front and side cameras are responsible for vehicle height detection; the front camera, left front camera, and right front camera are responsible for vehicle width detection; the left front, right front, left rear, and right rear cameras are responsible for vehicle centering and wheelbase detection; the left front and right front cameras are responsible for front overhang detection, and the left and right rear overhang cameras are responsible for rear overhang detection; the rear camera is responsible for vehicle modification detection.

[0251] The overall parking process is as follows:

[0252] Step 1: When the vehicle begins to enter the buffer parking space;

[0253] Step 2: Perform front and rear suspension checks during the vehicle's entry into position; it is important to note that checks are necessary during the process because it is difficult to accurately locate the front or rear of the vehicle after it has come to a complete stop due to the viewing angle. Checking during the entry process ensures that the camera can capture the moment when the front or rear of the vehicle is exactly below the camera, which facilitates accurate measurement.

[0254] Step 3: After the vehicle enters the parking space and the driver leaves, the vehicle's posture is automatically adjusted to achieve vehicle centering.

[0255] Step 4: After alignment is completed, the vehicle wheelbase, width, and height will be measured, and the vehicle modification status will be checked, including whether the vehicle has side ladders, tow hooks, luggage racks, or other modified facilities.

[0256] Step 5: After the vehicle posture adjustment is completed and all body dimensions are measured, the body dimensions will be verified, that is, the vehicle's ability to be parked in the garage will be verified.

[0257] Step 6: Once the vehicle body dimensions have passed verification, the equipment can be controlled to perform a parking operation. During this parking process, any parking anomalies will be detected and handled in real time.

[0258] Step 7: If there are no obstructive abnormalities, the vehicle transport device will clamp the wheels and store the vehicle in the designated parking space of the automated parking garage.

[0259] Step 8: If there is an obstructive anomaly, the vehicle storage will fail.

[0260] The following will provide a detailed introduction to each process.

[0261] Step 2, front and rear overhang detection, specifically includes:

[0262] Step 1: Calibrate the camera and obtain the camera intrinsic parameter matrix.

[0263] Step 2: Install the four cameras on the top left front, right front, right rear, and left rear of the parking space (buffer parking space), respectively, looking down at the vehicle body, ensuring the camera's optical axis is perpendicular to the ground. See the installation diagram below. Figure 2 , Figure 2 The left and right front attitude cameras are responsible for front suspension detection, while the left and right rear suspension cameras are responsible for rear suspension detection.

[0264] Step 3: Establish a world coordinate system on the parking space, as follows: Figure 2 Obtain the transformation matrix between the camera coordinate system and the world coordinate system, i.e., the camera extrinsic parameters, and mark the parking space edge lines on the image after correcting lens distortion. The left front camera marks the front and left edge lines, the right front camera marks the front and right edge lines, the left rear camera marks the rear and left edge lines, and the right rear camera marks the rear and right edge lines.

[0265] Step 4: As the vehicle drives into the parking space, the camera captures images at a rate of 20 frames per second. Figure 4 As shown, point a is the contact point between the wheel and the ground, point B is the vertical projection of the rear of the vehicle onto the ground, and point b is the projection of the rear of the vehicle onto the ground as captured by the cameras (the left and right rear-mounted cameras in Figure 2). Points B and b will coincide only when the rear of the vehicle is directly below the camera; this is the optimal time for measurement.

[0266] Step 5: Correct the footage captured in step 4, such as... Figure 5 , Figure 7 The trained keypoint detection model is used to detect targets in the corrected image. The keypoint to be detected is the contact point 'a' between the wheel and the ground. If point 'a' is detected, the distance between point 'a' and the marked edge line in the world coordinate system is calculated. Specifically, the foot point 'a' of point 'a' on the edge line in the image coordinate system is calculated, and the two points are converted to coordinates in the world coordinate system as follows:

[0267] First, this invention defines the following variables and matrices:

[0268] K: Camera intrinsic parameter matrix

[0269] R: Rotation matrix, representing the rotation from the world coordinate system to the camera coordinate system.

[0270] T: Translation vector, representing the translation from the world coordinate system to the camera coordinate system.

[0271] (x,y): pixel coordinates on the image

[0272] Z: The known depth value, representing the Z-axis coordinate of the point in the world coordinate system. Since the world coordinate system is established on the ground plane when calibrating the extrinsic parameters, the Z-axis coordinate of all points on the ground plane, including points a and b mentioned above, is 0.

[0273] p: Homogeneous pixel coordinate vector [xy1]T

[0274] pnorm: Normalized camera coordinate vector

[0275] pworld: World coordinate vector

[0276] First, normalized camera coordinates are calculated: pixel coordinates are converted to normalized camera coordinates using the inverse K−1 of the intrinsic parameter matrix. Since the depth is known, this invention requires scaling the results to obtain the correct normalized coordinates.

[0277]

[0278] Here, (K−1p)3 represents the third component of the vector K−1p (i.e., the w component of the homogeneous coordinates, used for normalization).

[0279] Calculate world coordinates: Use the transpose RT of the rotation matrix and the translation vector T to convert the normalized camera coordinates to world coordinates.

[0280]

[0281] Combine the above steps:

[0282]

[0283] Finally, this invention can extract the X, Y, and Z components of the world coordinate system from pworld:

[0284]

[0285] Where a(x1,y1,z1) and a'(x2,y2,z2) calculate the distance between two points in the world coordinate system.

[0286]

[0287] Thus, this invention obtains the ground contact points a1 (left front wheel), a2 (right front wheel), a3 (left rear wheel), and a4 (right rear wheel), and their distances from the edge line of the parking space. Since the dimensions of the parking space are known (let's say the length is L), the vehicle's tilt angle can be obtained from the distances of the wheel contact points on the same side of the wheel from the edge line. Taking the left wheel as an example, assuming the distances of the left front wheel contact point a1 from the front edge line of the parking space are d1f and d1l, and the distances of the left rear wheel contact point a3 from the rear edge line of the parking space are d2b and d2l, then the vehicle tilt angle theta = atan((d1l-d2l) / (L-d1f-d2b)).

[0288] Step 6: Rotate the image according to the vehicle's tilt angle so that the vehicle body is not tilted in the rotated image, such as... Figure 6 Figure 8 The trained vehicle body target detection model is used to detect and obtain the detection bounding box of the vehicle body. Taking the right rear overhang detection camera as an example, ... Figure 6 As shown, the distance *dist* of the lower right corner point *b* of the vehicle body detection frame in the rotated image from the center line of the image in the height direction is recorded. Taking the right front overhang detection camera as an example, as... Figure 8 Then dist is the distance from the upper right corner of the vehicle body detection frame to the center line of the screen in the height direction;

[0289] Step 7 Rear Overhang Calculation: Taking the right rear overhang detection camera as an example again, the lower right corner point b of the vehicle body detection box in the rotated image obtained in step 6 is restored to point b in the unrotated image according to the theta angle, as shown in Figure 4. The distance between point b and the wheel ground contact point a in the world coordinate system (rear overhang value lb2) is calculated. The calculation method is similar to step 5. The rear overhang value lb2 detected by the right rear overhang camera is calculated. The rear overhang value lb1 detected by the left rear overhang camera is calculated in the same way. The distance dist calculated in step 6 and the distance between the wheel ground contact point and the left and right edge lines calculated in step 5 are associated with the obtained rear overhang value and recorded. The detection results of the left and right cameras are recorded separately.

[0290] Step 8: Front Overhang Calculation: Taking the right front overhang camera as an example, the upper right corner point b of the vehicle body detection box in the rotated image obtained in step 6 is restored to point b in the unrotated image according to the theta angle, such as... Figure 7 Calculate the distance between point b and wheel ground contact point a in the world coordinate system (front overhang value lf2). The calculation method is similar to step 5. Calculate the front overhang value lf2 detected by the right front overhang camera. Calculate the front overhang value lf1 detected by the left front overhang camera in the same way. Correlate the distance dist calculated in step 6 and the distance between the wheel ground contact point and the left and right edge lines calculated in step 5 with the obtained front overhang value and record them. Record the detection results of the left and right cameras respectively.

[0291] Step 9: Repeat steps 4 to 8 until dist exceeds the set threshold, or the number of calculated frames exceeds the set number of times. The left and right front and rear cameras will be checked separately. Repeating the detection of each frame during the driving process ensures that the camera can capture the moment when the front or rear of the vehicle is directly below the license plate camera. This repeated calculation involves performing 4 to 8 steps on each frame. The results of the calculation for each frame will be filtered later.

[0292] Step 10 compares the dist values ​​acquired by the four cameras in step 6, and takes the minimum value of each dist value as the final front and back overhang values.

[0293] Step 11: Determining the Front Overhang Value. If step 10 yields the calculated values ​​Lf1 (left front overhang camera) and Lf2 (right front overhang camera), determine the distances of each value to the left and right edge lines (left edge line for the left front overhang camera, right edge line for the right front overhang camera). Since the wheel appears narrower and longer from the camera's perspective when it's too close to the parking space edge, the accuracy of the wheel-to-ground contact point decreases. Set an appropriate threshold. If the distance between the wheel-to-ground contact point and the edge line corresponding to either Lf1 or Lf2 is less than the threshold, discard that value and use the other as the final front overhang value. If both are greater than the threshold, take the average of the two values. If step 10 only yields one front overhang value and the other is not detected, use the previous front overhang value as the final front overhang value.

[0294] Step 12: Determine the rear suspension value. If the calculated values ​​Lb1 and Lb2 from the left and right rear suspension cameras are obtained in step 10, then determine the distances of the two values ​​to the left and right edge lines (left edge line for the left rear suspension camera, right edge line for the right rear suspension camera). Similar to the front suspension, if the distance between the wheel contact point and the edge line corresponding to one of Lb1 and Lb2 is less than a threshold, then discard this value and take the other value as the final rear suspension value. If both are greater than the threshold, then take the average of the two. If only one rear suspension value is obtained in step 10 and the other is not detected successfully, then take the latter rear suspension value as the final rear suspension value.

[0295] b. Adjust vehicle attitude and wheelbase detection

[0296] Step 1: Camera installation diagram as shown Figure 2 It uses a total of 4 cameras, namely the front left, front right, rear left, and rear right cameras. The front left and front right cameras are shared with the front suspension detection.

[0297] Step 2: Calibrate each of the four cameras and obtain their intrinsic parameters.

[0298] Step 3: Establish a world coordinate system on the parking space, as follows: Figure 2 Obtain the transformation matrix between the camera coordinate system and the world coordinate system, i.e., the camera extrinsic parameters, and mark the edge lines on the calibrated image: the front and left edge lines for the left front camera, the front and right edge lines for the right front camera, the rear and left edge lines for the left rear camera, and the rear and right edge lines for the right rear camera.

[0299] Step 4: After the vehicle is parked, acquire images from the four cameras. Correct the captured images using the intrinsic parameters calibrated in Step 2. For example, take the image from the right front camera. Figure 9 The trained keypoint detection model is used to detect targets in the corrected image. The keypoint to be detected is the contact point 'a' between the wheel and the ground. If point 'a' is detected, the distance between point 'a' and the marked edge line in the world coordinate system is calculated. Specifically, the foot point 'a' of point 'a' on the edge line in the image coordinate system is calculated, and the two points are converted to coordinates in the world coordinate system as follows:

[0300] First, this invention defines the following variables and matrices:

[0301] K: Camera intrinsic parameter matrix

[0302] R: Rotation matrix, representing the rotation from the world coordinate system to the camera coordinate system.

[0303] T: Translation vector, representing the translation from the world coordinate system to the camera coordinate system.

[0304] (x,y): pixel coordinates on the image

[0305] Z: The known depth value, representing the Z-axis coordinate of the point in the world coordinate system. Since the world coordinate system is established on the ground plane when calibrating the extrinsic parameters, the Z-axis coordinate of all points on the ground plane, including points a and b mentioned above, is 0.

[0306] p: Homogeneous pixel coordinate vector [x, y, 1] T

[0307] pnorm: Normalized camera coordinate vector

[0308] pworld: World coordinate vector

[0309] First, normalized camera coordinates are calculated: pixel coordinates are converted to normalized camera coordinates using the inverse K−1 of the intrinsic parameter matrix. Since the depth is known, this invention requires scaling the results to obtain the correct normalized coordinates.

[0310]

[0311] Here, (K−1p)3 represents the third component of the vector K−1p (i.e., the w component of the homogeneous coordinates, used for normalization).

[0312] Calculate world coordinates: Use the transpose RT of the rotation matrix and the translation vector T to convert the normalized camera coordinates to world coordinates.

[0313]

[0314] Combine the above steps:

[0315]

[0316] Finally, this invention can extract the X, Y, and Z components of the world coordinate system from pworld:

[0317]

[0318] Where a(x1,y1,z1) and a'(x2,y2,z2) calculate the distance between two points in the world coordinate system.

[0319]

[0320] Thus, this invention obtains the ground contact points a1 (left front wheel), a2 (right front wheel), a3 (left rear wheel), and a4 (right rear wheel), and their distances from the edge lines of the parking space. Assuming the calculated distances from the left front wheel contact point a1 to the front edge line of the parking space are d1f and d1l; the distances from the right front wheel contact point a2 to the front edge line of the parking space are d2f and d2r; the distances from the left rear wheel contact point a3 to the rear edge line of the parking space are d3b and d3l; and the distances from the right rear wheel contact point a4 to the rear edge line of the parking space are d4b and d4r, then the distances the front wheels should move left and right when aligned are (d1l-d2r) / 2, and the distances the rear wheels should move left and right when aligned are (d3l-d4r) / 2. Given the length of the parking space as l, the wheelbase can be obtained from the distances of the left front wheel and the left rear wheel from the edge of the parking space using trigonometric functions.

[0321]

[0322] The wheelbase is obtained from the distances of the right front wheel and the right rear wheel from the edge line of the parking space.

[0323]

[0324] Take the average of the two wheelbase values ​​as the final wheelbase value: axle = (axle1 + axle2) / 2

[0325] Step 5: The centering mechanism moves the belts corresponding to the left and right front wheels according to the distance the front wheels should move left and right as calculated in Step 4, and moves the belts corresponding to the left and right rear wheels according to the distance the rear wheels should move left and right.

[0326] Step 6 repeats steps 4 and 5 until the calculated lateral movement distance of the front and rear wheels is less than the set threshold. At this point, the vehicle is considered to be basically centered, and the wheelbase of the last calculation is taken as the wheelbase of this vehicle.

[0327] c. Vehicle width inspection

[0328] Step 1 Camera Setup (See) Figure 2 The vehicle width detection uses a left front camera, a right front camera, and a front-facing camera. The left and right front cameras are positioned vertically overhead to capture images of the vehicle, while the front-facing camera is installed in front of the parking space to capture images of the front of the vehicle horizontally.

[0329] Step 2 involves calibrating the three cameras to obtain their intrinsic parameters K1, K2, K3 and distortion coefficients D1, D2, D3.

[0330] Step 3: After vehicle alignment is completed, distortion correction is performed on the images captured by each camera using the pre-calibrated distortion coefficients D1, D2, and D3, and the corrected images are output.

[0331] Step 4: Using a keypoint detection model, the rearview mirrors are detected in the image after the front-end camera has been calibrated. The pixel coordinates (u1, v1) and (u1', v1') of the outermost points of the two rearview mirrors in the image are obtained respectively. The image is shown below. Figure 10 The rearview mirror is detected using a key point detection model on the image after correction by the left front camera, and the pixel coordinates (u2, v2) of the outermost point of the rearview mirror in the image are obtained, such as... Figure 11 The rearview mirror was detected using a keypoint detection model on the corrected image from the right front camera. The pixel coordinates (u2', v2') of the outermost point of the rearview mirror in the image were obtained. Figure 12 .

[0332] Step 5: The outermost point P of the left rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1, v1) and (u2, v2). The outermost point P' of the right rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1', v1') and (u2', v2'). Find the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the coordinate system of the front camera. The front camera, left front camera, and right front camera are Camera1, Camera2, and Camera3, respectively.

[0333] Backprojecting the pixel coordinates of each camera into a ray, for Camera 1:

[0334]

[0335] This is a direction vector originating from the optical center of Camera 1 (in the Camera 1 coordinate system), where the spatial point satisfies:

[0336]

[0337] For Camera2:

[0338] First, calculate the normalization direction:

[0339]

[0340] Then transform it to the coordinate system of Camera1:

[0341]

[0342] It is the rotation matrix from Camera2 to Camera1.

[0343] Meanwhile, the position of the optical center of Camera2 in the coordinate system of Camera1 is...

[0344] Therefore, in the camera1 coordinate system, the ray from Camera2 is:

[0345]

[0346] Triangulation to find points: Now there are two rays (both in the Camera1 coordinate system):

[0347] Ray 1:

[0348] Ray 2:

[0349] Find the point of shortest distance between these two rays, i.e., minimize:

[0350]

[0351] Solve Later Substituting into the formula for ray 1, we can obtain the coordinates of P in the Camera1 coordinate system.

[0352] The same principle can be used to determine the coordinates (X', Y', Z') of the outermost point P' of the right rearview mirror in Camera1, and then calculate the spatial distance between the two points.

[0353]

[0354] d represents the width of the vehicle.

[0355] d. Vehicle height detection

[0356] Step 1 Camera Setup Reference Figure 2 , Figure 3 The vehicle height detection system uses a front-end camera and side cameras. The side cameras are installed to the side of the parking space to capture a horizontal view of the side of the vehicle, while the front-end camera is installed in front of the parking space to capture a horizontal view of the front of the vehicle.

[0357] Step 2: Calibrate the two cameras separately to obtain the camera's intrinsic parameters K1, K2, and distortion coefficients D1, D2.

[0358] Parking space establishes world coordinate system X w O w Y w The extrinsic parameters of the front-end camera are calibrated, namely the transformation matrix R,t between its camera coordinate system and this world coordinate system, which represents that the transformation matrix includes the rotation matrix R and the translation matrix t.

[0359] Step 3: After the vehicle is aligned, the optical axes of the side and side cameras are aligned and parallel to the optical axis of the front camera, as shown below. Figure 2 At this point, the two cameras take pictures respectively. Using the pre-calibrated distortion coefficients D1 and D2, the distortion of the images taken by the side camera and the front camera is corrected, and the corrected images are output.

[0360] Step 4: Using a deep learning semantic segmentation model, the vehicle and background are segmented from the corrected image from the side camera. The vehicle outline is extracted, and a search is performed on the outline point pairs, such as... Figure 13 and 14 The pixel coordinates (u1, v1) of the highest point on the side of the vehicle in the image are obtained from the side view. The vehicle and background are segmented using a deep learning semantic segmentation model on the image after correction by the front camera, the vehicle outline is extracted, and the outline point pairs are searched to obtain the pixel coordinates (u2, v2) of the highest point on the side of the vehicle in the image from the front view.

[0361] Step 5: The same spatial point P has corresponding pixel observation points in both cameras: (u1, v1) and (u2, v2). Find the coordinates (X, Y, Z) of point P in the front-end camera coordinate system, referring to step 5 of the vehicle width detection method. Using the transformation matrix obtained in step 2, the coordinates (Xw, Yw, Zw) of the point in the world coordinate system are obtained as follows:

[0362]

[0363] Where Zw is the height component, which is the desired vehicle height.

[0364] e. Vehicle modification inspection

[0365] Vehicle modifications are detected using a rear camera. First, a multi-class object detection model is trained, constructing a labeled dataset containing five categories: roof rack, tow hook, side bag, side ladder, and vehicle (i.e., the main body of the vehicle). An end-to-end multi-class detection model is then trained. During training, data augmentation strategies (random cropping, rotation, brightness adjustment, Mosaic enhancement, etc.) are employed to improve the model's robustness to different lighting conditions, angles, and occlusions. After vehicle centering, the rear camera captures an image of the upper rear of the vehicle. This image is input into the object detection model for inference. For each detected modification category (roof rack, tow hook, side bag, side ladder), the spatial geometric relationship between its bounding box and the main bounding box of the "vehicle" category is analyzed to determine whether it belongs to the current vehicle. The judgment logic is based on whether the modified part detection frame is mostly located inside or adjacent to the vehicle detection frame and the reasonableness of the relative position: for example, the tow hook should be located below the vehicle detection frame, the luggage rack should be located above the vehicle detection frame, and the side ladder / side backpack should be located on the side of the vehicle detection frame; the size of the modified part should be within a reasonable range compared with the size of the vehicle body. If the above spatial logic relationship is met, the modified part is determined to belong to the current vehicle; otherwise, it is considered a false detection or a part of an adjacent vehicle and is rejected.

[0366] f. Vehicle dimensions verification

[0367] Vehicle body size verification mainly refers to verifying the vehicle's wheelbase, length, width, height, front and rear overhangs, and any modifications. The verification process is as follows: Figure 15 As shown. Prioritize verifying the accuracy of the wheelbase data; only proceed with verifying other vehicle body dimensions if the verification passes.

[0368] The wheelbase measurement data comes from two testing methods: AI and radar. The specific verification process is as follows: Figure 16 As shown.

[0369] Step 1: Determine if the vehicle has a predicted value. If it does, proceed to Step 2; otherwise, proceed to Step 4.

[0370] Step 2: Determine if the difference between the radar measurement value and the estimated value is within 20mm. If so, the verification is successful. Take the median of the current radar measurement value and the vehicle's historical measurement values ​​as the new estimated value and use it for this vehicle storage. If the difference exceeds 20mm, proceed to Step 3.

[0371] Step 3: Determine whether the difference between the AI ​​measurement value (visual algorithm measurement value) and the inferred value is within 20mm. If so, the verification is successful. Take the median of the current AI measurement value and the vehicle's historical measurement values ​​as the new inferred value and use it for this parking. If the difference exceeds 20mm, proceed to step 4.

[0372] Step 4: Determine whether the difference between the current radar measurement and the AI ​​measurement is within 40mm. If so, the verification is successful. Take the median of the current radar measurement and the vehicle's historical measurement as the new inferred value and use it for this vehicle storage. If the difference exceeds 40mm, proceed to step 5.

[0373] Step 5: Determine whether the difference between the current radar measurement and the AI ​​measurement is within 60mm. If so, the verification is successful. Take the median of the current radar measurement and the vehicle's historical measurement as the new inferred value and use it for parking this time. Low-speed parking is required. If the difference exceeds 60mm, the verification fails and parking is not allowed.

[0374] If the wheelbase data accuracy verification is successful, the vehicle body dimensions will be further verified. The specific limits in the rules depend on the specific dimensions of the garage. The verification rules are as follows, in mm:

[0375] 1. Wheelbase: 2300 ≤ wheelbase -60, wheelbase +60 ≤ 3250, wheelbase error: ±60mm

[0376] 2. 1350mm < vehicle height +|error| ≤ 2000mm (first / third floor), 1590mm (second floor), vehicle height error: ±50mm

[0377] 3. Vehicle width ±|error| ≤ 2160mm, vehicle width error: ±40mm

[0378] 4. Wheelbase + front overhang ≤ 4190mm, and front overhang + vehicle wheelbase + rear overhang ≤ 5400mm, rear overhang + |error| ≤ 1230mm

[0379] 5. The vehicle lacks a side ladder, backpack, or tow hook.

[0380] If all the above conditions are met, the verification is considered successful and the vehicle can be stored. Otherwise, the verification fails and the vehicle cannot be stored.

[0381] g. Vehicle storage and abnormal inspection and handling during the storage process

[0382] After the vehicle body dimensions are verified, the system will control the vehicle transport device to perform the vehicle storage operation. During the vehicle transport process, the vehicle transport device will detect any abnormalities in real time. If there are no abnormalities, the vehicle will be normally transported from the buffer parking space to the chassis parking space. If there are abnormalities, it needs to attempt self-recovery. If self-recovery is not possible, the vehicle will not be able to be stored in the garage and will need to wait for manual maintenance.

[0383] This section only describes the handling process after an exception occurs, such as... Figure 17 As shown.

[0384] The system first identifies the type of exception, and then performs targeted processing for each specific type.

[0385] The target vehicle frame and parking space are not suitable.

[0386] Step 1: Send an early warning notification to inform relevant personnel of the abnormal parking situation;

[0387] Step 2: The system reassigns a chassis parking space to the target vehicle. If there is a chassis parking space available for the vehicle, the control device will change the parking space for the vehicle. If there is no chassis parking space, proceed to Step 3.

[0388] Step 3: The system allocates a buffer parking space to the target vehicle. If there is a free buffer parking space available, the control equipment will move the vehicle to the buffer parking space. If there is no available buffer parking space, the process will be stuck and await manual maintenance.

[0389] Vehicle conveyor malfunction

[0390] Step 1: Send an early warning notification to inform relevant personnel of the abnormal parking situation;

[0391] Step 2: Set the parking area where the vehicle conveyor is located to be temporarily unavailable.

[0392] Step 3: Send departure reminders to users currently in the queue, informing them that service in the area has been suspended.

[0393] Step 4: Wait for manual maintenance.

[0394] Buffer parking space failure

[0395] Step 1: Send an early warning notification to inform relevant personnel of the abnormal parking situation;

[0396] Step 2: Set the currently faulty buffer parking space to suspended service to prevent it from being reassigned to other vehicles;

[0397] Step 3: Send a reminder to the owner of the vehicle to be moved, notifying them to move their vehicle;

[0398] Step 4: Wait for manual maintenance.

[0399] Beyond RCV service range

[0400] RCV is a device that grips the wheels and transports the vehicle to the vehicle transfer unit (ASV). For anomalies such as exceeding the RCV's service range, due to structural limitations of the equipment, the vehicle transfer unit can only move the vehicle out of its original buffer parking position.

[0401] Step 1: Send an early warning notification to inform relevant personnel of the abnormal parking situation;

[0402] Step 2: A buffer parking space needs to be assigned to the vehicle, and it must be the parking space where the vehicle was parked at that time. If the parking space is available at this time, the allocation can be carried out directly, and the vehicle moving device will be controlled to move the vehicle to the buffer parking space; otherwise, proceed to step 3.

[0403] Step 3: Send a reminder to the owner of the vehicle occupying the buffer parking space to move their car to free up the buffer parking space. Proceed to Step 4 after the vehicle has left.

[0404] Step 4: Once the buffer parking space is vacant, control the vehicle transport device to send the vehicle to the buffer parking space and notify the vehicle owner that the parking failed.

[0405] This invention autonomously and in real-time measures the dimensions of vehicles to be stored to create a vehicle body model. The modeling process utilizes mature 2D camera technology, which not only has low deployment costs but also high measurement accuracy. For scenarios with high requirements for vehicle dimensions, such as stacked automated parking systems, this invention can meet the vehicle storage needs of public spaces like hospitals and shopping malls. Furthermore, the invention features self-checking and self-recovery functions during vehicle storage, and its architecture supports continuous improvement in handling anomalies, thus sustainably enhancing the reliability of the parking system.

[0406] For stacked automated parking garages, the accuracy of vehicle dimensions is extremely important. In order to store vehicles smoothly in the garage, the vehicle model can be set for each vehicle in the garage system, that is, the vehicle size data is stored in the database in advance. This way, when a vehicle visits the garage, there is no need to perform vehicle modeling. The vehicle can be stored by directly querying the database. Therefore, this solution can be used as an alternative to vehicle modeling in the vehicle storage process.

Claims

1. A method for automatic parking in a multi-sensor automated parking garage, characterized in that, include: Step 1: The vehicle drives into the buffer parking space of the stacked multi-level parking garage. The buffer parking space is equipped with visual sensors on its four sides and top. Step 2: Using the multiple vision sensors, obtain the side view, top view, front view and rear view of the vehicle. Use the key point detection model to detect the top view taken from the angle above the parking space to obtain the ground contact points of the front and rear tires, and obtain the wheelbase based on the distance between the tire ground contact points. The front view taken from the front angle of the parking space and the side view taken from the side angle of the parking space are detected by a key point detection model, and the distance between the rearview mirrors is used as the vehicle width; the side view is detected by a deep learning semantic segmentation model to obtain the vehicle outline, and the vehicle height is calculated based on the high point of the vehicle outline. The vehicle modification status is obtained by detecting the rear view captured by the camera behind the buffer parking space using a multi-class detection model; the vehicle body size measurement results include vehicle width, vehicle height, wheelbase and modification status. Step 3: Based on the vehicle body size measurement results and the vehicle storage size limit of the stacked automated parking garage, determine whether the vehicle can be parked in the stacked automated parking garage. If yes, proceed to step 4; otherwise, notify the vehicle to leave the stacked automated parking garage. Step 4: Control the vehicle transport device to grip the wheels of the vehicle located on the buffer parking space to transfer the vehicle to the designated parking space in the stacked automated parking garage.

2. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 1, characterized in that, Step 2 includes: A vehicle database containing multiple vehicle models and their corresponding body dimensions is constructed. The vehicle model is obtained through the target detection model. Based on the vehicle model, the vehicle database is searched to obtain the estimated values ​​of wheelbase, width and height, which are used as the measurement results of the vehicle body dimensions.

3. The automatic parking method for a multi-visual sensor collaborative three-dimensional parking garage as described in claim 1, characterized in that, Step 2 includes: Step 22: Use a visual sensor to detect the front and rear overhangs as the vehicle enters the buffer parking space, and obtain the front and rear overhang length values ​​of the vehicle. Step 23: After the vehicle stops in the buffer parking space and the driver leaves the buffer parking space, the vehicle angle is determined based on the image collected by the vision sensor, and the vehicle posture is automatically adjusted to center the vehicle in the buffer parking space. Step 24: After adjusting the vehicle's posture, measure the wheelbase, width, and height of the vehicle, and inspect the vehicle's exterior modifications to obtain the vehicle's dimensions, including wheelbase, width, height, and modifications.

4. The automatic parking method for a multi-visual sensor collaborative three-dimensional parking garage as described in claim 3, characterized in that, In step 2, the visual sensor is a camera, and the front and rear overhang detection includes: Step 221: Calibrate the camera and obtain the camera intrinsic parameter matrix; Step 222: Install cameras on the top left front, right front, right rear and left rear of the buffer parking space respectively to shoot the vehicle body from above, so that the camera optical axis is perpendicular to the ground. The left front and right front cameras are responsible for front suspension detection, and the left rear and right rear cameras are responsible for rear suspension detection. Step 223: Establish a world coordinate system on the buffer parking space, obtain the transformation matrix between the camera coordinate system and the world coordinate system, and mark the edge lines on the corrected image. The left front camera marks the front edge line and the left edge line, the right front camera marks the front edge line and the right edge line, the left rear camera marks the rear edge line and the left edge line, and the right rear camera marks the rear edge line and the right edge line. Step 224: The camera continuously captures the process of the vehicle entering the buffer parking space. The contact point between the wheel and the ground is point a, the vertical projection point of the rear of the vehicle on the ground is point B, and the camera captures the projection point of the rear of the vehicle on the ground as point b. When the rear of the vehicle is directly below the camera, point B coincides with point b, and step 225 is executed. Step 225: Perform target detection on the captured image using a key point detection model to detect the contact points a between the wheels and the ground, including: ground contact point a1 of the left front wheel, ground contact point a2 of the right front wheel, ground contact point a3 of the left rear wheel, and ground contact point a4 of the right rear wheel. Calculate the distance from the contact point to the edge line of the parking space. Obtain the tilt angle theta of the vehicle from the distance between the wheel contact point on the same side of the wheel and the edge line. Step 226: Rotate the image according to the tilt angle so that the car body is not tilted in the rotated image. Use the car body target detection model to detect and obtain the detection box of the car body. Record the distance dist of the corner point b of the car body detection box in the rotated image from the center line of the image in the height direction. Step 227: Restore corner point b back to point b in the unrotated image according to the theta angle, calculate the distance between point b and wheel ground contact point a in the world coordinate system, and obtain the rear overhang value lb2 detected by the right rear overhang camera and the rear overhang value lb1 detected by the left rear overhang camera. Step 228: Restore the upper right corner point b of the vehicle body detection box in the rotated image back to point b in the unrotated image according to the theta angle. Calculate the distance between point b and the wheel ground contact point a in the world coordinate system to obtain the front suspension value lf2 detected by the right front suspension camera and the front suspension value lf1 detected by the left front suspension camera. Record the distance dist calculated in step 226 and the distance between the wheel ground contact point and the left and right edge lines calculated in step 225 with the obtained front suspension values. Record the detection results of the left and right cameras respectively. Step 229: Repeat steps 224 to 228 until the distance dist is greater than the set threshold, or the number of calculations exceeds the set number of times. Step 2210: Compare the distances (dist) acquired by the four cameras in step 226, and take the front or back overhang value corresponding to the smallest distance (dist) value as the final front and back overhang values.

5. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 4, characterized in that, Step 2210 includes: In the front suspension value determination step, if step 2210 obtains the calculated value Lf1 from the left front suspension camera and the calculated value Lf2 from the right front suspension camera, then determine the distances of the two values ​​to the left and right edge lines respectively. If the distance between the wheel ground contact point corresponding to one of Lf1 and Lf2 and the edge line is less than the threshold, then discard this value and take the other value as the final front suspension value. If both are greater than the threshold, then take the average of the two. In the rear suspension value determination step, if the calculated values ​​Lb1 and Lb2 of the left rear suspension camera are obtained in step 2210, then the distances of the two values ​​to the left and right edge lines are determined respectively. If the distance between the wheel ground contact point corresponding to one of Lb1 and Lb2 and the edge line is less than the threshold, then this value is discarded and the other value is taken as the final rear suspension value; if both are greater than the threshold, then the average value of the two is taken.

6. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 2, characterized in that, Step 23 includes: Step 231: Install the cameras at the front left, front right, rear left, and rear right of the buffer parking space to obtain four cameras including a front left camera, a front right camera, a rear left camera, and a rear right camera. Step 232: Calibrate the four cameras separately and obtain the intrinsic parameters of each camera; Step 233: Establish a world coordinate system on the buffer parking space, obtain the transformation matrix between the camera coordinate system and the world coordinate system for each camera, use it as the camera extrinsic parameter, and mark the edge lines of the parking space on the corrected image. The left front camera marks the front edge line and left edge line of the parking space, the right front camera marks the front edge line and right edge line of the parking space, the left rear camera marks the rear edge line and left edge line of the parking space, and the right rear camera marks the rear edge line and right edge line of the parking space. Step 234: After the vehicle stops, each wheel is located on the belt of the centering mechanism. The images captured by the four cameras are obtained. The captured images are corrected using the corresponding intrinsic parameters. The corrected images are then used to perform target detection using a key point detection model to obtain the contact point a between each wheel and the ground. The distance between the contact point a and the marked parking space edge line in the world coordinate system is calculated. Step 235: Based on the distance of each wheel from the edge of the parking space in the world coordinate system, calculate the distance the front and rear wheels have moved, and control the centering mechanism to move the belt under each wheel to adjust the vehicle's posture and center the vehicle in the buffer parking space.

7. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 6, characterized in that, Step 234 calculates the distance between contact point a and the marked edge line of the parking space in the world coordinate system, specifically including: Calculate the perpendicular point a' of contact point a on the edge line of the parking space in the image coordinate system, and convert the two points to coordinates in the world coordinate system as follows: K is the camera's intrinsic parameter matrix; R is the rotation matrix, representing the rotation from the world coordinate system to the camera coordinate system; T is the translation vector, representing the translation from the world coordinate system to the camera coordinate system; (x, y) are the pixel coordinates on the image; Z is the known depth value, representing the Z-axis coordinate of the point in the world coordinate system; p is the homogeneous pixel coordinate vector [xy1]T; p norm p is the normalized camera coordinate vector; world World coordinate vector; Calculate normalized camera coordinates: Use the inverse K−1 of the intrinsic parameter matrix to convert the pixel coordinates to normalized camera coordinates; Where (K−1p)3 represents the third component of vector K−1p, namely the w component of homogeneous coordinates; Calculate world coordinates: Using the transpose of the rotation matrix The translation vector T converts the normalized camera coordinates to world coordinates. Combine the above steps: From p world Extract the X, Y, and Z components of the world coordinate system: Where a(x1,y1,z1) and a'(x2,y2,z2) are used to calculate the distance between two points in the world coordinate system: The ground contact points a1 (left front wheel), a2 (right front wheel), a3 (left rear wheel), and a4 (right rear wheel) are obtained. The distances from each contact point to the edge of the parking space are also calculated. Specifically, the distances from the left front wheel contact point a1 to the front edge of the parking space are d1f and d1l; the distances from the right front wheel contact point a2 to the front edge of the parking space are d2f and d2r; the distances from the left rear wheel contact point a3 to the rear edge of the parking space are d3b and d3l; and the distances from the right rear wheel contact point a4 to the rear edge of the parking space are d4b and d4r. The distance the front wheels should move when aligned is (d1l-d2r) / 2; the distance the rear wheels should move when aligned is (d3l-d4r) / 2.

8. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 3, characterized in that, Step 24 includes: Wheelbase measurement steps: The length of the buffer parking space is l. The wheelbase is obtained from the distances of the left front wheel and the left rear wheel from the edge line of the parking space. The wheelbase is calculated from the distances of the right front wheel and the right rear wheel from the edge of the parking space. Take the average of the two wheelbases as the final wheelbase value axle = (axle1 + axle2) / 2; Vehicle width measurement steps: The cameras were installed at the left front, right front, and front of the buffer parking space, respectively, resulting in three cameras: a left front camera, a right front camera, and a front camera. The left front camera and the right front camera were used to shoot the vehicle vertically from above, while the front camera was installed in front of the buffer parking space to shoot the front of the vehicle horizontally. The three cameras were calibrated to obtain the camera's intrinsic parameters K1, K2, K3 and distortion coefficients D1, D2, D3; The distortion coefficients D1, D2, and D3 are used to correct the distortion of the images captured by each camera, and the corrected images are output. The rearview mirrors of the vehicle are detected using a key point detection model on the image corrected by the front-end camera, and the pixel coordinates (u1, v1) and (u1', v1') of the outermost point of each rearview mirror in the image are obtained respectively. The rearview mirrors are detected using a key point detection model on the image corrected by the left front camera, and the pixel coordinates (u2, v2) of the outermost point of the rearview mirror in the image are obtained. The rearview mirrors are detected using a key point detection model on the image corrected by the right front camera, and the pixel coordinates (u2', v2') of the outermost point of the rearview mirror in the image are obtained. The outermost point P of the left rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1, v1) and (u2, v2). The outermost point P' of the right rearview mirror is at the front, and the corresponding pixel observation points in the right front camera are (u1', v1') and (u2', v2'). Find the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the coordinate system of the front camera. The front camera, the left front camera, and the right front camera are Camera1, Camera2, and Camera3, respectively. Find the coordinates (X', Y', Z') of the outermost point P' of the right rearview mirror in Camera1, and calculate the spatial distance d between the two points as the vehicle width: Vehicle height measurement steps: The vehicle height detection uses a front-end camera and a side camera. The side camera is installed on the side of the buffer parking space to capture the side of the vehicle horizontally. Establish a world coordinate system for the parking space and calibrate the extrinsic parameters of the front-end camera, namely the transformation matrix Rt between its camera coordinate system and the world coordinate system; After the vehicle comes to a complete stop and is centered, the optical axis of the side camera is aligned with the optical axis of the front camera, and the front camera and the side camera take pictures respectively. The distortion of the images taken by the side camera and the front camera is corrected, and the corrected image is output. The vehicle and background are segmented using a deep learning semantic segmentation model in the image corrected by the side camera. The vehicle outline is extracted, and the pixel coordinates (u1, v1) of the highest point on the side of the vehicle in the image are obtained by searching the outline point pairs. Similarly, the vehicle and background are segmented using a deep learning semantic segmentation model in the image corrected by the front camera. The vehicle outline is extracted, and the pixel coordinates (u2, v2) of the highest point on the side of the vehicle in the image are obtained by searching the outline point pairs. The same spatial point P has corresponding pixel observation points in both cameras: (u1, v1) and (u2, v2). To find the coordinates (X, Y, Z) of point P in the front-end camera coordinate system, refer to step 5 of the vehicle width detection method. Using the transformation matrix obtained in step 2, the coordinates (Xw, Yw, Zw) of the point in the world coordinate system are obtained as follows: Where Zw is the height component, which is the desired vehicle height; Inspection steps for vehicle exterior modifications: Vehicle modifications are detected by a rear camera located above and behind the buffer parking space. A labeled dataset containing multiple exterior modification categories is constructed, and an end-to-end multi-class detection model is trained. After vehicle alignment, the rear camera captures an image of the area above and behind the vehicle. This image is then input into the target detection model for inference. For each detected exterior modification, the spatial geometric relationship between its detection box and the main detection box of the vehicle category is analyzed to determine whether it belongs to the current vehicle. The determination process includes: whether the modification detection box is located within or adjacent to the vehicle detection box, and whether it meets the preset position rationality rules. If so, the exterior modification is determined to belong to the current vehicle; otherwise, it is considered a false detection or a component of a neighboring vehicle and is removed.

9. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 8, characterized in that, The specific calculation process for the coordinates (X, Y, Z) in the front-end camera coordinate system is as follows: Backprojecting the pixel coordinates of each camera into a ray, for Camera 1: In the Camera 1 coordinate system, this is ray 1 originating from the optical center of Camera 1: For Camera2: Calculate the normalization direction: Transform it to the coordinate system of Camera1: It is the rotation matrix from Camera2 to Camera1; In the camera1 coordinate system, ray 2 from Camera2 is: Triangulation to find points: Both rays are in the Camera1 coordinate system: Ray 1: Ray 2: Find the point of shortest distance between these two rays: Solve Substituting these values ​​into the system will give you the coordinates (X, Y, Z) of the outermost point P of the left rearview mirror in the Camera1 coordinate system.

10. The automatic parking method for a multi-visual sensor collaborative three-dimensional parking garage as described in claim 1 or 8, characterized in that, Step 2 includes: Step 241: The multi-view image is analyzed collaboratively by the key point detection model, semantic segmentation model, and object detection model to obtain the vehicle body size measurement results, including wheelbase, width, height, length, and front and rear overhang values, as AI measurement values; a vehicle database including multiple vehicle models and their corresponding body sizes is constructed; the vehicle model is obtained through the object detection model; the vehicle database is searched based on the vehicle model to determine whether the vehicle is located in the vehicle database; if so, the inferred values ​​including wheelbase, width, height, length, and front and rear overhang values ​​are obtained, and step 242 is executed; otherwise, step 244 is executed. Step 242: Determine whether the difference between the radar measurement value and the predicted value is within the first preset range. If yes, the verification is successful. Take the median of the radar measurement value and the vehicle's historical measurement value as the new predicted value and use it for this vehicle storage. If the difference exceeds the first preset range, proceed to step 243. Step 243: Determine whether the difference between the AI ​​measurement value and the predicted value is within the first preset range. If so, the verification is successful. Take the median of the current AI measurement value and the vehicle's historical measurement value as the new predicted value and use it for this parking. If the difference exceeds the first preset range, proceed to step 244. Step 244: Determine whether the difference between the current radar measurement value and the AI ​​measurement value is within the second preset range. If yes, the verification is successful. Take the median of the current radar measurement value and the vehicle's historical measurement values ​​as the new inferred value and use it for this vehicle storage. If the difference exceeds the second preset range, proceed to step 45. Step 245: Determine whether the difference between the current radar measurement value and the AI ​​measurement value is within the third preset range. If so, the verification is successful. Take the median of the current radar measurement value and the vehicle's historical measurement value as the new inferred value and use it for this parking, and reduce the parking speed. If the difference exceeds the third preset range, the verification fails and parking is not allowed.

11. The automatic parking method for a three-dimensional parking garage based on multi-vision sensor collaboration as described in claim 1, characterized in that, Step 4 includes: During the vehicle transport process, the vehicle transport device performs real-time anomaly detection. If no anomaly is found, the vehicle is transferred to a designated parking space in the stacked automated parking garage. If the target vehicle frame parking space is abnormal, step 41 is executed. If the vehicle transport device malfunctions, step 42 is executed. If the buffer parking space malfunctions, step 43 is executed. If the vehicle exceeds the RCV service range, step 44 is executed. Step 41: Reassign vehicle rack parking spaces. If there is a available rack parking space for the vehicle, the control device will change the vehicle's parking space. If there is no available rack parking space, the vehicle will be assigned to an empty buffer parking space. Step 42: Set the parking area where the vehicle conveyor is located to be temporarily suspended, and send a departure reminder to users currently queuing in the area, informing them that the area has been temporarily suspended. Step 43: Set the currently faulty buffer parking space to suspended service, send a vehicle relocation reminder to the owner of the vehicle waiting to be parked, and guide them to move their vehicle away from the faulty buffer parking space. Step 44: Determine whether the buffer parking space where the vehicle was parked is occupied by another vehicle. If so, the vehicle transport device will move the vehicle to the buffer parking space. Otherwise, send a reminder to the owner of the vehicle occupying the buffer parking space to move the vehicle. After the occupying vehicle leaves, the vehicle transport device will move the vehicle to the buffer parking space.

12. An automated parking system for a multi-visual sensor-based automated parking garage, characterized in that, include: The initial module involves vehicles entering buffer parking spaces in a stacked multi-level parking garage, where visual sensors are installed around and on top of the buffer parking spaces. The fusion module obtains the side view, top view and front view of the vehicle through multiple vision sensors. It detects the top view taken from the angle above the parking space through a key point detection model, obtains the ground contact points of the front and rear tires, and obtains the wheelbase based on the distance between the tire ground contact points. The distance between the rearview mirrors is obtained as the vehicle width by detecting the front view and the side view using a key point detection model; The side view is detected by a deep learning semantic segmentation model to obtain the vehicle outline, and the vehicle height is calculated based on the high point of the vehicle outline. The vehicle modification status is obtained by detecting the rear view captured by the camera behind the buffer parking space using a multi-category detection model; the vehicle body size measurement results include vehicle width, vehicle height, wheelbase and modification status. The judgment module determines whether the vehicle can be parked in the stacked automated parking garage based on the vehicle size measurement results and the parking size limit of the stacked automated parking garage. If yes, the transfer module is called; otherwise, the vehicle is notified to leave the stacked automated parking garage. The transfer module controls the vehicle transport device to grip the wheels of a vehicle located in the buffer parking space, so as to transfer the vehicle to a designated parking space in the stacked automated parking garage.

13. The automated parking system for a multi-visual sensor-based automated parking garage as described in claim 12, characterized in that, This fusion module is used for: The front and rear overhangs of the vehicle are detected by a visual sensor as the vehicle enters the buffer parking space, and the front and rear overhang lengths of the vehicle are obtained. Once the vehicle has come to a complete stop in the buffer parking space and the driver has left the buffer parking space, the angle of the vehicle is determined based on the image collected by the vision sensor, and the vehicle's posture is automatically adjusted to center the vehicle in the buffer parking space. After adjusting the vehicle's posture, wheelbase, width, and height are measured, and the vehicle's exterior modifications are inspected to obtain the vehicle's dimensional measurements, including wheelbase, width, height, and modifications.

14. A multi-level parking garage, characterized in that, Includes the automated parking system for multi-visual sensors based on collaborative multi-level parking garage as described in claim 12 or 13.

Images