Monocular visual perception enhancement method based on laser radar

Abstract

The invention discloses a monocular visual perception enhancement method based on a laser radar, and relates to the technical field of intelligent traffic. The method comprises the following steps: acquiring a vehicle video image through a monocular camera, acquiring three-dimensional point cloud data containing diffuse reflection calibration plate characteristics through a laser radar, establishing two-dimensional pixel and three-dimensional world coordinate mapping through internal reference calibration and joint calibration, generating a full-resolution lookup table, and realizing real-time acquisition of vehicle three-dimensional coordinates in combination with YOLO target detection. And parameter drift is corrected through motion trail verification and fixed reference feature iterative optimization. According to the invention, the problem that the existing single equipment is limited in positioning precision, economy and stability in a hub scene is effectively solved, and the method has extremely high practical use value.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a monocular vision perception enhancement method based on lidar. Background Technology

[0002] The rapid development of intelligent transportation is driving the upgrading of integrated transportation hubs towards intelligence and efficiency. Real-time and accurate vehicle positioning is the core support for realizing intelligent functions such as parking space management, traffic scheduling, and safety control within the hub. As key areas with dense traffic flow and complex scenarios, parking garages in integrated transportation hubs generally face problems such as variable lighting, frequent vehicle intersections, and significant occlusion, which places dual demands on the accuracy, stability, and economy of positioning technology, urgently requiring efficient positioning solutions adapted to this scenario.

[0003] Currently, vehicle positioning in this scenario mainly relies on a single LiDAR or a single monocular camera solution. While a single LiDAR can provide high-precision 3D point cloud data to meet positioning accuracy requirements, the high cost of equipment procurement and maintenance makes it difficult to deploy widely in large-scale hub scenarios. A single monocular camera has gained some application due to its low cost and convenient deployment, but its own technical limitations prevent it from acquiring depth information, making it difficult to accurately solve the vehicle's 3D coordinates and adapt to the actual positioning needs of complex scenarios. Neither of these two solutions can balance positioning accuracy and economy, becoming a key bottleneck restricting the intelligent implementation of parking garages in integrated transportation hubs. Summary of the Invention

[0004] The purpose of this invention is to provide a monocular vision perception enhancement method based on lidar to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a monocular vision perception enhancement method based on lidar, comprising the following steps: Step 100: Acquire vehicle video images of the parking lot scene of the integrated transportation hub using a monocular camera, and acquire three-dimensional point cloud data of the scene using a lidar. The three-dimensional point cloud data includes feature information of the diffuse reflection calibration plate. Step 200: Based on the vehicle video image and 3D point cloud data, perform intrinsic parameter calibration on the monocular camera and joint calibration of the camera and lidar to determine the optical parameters of the monocular camera and the 3D spatial relative position relationship between the monocular camera and the lidar deployment point. Step 300: Based on the optical parameters and the relative positional relationship in three-dimensional space, establish a transformation model from two-dimensional pixel coordinates to three-dimensional world coordinates and generate a full-resolution lookup table; Step 400: The YOLO object detection algorithm is used to extract the vehicle bounding box from the vehicle video image, and the vehicle's three-dimensional spatial coordinates are obtained by mapping through the full-resolution lookup table to achieve real-time vehicle positioning. Step 500: Based on the vehicle motion trajectory consistency verification and scene fixed reference features constructed from the vehicle's three-dimensional spatial coordinates, iteratively optimize the optical parameters and the relative positional relationship in three-dimensional space to correct parameter drift.

[0006] Preferably, step 100 includes: Step 110: Deploy the lidar in an unobstructed position behind the monocular camera, and deploy the diffuse reflection calibration plate in the integrated transportation hub parking garage scene; Step 120: Control the monocular camera and lidar to synchronously record multiple sets of calibration data with different poses using ROS-docker script, and change the pose of the diffuse reflection calibration plate multiple times during the recording process; Step 130: In the low-light, occluded, and multi-directional driving environment of the integrated transportation hub parking garage scenario, vehicle images are acquired using the monocular camera. After filtering the acquired vehicle images, the labelimg tool is used to annotate the filtered vehicle images to construct a training dataset.

[0007] Preferably, step 200 includes: Step 210: Extract clear frames of the calibration images from the calibration data, process the clear frames using MATLAB, and solve for the intrinsic parameter matrix and distortion coefficients of the monocular camera. The intrinsic parameter matrix and distortion coefficients together constitute the optical parameters. Step 220: Use CloudCompare to filter the three-dimensional point cloud data. Initially filter out irrelevant point clouds by limiting the coordinate range, and then accurately filter out point clouds related to the diffuse reflection calibration plate fixing device by calculating the minimum height of the z-axis, while retaining the effective point cloud of the diffuse reflection calibration plate. Step 230: Perform corner point extraction processing on the effective point cloud to obtain the three-dimensional corner coordinates of the diffuse reflection calibration plate; Step 240: Establish point-pair constraints between the three-dimensional corner coordinates and the pixel coordinates of the corresponding corner points in the calibration image; input the optical parameters to remove the distortion of the pixel coordinates and establish a camera imaging model; under the point-pair constraints, use the solvePnP algorithm to deduce the pose of the monocular camera by minimizing the reprojection error based on the perspective projection equation, and obtain the optimal rotation vector and translation vector; convert the rotation vector into a three-dimensional rotation matrix using the Rodrigues transform; merge the three-dimensional rotation matrix and the translation vector to form a homogeneous extrinsic transformation matrix; the extrinsic transformation matrix constitutes the relative positional relationship in three-dimensional space.

[0008] Preferably, the defined coordinate range is set according to the actual size of the integrated transportation hub parking garage scene, and the minimum height of the z-axis is determined by the actual installation height of the diffuse reflection calibration plate fixing device.

[0009] Preferably, step 230 includes: Step 231: Detect the number of valid point clouds. If the number of valid point clouds is less than a preset threshold, output an abnormal prompt message indicating insufficient point cloud quantity and terminate subsequent operations. If the number of valid point clouds reaches or exceeds the preset threshold, proceed to step 232. Step 232: Perform planar segmentation on the effective point cloud using the random sampling consensus algorithm to identify the inner point cloud that conforms to the planar features of the diffuse reflection calibration plate; Step 233: Reduce the dimensionality of the interior point cloud to a two-dimensional planar coordinate system using the principal component analysis algorithm, and map the interior point cloud to the optimal fitting plane; Step 234: The reduced interior point cloud is automatically clustered using a density-based spatial clustering algorithm, with a preset optimal clustering radius parameter. Step 235: Extract the extreme points distributed along the main direction of each cluster as feature boundary endpoints, fit the feature boundary endpoints using the minimum bounding rectangle algorithm to obtain the two-dimensional rectangular parameters of the diffuse reflection calibration plate, and map the two-dimensional rectangular parameters back to the original three-dimensional spatial coordinate system using the inverse transformation function of principal component analysis to obtain the three-dimensional corner coordinates.

[0010] Preferably, step 300 includes: Step 310: Read the intrinsic parameter matrix and distortion coefficients in the optical parameters. For the input image pixel coordinates, calculate the normalized coordinates of the image pixel coordinates according to the intrinsic parameter matrix, and then remap them back to the pixel coordinate system after distortion correction. Step 320: Expand the distortion-corrected pixel into homogeneous coordinate form, and transform the pixel in homogeneous coordinate form into the camera normalized coordinate system through the inverse matrix of the intrinsic parameter matrix to obtain the line-of-sight direction vector starting from the optical center of the monocular camera and passing through the pixel. Step 330: Using the three-dimensional rotation matrix in the relative positional relationship of the three-dimensional space, the viewing direction vector is transformed from the camera coordinate system to the world coordinate system. The viewing direction vector in the world coordinate system is described by the parametric equation as the world coordinate point being equal to the viewing direction vector multiplied by the scale parameter in the ray direction and the optical center coordinate of the monocular camera. The parametric equation is substituted into the ground plane equation to obtain the scale parameter. The three-dimensional world coordinates corresponding to the image pixel coordinate point are calculated based on the scale parameter. Step 340: Create a three-dimensional array structure based on the image resolution of the monocular camera. The dimensions of the three-dimensional array structure are the vertical pixel resolution, the horizontal pixel resolution, and the three-dimensional world coordinates of the corresponding pixels. Iterate through all pixels row by row and assign the three-dimensional world coordinates of each pixel to the corresponding position of the three-dimensional array structure according to the operations in steps 310 to 330. Generate and save a full-resolution lookup table. The full-resolution lookup table is saved in npy format.

[0011] Preferably, step 400 includes: Step 410: Select a lightweight YOLO series model and train the model based on the training dataset to obtain a vehicle detection model; Step 420: Process the vehicle video image using the vehicle detection model to output the vehicle's identification information, bounding box coordinates, and pixel coordinates of the bottom border of the bounding box. Step 430: Input the bounding box coordinates and the pixel coordinates of the bottom border of the bounding box into the full-resolution lookup table, and obtain the three-dimensional spatial coordinates of the vehicle through index mapping.

[0012] Preferably, step 500 includes: Step 510: Construct the vehicle motion trajectory based on the vehicle's three-dimensional spatial coordinates, and perform a consistency judgment on the vehicle motion trajectory using a kinematic model; if the vehicle's three-dimensional spatial coordinates deviate from the predicted value of the kinematic model for a consecutive preset number of frames by more than a preset deviation threshold, then continue to step 520; if the vehicle's three-dimensional spatial coordinates do not deviate from the predicted value of the kinematic model for a consecutive preset number of frames or the degree of deviation does not exceed the preset deviation threshold, then jump to step 420, and continue to process the vehicle video images acquired in real time by the monocular camera using a vehicle detection model; Step 520: Identify fixed reference features in the integrated transportation hub parking garage scene using the YOLO object detection algorithm, extract the current pixel coordinates of the fixed reference features, input the current pixel coordinates into the full-resolution lookup table to map and obtain the current three-dimensional coordinates, and store the true three-dimensional coordinates of the fixed reference features synchronously during the initial joint calibration stage; Step 530: Based on the true three-dimensional coordinates of the fixed reference feature, construct an objective function to minimize the reprojection error, and iteratively correct the optical parameters and the relative positional relationship in three-dimensional space through the objective function; Step 540: Substitute the optimized optical parameters and the relative positional relationship in three-dimensional space into the transformation model, and repeat the operations from step 310 to step 340 to generate a new full-resolution lookup table.

[0013] Preferably, the kinematic model includes equations for uniform linear motion and circular motion; the preset frame count is five frames; the preset deviation threshold is three centimeters; and the objective function for minimizing the reprojection error is... In the formula, The pixel coordinates of the current reference feature. The true three-dimensional coordinates of the reference feature. For the camera intrinsic parameter matrix, It is a three-dimensional rotation matrix. It is a translation vector. This is the projection function.

[0014] Preferably, the iterative optimization process triggered in step 500 needs to meet two triggering conditions: the first triggering condition is that the consistency deviation of the vehicle's motion trajectory continuously exceeds the preset deviation threshold, and the second triggering condition is that the number of effectively identified fixed reference features is not less than three, and the iterative optimization process is executed asynchronously.

[0015] Compared with the prior art, the beneficial effects of the present invention are: By integrating the advantages of LiDAR and monocular cameras, the spatial coordinate relationship between the two is established through a single calibration using a calibration board, compensating for the lack of depth information in monocular cameras and improving vehicle positioning accuracy. It avoids the high cost of using a single LiDAR, achieving rapid coordinate mapping through a full-resolution lookup table while also meeting real-time positioning requirements. Adapting to the complex scenarios of integrated transportation hubs, it solves the problem of point cloud loss caused by single LiDAR in complex lighting, dense vehicle traffic, and occlusion environments, improving model generalization ability. Through motion trajectory verification and iterative optimization of fixed reference features, parameter drift is corrected, ensuring long-term positioning stability. This effectively overcomes the limitations of single devices in positioning accuracy, economy, and stability in hub scenarios, demonstrating significant practical value. Attached Figure Description

[0016] Figure 1 The main flowchart of a monocular vision perception enhancement method based on lidar provided in an embodiment of the present invention; Figure 2 This is an optical schematic diagram of a monocular vision perception enhancement method based on lidar provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 and Figure 2 This invention provides a monocular vision perception enhancement method based on lidar. The method is applied to the dynamic localization and correction of vehicle data captured by a monocular camera using lidar throughout the entire process, including: Step 100: Acquire vehicle video images of the parking lot scene of the integrated transportation hub using a monocular camera, and acquire three-dimensional point cloud data of the scene using a lidar. The three-dimensional point cloud data includes feature information of the diffuse reflection calibration plate.

[0019] Specifically, the core purpose of step 100 is to collect two types of basic data: first, vehicle video images used for YOLO model training, which support subsequent vehicle recognition and bounding box extraction; and second, three-dimensional point cloud data containing diffuse calibration plate features, which support subsequent joint calibration and provide a data foundation for the entire technical solution.

[0020] Step 110: Deploy the lidar in an unobstructed position behind the monocular camera, and deploy the diffuse reflection calibration plate in the integrated transportation hub parking garage scene.

[0021] The monocular camera is a standard security surveillance camera, and its deployment location must ensure that the field of view covers the target monitoring area, and that there are no obstructions or reflective interference. Preferred locations are the side of a garage pillar or ceiling-mounted positions. The lidar is deployed in an unobstructed area behind the monocular camera, ensuring that the overlap rate between the lidar and the camera's field of view is no less than 80%. This avoids the lidar itself obstructing the camera's imaging, and also ensures that both can simultaneously capture the characteristic information of the diffuse reflection calibration board during the calibration process.

[0022] The diffuse reflection calibration board is the core reference for joint calibration. Its surface has uniform diffuse reflection characteristics, ensuring that it can still be clearly captured by LiDAR and cameras under complex lighting conditions. During deployment, the calibration board is first fixed on an adjustable pose bracket. The point cloud data and image data of the calibration board are observed through a computing terminal (equipped with monitoring software) connected to the camera and LiDAR to check whether they are completely recorded. This eliminates feature loss problems caused by occlusion or excessive distance. Finally, a location with no missing data and clear features is selected as the formal calibration data acquisition point.

[0023] In one possible implementation, the installation height of the monocular camera can be set to 2.5-3.5 meters, and the horizontal distance between the lidar and the camera should not exceed 1 meter to ensure that the overlap of their fields of view meets the calibration requirements; the size of the diffuse reflection calibration plate can be selected as 50cm×50cm or 100cm×100cm, depending on the size of the parking lot, to ensure that the lidar can completely collect its point cloud data within a range of 5-15 meters.

[0024] Step 120: Control the monocular camera and lidar to synchronously record multiple sets of calibration data with different poses using the ROS-docker script, and change the pose of the diffuse reflection calibration plate multiple times during the recording process.

[0025] The core of calibration data acquisition is obtaining multiple sets of synchronized "image-point cloud" data in different poses to provide sufficient geometric constraints for joint calibration. In practice, the ROS (Robot Operating System) on the computing terminal is started, and a pre-written ROS-docker script is used to synchronize the monocular camera and LiDAR. The ROS-docker script unifies the timestamps of both, ensuring that each frame of image data is completely synchronized with the corresponding point cloud data in time, avoiding calibration errors caused by time differences.

[0026] During recording, the pose of the diffuse reflection calibration board was manually adjusted, including translation and rotation. After each adjustment, the calibration board was kept stable for 1-2 seconds to ensure that the sensor could fully acquire the feature information under that pose. Translation included movement in the forward, backward, left, right, and up / down directions, while rotation included rotation around the x, y, and z axes. The recorded calibration data format was as follows: JPG image frames output by the monocular camera and PCD point cloud frames output by the LiDAR. Each set of data contained one image frame and one point cloud frame.

[0027] In one possible implementation, 20-50 sets of data with different poses are collected, with the translation range controlled within ±1 meter and the rotation angle controlled within ±30° to ensure full coverage of the calibration plate's angular features and provide sufficient constraints for subsequent calibration.

[0028] Step 130: In the low-light, occluded, and multi-directional driving environment of the integrated transportation hub parking garage scenario, vehicle images are acquired using the monocular camera. After filtering the acquired vehicle images, the labelimg tool is used to annotate the filtered vehicle images to construct a training dataset.

[0029] The training dataset must be collected based on the typical environment of a comprehensive transportation hub parking garage to ensure the diversity and representativeness of the dataset, so that the trained YOLO model can adapt to the vehicle recognition needs in complex scenarios. Low-light environments refer to areas in underground parking garages without natural light; occlusion environments include vehicles partially obscured by pillars or densely packed vehicles; and multi-directional driving environments include different driving states such as vehicles going straight, turning, and reversing.

[0030] During the data acquisition process, a monocular camera continuously captured images at a fixed frame rate for at least one hour to ensure the dataset included images of various vehicle types, such as sedans, SUVs, and vans. After acquisition, the original images were filtered to remove images where the ground was completely occluded, vehicle features were blurred, or there were no complete vehicle bodies, ultimately retaining 500-1000 valid images.

[0031] The labelimg tool was used to annotate the selected valid images, with the annotations being the bounding boxes of the vehicles. The annotation rule was that the bounding boxes should fit the contact points between the vehicles and the ground to ensure that subsequent model training could accurately output the pixel coordinates of the bottom of the vehicles. The annotation format adopted was VOC format (XML file), and each XML file contained information such as the bounding box coordinates of the vehicles in the corresponding image and the vehicle category. Finally, the training dataset for YOLO model training was constructed.

[0032] In this embodiment, the reasonable deployment of equipment and accurate data collection provide a high-quality data foundation for subsequent model training and joint calibration, avoiding problems such as low model recognition accuracy and large calibration errors caused by missing data or poor data quality.

[0033] Step 200: Based on the vehicle video image and 3D point cloud data, perform intrinsic parameter calibration on the monocular camera and joint calibration of the camera and lidar to determine the optical parameters of the monocular camera and the 3D spatial relative position relationship between the monocular camera and the lidar deployment point.

[0034] Specifically, the core purpose of step 200 is to determine two key parameters using the calibration data collected in step 100: one is the optical parameters of the monocular camera, which is used to correct the distortion error of the camera imaging; the other is the three-dimensional spatial relative position relationship between the monocular camera and the lidar deployment point, which is used to establish a mapping bridge between two-dimensional pixel coordinates and three-dimensional world coordinates.

[0035] Step 210: Extract clear frames of the calibration images from the calibration data, process the clear frames using MATLAB, and solve for the intrinsic parameter matrix and distortion coefficients of the monocular camera. The intrinsic parameter matrix and distortion coefficients together constitute the optical parameters.

[0036] The camera's optical parameters include an intrinsic parameter matrix and distortion coefficients. The intrinsic parameter matrix contains parameters such as focal length and principal point coordinates, describing the camera's imaging geometry. The distortion coefficients include radial distortion coefficients and tangential distortion coefficients, used to correct imaging distortion caused by lens optical defects. Radial distortion manifests as barrel distortion or pincushion distortion, while tangential distortion is caused by lens manufacturing errors; both affect the accuracy of pixel coordinates.

[0037] In practice, from the calibration data collected in step 120, extract 30-50 clear image frames with complete calibration board features to ensure coverage of different poses of the calibration board, and import them into the camera calibration toolbox of MATLAB software. In the toolbox, select the "monocular camera calibration" mode and input the pixel resolution of the image and the actual size parameters of the diffuse reflection calibration board.

[0038] The MATLAB toolbox automatically identifies the corner points of the calibration board in the image. It iteratively calculates the intrinsic parameter matrix and distortion coefficients using the corner point coordinates of multiple image frames, and outputs the calibration error. If the calibration error is greater than 1 pixel, the image frames are re-selected for calibration until the error meets the requirements. Finally, the obtained optical parameters are saved as a JSON file for subsequent distortion correction and joint calibration.

[0039] In one possible implementation, the pixel resolution of the monocular camera can be selected as 1920×1080 or 2560×1440, the diffuse reflection calibration plate is 10×10 grids, each grid has a side length of 5cm or 10cm, and the calibration error needs to be controlled within 0.5 pixels to ensure the accuracy of optical parameters.

[0040] Step 220: Use CloudCompare to filter the three-dimensional point cloud data. First, filter out irrelevant point clouds by limiting the coordinate range. Then, accurately filter out point clouds related to the diffuse reflection calibration plate fixing device by calculating the minimum height of the z-axis, and retain the effective point cloud of the diffuse reflection calibration plate.

[0041] The core of point cloud data filtering is to remove irrelevant and interfering points from the original point cloud, retaining only the valid point cloud of the diffuse calibration plate, thus avoiding the impact of irrelevant data on subsequent corner point extraction and calibration accuracy. Irrelevant points include background points such as garage walls, ground, and columns, while interfering points include the point cloud of the calibration plate's fixing bracket.

[0042] In practice, the point cloud frame corresponding to the selected image frame in step 210 is extracted, and imported into CloudCompare software for filtering by timestamp matching. Initial filtering is performed first: Based on the actual size of the parking garage, a limited coordinate range for the x, y, and z axes is set, and irrelevant point clouds outside this range are filtered out, retaining only the point cloud data of the area where the calibration board is located.

[0043] Next, precise filtering is performed: by calculating the minimum z-axis height of the point cloud, which is determined by the actual installation height of the diffuse reflection calibration plate fixing equipment, point clouds with z-axis coordinates below this value are filtered out, ultimately retaining only the valid point cloud of the diffuse reflection calibration plate. The setting of the limited coordinate range needs to be combined with the actual size of the specific parking garage to avoid the background point cloud not being completely filtered out due to the range being too large, or the calibration plate point cloud being mistakenly filtered out due to the range being too small.

[0044] In one possible implementation, the x-axis is defined as 0-10m, the y-axis as 0-10m, and the z-axis as 0-3m, which can be adjusted according to the actual size of the parking lot. The installation height of the diffuse reflection calibration plate fixing device is 0.5m, so the minimum height of the z-axis is set to 0.5m to ensure accurate filtering of the bracket point cloud.

[0045] Step 230: Perform corner point extraction processing on the effective point cloud to obtain the three-dimensional corner coordinates of the diffuse reflection calibration plate.

[0046] Specifically, point cloud corner point extraction is a key step in obtaining the three-dimensional corner coordinates of the diffuse reflection calibration board. Through a series of algorithmic operations, the coordinates of the four corner points of the calibration board are extracted from the effective point cloud, providing three-dimensional spatial constraint points for joint calibration.

[0047] Step 231: Detect the number of valid point clouds. If the number of valid point clouds is less than a preset threshold, output an abnormal prompt message indicating insufficient point cloud quantity and terminate subsequent operations. If the number of valid point clouds reaches or exceeds the preset threshold, proceed to step 232.

[0048] The purpose of point cloud quantity detection is to ensure that there is sufficient point cloud data for corner extraction, avoiding unstable fitting or misjudgment in subsequent algorithms due to insufficient data. The preset threshold is 500 points, which is determined based on the size of the calibration board and the resolution of the LiDAR, ensuring that the planar features of the calibration board are fully reflected.

[0049] If the number of valid point clouds detected is less than 500, an error message "insufficient number of point clouds" will be output and subsequent calculations will be terminated, prompting the operator to check the deployment position of the calibration board, the working status of the lidar, or to re-collect data; if the number of valid point clouds reaches or exceeds 500, then step 232 will continue.

[0050] In one possible implementation, the preset threshold can be adjusted according to the point cloud density of the lidar. If the lidar point cloud density is high, the threshold can be set to 800 points to further ensure the integrity of the point cloud data.

[0051] Step 232: The effective point cloud is segmented into planes using a random sampling consistency algorithm to identify the inner point cloud that conforms to the planar features of the diffuse reflection calibration plate.

[0052] The core of plane segmentation is to separate the interior point cloud from the effective point cloud, which conforms to the characteristics of the diffuse reflection calibration plate plane, thus eliminating the influence of noise points and abnormal elevation points. The random sample consensus algorithm is a commonly used algorithm for plane segmentation. Its core idea is to fit a candidate plane by randomly sampling a small number of points, then calculate the distance from all points to the plane, and regard points with a distance less than a preset threshold as interior point clouds, and the rest as exterior point clouds.

[0053] During implementation, the distance threshold for plane fitting is set to 0.01m, which can be adjusted according to the accuracy of the lidar. The number of sampling points is 3, the minimum number of points for plane fitting is 1000, and the number of iterations is 1000. Through multiple iterations, the optimal calibration board plane model is obtained, and finally the inner point cloud is extracted to filter out noise points and outliers.

[0054] The random sampling consensus algorithm is a common algorithm in the field of computer vision. It is only used here as a general reference and does not require further description.

[0055] Step 233: The internal point cloud is reduced to a two-dimensional plane coordinate system by principal component analysis algorithm, and the internal point cloud is mapped to the optimal fitting plane.

[0056] Since the calibration board is a two-dimensional planar structure, reducing the dimensionality of the three-dimensional point cloud to a two-dimensional planar coordinate system simplifies subsequent clustering and boundary fitting operations. Principal component analysis (PCA) is a commonly used algorithm for dimensionality reduction. By performing eigenvalue decomposition on the covariance matrix of the data, it finds the principal direction of the data distribution and projects the three-dimensional point cloud onto a two-dimensional plane composed of the first two principal components. This removes redundant dimensional information while preserving the main geometric structure information.

[0057] During implementation, principal component analysis is performed on the interior point cloud obtained in step 232 to decompose it into three principal components. The variance contributions of these three components in the x, y, and z directions are calculated. The two principal components with the largest variance contributions are selected to construct a two-dimensional plane coordinate system. The three-dimensional interior point cloud is then mapped onto this two-dimensional plane to obtain two-dimensional point cloud data.

[0058] Principal component analysis is a common algorithm in the field of computer vision, and it is only mentioned here for general purposes without further elaboration.

[0059] Step 234: The reduced interior point cloud is automatically clustered using a density-based spatial clustering algorithm with a preset optimal clustering radius parameter.

[0060] The purpose of automatic clustering is to cluster points in a 2D point cloud belonging to different regions of the calibration board, preparing for subsequent boundary endpoint extraction. Density-based spatial clustering algorithms are commonly used in automatic clustering. The core idea is to cluster points based on their density, treating regions with sufficiently high density as a single cluster. This approach is suitable for handling irregularly shaped clustering targets.

[0061] During implementation, a preset optimal clustering radius parameter was used, which was determined through preliminary experiments: 5-10 sets of calibration board point cloud data in different poses were selected, and different radius values ​​within the range of 0.03m-0.08m were tested. The clustering coverage (the proportion of effective cluster points to the total number of point clouds) and clustering purity (the proportion of points within the same cluster belonging to the same calibration board region) were calculated for each radius. The smallest radius with a coverage of ≥95% and a purity of ≥98% was selected as the optimal clustering radius, with an example value of 0.05m, to ensure accurate differentiation of different regions of the calibration board. Clustering operations were performed on the dimensionality-reduced two-dimensional point cloud, ultimately resulting in multiple clusters, each corresponding to a region on the calibration board surface.

[0062] Density-based spatial clustering algorithms are common algorithms in the field of computer vision, and are only cited here for general purposes without further elaboration.

[0063] Step 235: Extract the extreme points distributed along the main direction of each cluster as feature boundary endpoints, fit the feature boundary endpoints using the minimum bounding rectangle algorithm to obtain the two-dimensional rectangular parameters of the diffuse reflection calibration plate, and map the two-dimensional rectangular parameters back to the original three-dimensional spatial coordinate system using the inverse transformation function of principal component analysis to obtain the three-dimensional corner coordinates.

[0064] The core of extreme point extraction is to find the boundary endpoints of each cluster, providing a foundation for subsequent rectangle fitting. The extreme points distributed along the main direction of each cluster are extracted, namely the leftmost, rightmost, topmost, and bottommost points of the cluster boundary, and used as feature boundary endpoints.

[0065] The minimum bounding rectangle algorithm is used to fit all feature boundary endpoints to obtain the two-dimensional rectangular parameters of the calibration plate, including the rectangle's center coordinates, rotation angle, and side length. The minimum bounding rectangle algorithm calculates the convex hull of the boundary endpoints and then fits the smallest rectangle that can enclose all convex hull points, ensuring that the rectangle matches the actual shape of the calibration plate.

[0066] Finally, the four corner points of the two-dimensional rectangle are mapped back to the original three-dimensional spatial coordinate system using the inverse transformation function of principal component analysis, and the three-dimensional corner coordinates of the diffuse reflection calibration plate are obtained and saved as a txt file for subsequent joint calibration.

[0067] The minimum bounding rectangle algorithm is a common algorithm in the field of computer vision. It is only used here as a general reference and does not require further description.

[0068] Step 240: Establish point-pair constraints between the three-dimensional corner coordinates and the pixel coordinates of the corresponding corner points in the calibration image; input the optical parameters to remove the distortion of the pixel coordinates and establish a camera imaging model; under the point-pair constraints, use the solvePnP algorithm to deduce the pose of the monocular camera by minimizing the reprojection error based on the perspective projection equation, and obtain the optimal rotation vector and translation vector; convert the rotation vector into a three-dimensional rotation matrix using the Rodrigues transform; merge the three-dimensional rotation matrix and the translation vector to form a homogeneous extrinsic transformation matrix; the extrinsic transformation matrix constitutes the relative positional relationship in three-dimensional space.

[0069] The core of joint calibration is to establish the mapping relationship between the three-dimensional corner coordinates and the two-dimensional pixel coordinates of the diffuse reflection calibration board, and to solve for the extrinsic parameter transformation matrix describing the relative position and attitude of the two. The extrinsic parameter transformation matrix includes a three-dimensional rotation matrix and a translation vector, which are finally combined into a homogeneous 4×4 matrix.

[0070] In practice, the three-dimensional corner coordinates obtained in step 235 are first matched one-to-one with the two-dimensional pixel coordinates of the calibration plate corners in the corresponding image frames in step 210 to establish point-pair constraint relationships. At least four pairs of valid point pairs are established for each frame of image to ensure that the solution conditions of the perspective projection equation are met.

[0071] The optical parameters obtained in step 210 are then input into the joint calibration algorithm. The distortion correction formula is used to remove the distortion effect of pixel coordinates, thus establishing a complete camera imaging model. The solvePnP algorithm is used to solve for the extrinsic parameters. This algorithm is based on the perspective projection equation and deduces the camera pose by minimizing the reprojection error. The reprojection error is the difference between the projected coordinates of the three-dimensional corner points on the image and the actual pixel coordinates.

[0072] The rotation vector is transformed into a 3D rotation matrix using Rodrigues transform. This matrix describes the rotation relationship between the camera coordinate system and the LiDAR coordinate system. The translation vector describes the translation relationship between the origin of the camera coordinate system and the origin of the LiDAR coordinate system. The 3D rotation matrix and the translation vector are combined to obtain a homogeneous extrinsic transformation matrix, which is used for subsequent transformations from 2D pixel coordinates to 3D world coordinates.

[0073] Both the solvePnP algorithm and the Rodriguez transform are common algorithms in the field of computer vision. They are only cited here for general purposes and do not require further elaboration.

[0074] In this embodiment, the camera optical parameters and extrinsic parameter transformation matrix were calibrated through the above algorithm operation, providing core parameter support for the subsequent construction of coordinate transformation model. The calibration accuracy directly determines the accuracy of the final vehicle positioning.

[0075] Step 300: Based on the optical parameters and the relative positional relationship in three-dimensional space, establish a conversion model from two-dimensional pixel coordinates to three-dimensional world coordinates, and generate a full-resolution lookup table.

[0076] Specifically, the core purpose of step 300 is to establish a transformation model of "two-dimensional pixel coordinates → three-dimensional world coordinates" based on the optical parameters and three-dimensional spatial relative position relationship obtained in step 200, and generate a full-resolution lookup table to realize the fast mapping between pixel points and three-dimensional spatial points.

[0077] Step 310: Read the intrinsic parameter matrix and distortion coefficients in the optical parameters. For the input image pixel coordinates, calculate the normalized coordinates of the image pixel coordinates based on the intrinsic parameter matrix, and then remap them back to the pixel coordinate system after distortion correction.

[0078] Because of optical distortion in camera lenses, directly using the original pixel coordinates for conversion will lead to errors in 3D coordinate calculation. Therefore, distortion correction of the pixel coordinates is necessary first. The distortion correction process is strictly based on the optical parameters obtained in step 200 to ensure that the corrected pixel coordinates accurately reflect the actual geometric relationships.

[0079] In practice, the system reads the optical parameters obtained in step 210 from an external storage file. For the input image pixel coordinates, it first calculates the normalized coordinates using the following formula: , In the formula, Represents the horizontal coordinates of a pixel in the image (column index, dimensionless, unit: pixel count). Represents the coordinates of a pixel in the vertical direction of the image (row index, dimensionless, unit is pixel count). and Represents the principal point coordinates of the image (i.e., the coordinates of the intersection of the camera's optical axis and the imaging plane; dimensionless, with the unit being pixel count). and These are the focal length parameters of the camera in the x-axis and y-axis directions, respectively (measured in pixels per meter, used to convert length units to pixel counts).

[0080] The normalized coordinates are then corrected using distortion coefficients to eliminate the effects of radial and tangential distortion, resulting in corrected normalized coordinates. Finally, the corrected normalized coordinates are remapped back to the pixel coordinate system using an intrinsic parameter matrix to obtain the distortion-corrected pixel coordinates, providing accurate input data for subsequent coordinate transformations.

[0081] Step 320: Expand the distortion-corrected pixel into homogeneous coordinate form, and transform the pixel in homogeneous coordinate form into the camera normalized coordinate system through the inverse matrix of the intrinsic parameter matrix to obtain the line-of-sight vector originating from the optical center of the monocular camera and passing through the pixel.

[0082] In this system, each pixel of a monocular camera corresponds to a line of sight originating from the camera's optical center, and the ray direction vector describes the direction of that line of sight in the camera coordinate system. The purpose of homogeneous coordinate extension is to facilitate coordinate transformations through matrix operations. The homogeneous coordinate form is: , Represents the horizontal coordinate of a pixel after distortion correction (dimensionless, unit is pixel count). It represents the vertical coordinate of the pixel after distortion correction (dimensionless, unit is pixel count), 1 is the homogeneous term (used to unify the dimension of matrix operations), which can more completely express the spatial position relationship of the pixel points compared with two-dimensional coordinates.

[0083] By inversely transforming the intrinsic parameter matrix, the pixels in homogeneous coordinates are projected onto the camera's normalized coordinate system, thereby obtaining the corresponding 3D vectors. This vector represents the line-of-sight vector originating from the optical center of the monocular camera and passing through the pixel. Its direction is consistent with the corresponding ray direction of the pixel in three-dimensional space and has been normalized (magnitude is 1).

[0084] Step 330: Using the three-dimensional rotation matrix in the relative positional relationship of the three-dimensional space, the viewing direction vector is transformed from the camera coordinate system to the world coordinate system. The viewing direction vector in the world coordinate system is described by the parametric equation as the world coordinate point being equal to the viewing direction vector multiplied by the scale parameter in the ray direction and the optical center coordinate of the monocular camera. The parametric equation is substituted into the ground plane equation to obtain the scale parameter. The three-dimensional world coordinates corresponding to the image pixel coordinate point are calculated based on the scale parameter.

[0085] The core of solving the 3D world coordinates is to transform the ray direction vector in the camera coordinate system to the world coordinate system, and then calculate the 3D world coordinates of the pixel point through the intersection with the ground plane. The coordinate system transformation is based on the 3D rotation matrix obtained in step 200, ensuring the consistency of the vector direction in different coordinate systems.

[0086] First, based on the viewing direction vector in the camera coordinate system obtained in step 320... The three-dimensional rotation matrix determined in step 240 Perform a coordinate system transformation to obtain the line-of-sight direction vector in the world coordinate system. The transformation formula is: , Secondly, clarify the coordinates of the optical center of the monocular camera in the world coordinate system. The coordinates are transformed by the extrinsic transformation matrix obtained in step 240. The derivation formula is as follows: , In the formula, The inverse of the three-dimensional rotation matrix (because) (It is an orthogonal matrix, and its inverse is equal to its transpose). It is a translation vector.

[0087] Based on the line-of-sight vector and optical center coordinates in the world coordinate system, the ray parametric equation is established: For any point in the world coordinate system... The necessary and sufficient condition for being located on this ray is: , , , In the formula, Let be the scale parameter (in meters) along the ray direction, representing the distance from the camera optical center to the world coordinate point. The distance; when hour, This refers to the coordinates of the camera's optical center in the world coordinate system.

[0088] After establishing the ray parameter equations, the accurate solution of the scale parameter becomes the core step. This parameter quantifies the proportional relationship between the distance from the camera's optical center to the world coordinate point, and its accuracy directly affects the precision of the sensing system. In practical engineering applications, the horizontal ground of a parking garage is often used as the reference plane, and its spatial plane equation is simplified to: The standard form. When faced with complex scenarios including ramps, this simplified model can no longer meet the accuracy requirements, necessitating an adaptive extension of the plane equations by introducing a general expression for the three-dimensional plane. , where parameters , Respectively characterize the ramp at shaft and Rate of change of slope in the axial direction, parameter The above parameters can be obtained from the number of point clouds in the slope area collected by lidar, where the plane intercept is the angle of attack.

[0089] Substituting the ray parameter equations into the ground plane equations, the scale parameters are obtained by solving. Then the scale parameters Substituting back into the ray parameter equations, the 3D world coordinates corresponding to the input image pixel coordinates can be calculated. This completes the transformation from two-dimensional pixel coordinates to three-dimensional world coordinates.

[0090] In one feasible implementation, for horizontal ground scenarios ( ), scale parameters The solution formula can be simplified to: This ensures both computational efficiency and coordinate transformation accuracy.

[0091] Step 340: Create a three-dimensional array structure based on the image resolution of the monocular camera. The dimensions of the three-dimensional array structure are the vertical pixel resolution, the horizontal pixel resolution, and the three-dimensional world coordinates of the corresponding pixels. Iterate through all pixels row by row and assign the three-dimensional world coordinates of each pixel to the corresponding position of the three-dimensional array structure according to the operations in steps 310 to 330. Generate and save a full-resolution lookup table. The full-resolution lookup table is saved in npy format.

[0092] The full-resolution lookup table is a fast indexing tool designed to improve coordinate transformation efficiency. It is essentially a three-dimensional array that stores the three-dimensional world coordinates corresponding to each pixel. The corresponding three-dimensional coordinates can be obtained directly through the pixel coordinates without having to repeat complex processes such as distortion correction and ray calculation.

[0093] In practice, a three-dimensional array structure is created based on the image resolution of the monocular camera. The array dimensions are (vertical pixel resolution, horizontal pixel resolution, 3), where the third dimension corresponds to the three-dimensional world coordinates. Quantity, The component is always 0 (horizontal ground scene) or conforms to the ramp plane equation (ramp scene), and the dimension is meter.

[0094] The system iterates through each pixel row by row, assigning the corresponding 3D world coordinates of each pixel to the corresponding position in the 3D array according to the operation flow of steps 310-330. During the traversal, the processing progress can be displayed through the tqdm module, which facilitates monitoring of the calculation status and avoids process interruption due to calculation abnormalities.

[0095] After all pixels have been processed, the generated 3D array is saved in .npy format to a specified path on the computing terminal. The .npy format is a NumPy array format, which supports fast reading and indexing, and can significantly improve the real-time performance of subsequent vehicle positioning.

[0096] In this embodiment, by constructing a coordinate transformation model and a full-resolution lookup table, the core pain point of monocular cameras lacking depth information is solved, and accurate and rapid conversion from two-dimensional pixel coordinates to three-dimensional world coordinates is achieved, providing core technical support for subsequent real-time vehicle positioning.

[0097] Step 400: The YOLO object detection algorithm is used to extract the vehicle bounding box from the vehicle video image, and the vehicle's three-dimensional spatial coordinates are obtained by mapping through the full-resolution lookup table to achieve real-time vehicle positioning.

[0098] Specifically, the core purpose of step 400 is to train the YOLO model using the training dataset constructed in step 100, extract the pixel coordinates of the vehicle bounding box through the model, and then combine them with the full-resolution lookup table generated in step 300 to quickly map and obtain the three-dimensional spatial coordinates of the vehicle, thereby achieving real-time vehicle positioning.

[0099] Step 410: Select a lightweight YOLO series model and train the model based on the training dataset to obtain a vehicle detection model.

[0100] Among them, the YOLO series models are real-time object detection frameworks based on deep learning. Their core advantages are fast detection speed and accurate bounding box regression, making them suitable for real-time vehicle detection scenarios. Lightweight YOLO series models, such as YOLOv5s and YOLOv7-tiny, are selected to balance detection speed and accuracy, meeting the real-time positioning requirements in parking garage scenarios.

[0101] In practice, the training dataset constructed in step 100 is divided into a training set and a validation set in an 8:2 ratio. The training set is used for model parameter learning, and the validation set is used for model performance evaluation. A PyTorch training framework is set up on a computing terminal, with the following training parameters set: batch size of 16 (adjusted according to the GPU memory of the computing terminal), learning rate of 0.001, training epochs of 100, stochastic gradient descent as the optimizer, and CIoU loss function to optimize the bounding box regression accuracy.

[0102] During training, the model learns vehicle feature information from the training set and optimizes the bounding box coordinate prediction accuracy through the bounding box regression branch, with the optimization objective being that "the bounding box output by the model fits the contact point between the vehicle and the ground." Simultaneously, the model's performance is monitored in real time using the validation set, and the average accuracy on the validation set is calculated. When the average accuracy reaches above 95% and there is no significant improvement after five consecutive training epochs, training is stopped, and the final vehicle detection model is saved.

[0103] The training process for the YOLO series models is a standard procedure in the field of deep learning, and is only cited here without further elaboration.

[0104] In one possible implementation, data augmentation operations, such as random cropping, flipping, and brightness adjustment, can be performed on the training dataset to improve the model's generalization ability. The number of training rounds can be adjusted according to the size of the dataset. If the dataset contains more than 1,000 valid images, the number of training rounds can be set to 80-120 rounds.

[0105] Step 420: Process the vehicle video image using the vehicle detection model to output the vehicle's identification information, bounding box coordinates, and pixel coordinates of the bottom border of the bounding box.

[0106] The core of vehicle bounding box extraction is to use a trained YOLO model to accurately extract the bounding box coordinates and related information of each target vehicle from real-time video images captured by a monocular camera. The monocular camera captures real-time video images of the parking garage at a fixed frame rate of 15-30 FPS to ensure that the dynamic motion trajectory of the vehicles can be captured.

[0107] After processing each frame of the image, the model outputs three key types of information: first, vehicle identification information, used to distinguish different vehicles and avoid confusion during vehicle tracking; second, the coordinates of the vehicle bounding box, including the pixel coordinates of the top-left corner of the bounding box. and the pixel coordinates of the bottom right corner The first is used to determine the vehicle's position in the image; the second is the pixel coordinates of the lower bounding box, i.e. All pixel coordinates at time This coordinate represents the pixel position of the point of contact between the vehicle and the ground, and is used for accurate mapping of subsequent three-dimensional spatial coordinates.

[0108] If the model does not detect a vehicle, it outputs a message "No target vehicle"; if a vehicle is detected but the bounding box is not fully extracted, it outputs a message "Vehicle occlusion" to avoid localization errors caused by invalid data.

[0109] Step 430: Input the bounding box coordinates and the pixel coordinates of the bottom border of the bounding box into the full-resolution lookup table, and obtain the three-dimensional spatial coordinates of the vehicle through index mapping.

[0110] The core of the vehicle's 3D spatial coordinate mapping is to use the full-resolution lookup table generated in step 300 to quickly map the pixel coordinates of the vehicle's lower bounding box extracted in step 420 to 3D world coordinates. The indexing operation of the lookup table does not require complex calculations; the corresponding 3D coordinates can be directly obtained simply by using the row and column indexes of the pixel coordinates, significantly improving the positioning speed.

[0111] In practice, the coordinates of all pixels of the bottom border of the bounding box will be... As an index, a full-resolution lookup table is input, and the lookup table directly returns the corresponding 3D world coordinates from the pixel coordinates. To improve positioning accuracy, the average of the 3D world coordinates corresponding to all pixel coordinates of the lower bounding box is taken to obtain the final 3D spatial coordinates of the vehicle.

[0112] The positioning results can be output to the display terminal in real time, showing the vehicle's specific location in the parking lot in the form of a digital or visual map. The positioning frequency is consistent with the camera's frame rate, meeting the real-time positioning requirements.

[0113] In this embodiment, by combining the YOLO model with a full-resolution lookup table, the vehicle can be located quickly and accurately, which solves the problems of low positioning accuracy and poor real-time performance of traditional monocular cameras, while avoiding the high cost caused by the full participation of lidar in positioning.

[0114] Step 500: Based on the vehicle motion trajectory consistency verification and scene fixed reference features constructed from the vehicle's three-dimensional spatial coordinates, iteratively optimize the optical parameters and the relative positional relationship in three-dimensional space to correct parameter drift.

[0115] Specifically, the core purpose of step 500 is to solve the problem of calibration parameter drift caused by factors such as equipment vibration and temperature changes during long-term use. By verifying the consistency of vehicle motion trajectory and using fixed reference features of the scene, the optical parameters and the relative positional relationship in three-dimensional space are iteratively optimized to ensure the long-term stability of vehicle positioning accuracy.

[0116] Step 510: Construct the vehicle motion trajectory based on the vehicle's three-dimensional spatial coordinates, and use a kinematic model to determine the consistency of the vehicle motion trajectory; if the vehicle's three-dimensional spatial coordinates deviate from the predicted value of the kinematic model for a consecutive preset number of frames by more than a preset deviation threshold, then continue to execute the subsequent steps; if the vehicle's three-dimensional spatial coordinates do not deviate from the predicted value of the kinematic model for a consecutive preset number of frames or the degree of deviation does not exceed the preset deviation threshold, then jump to step 420, and continue to process the vehicle video images acquired in real time by the monocular camera using the vehicle detection model.

[0117] The core of the vehicle trajectory consistency judgment is to verify the rationality of the vehicle positioning results through a kinematic model. If the positioning results deviate from the kinematic laws, it is judged as parameter drift, triggering the optimization process. Based on the vehicle's three-dimensional spatial coordinates output in step 400, the vehicle trajectory is constructed according to the time series, and the trajectory data includes vehicle coordinate information at different times.

[0118] Two types of kinematic models were selected to determine the consistency of the trajectory: one is the uniform linear motion equation, suitable for vehicles traveling straight; the other is the circular motion equation, suitable for vehicles turning. The preset frame count is five frames, and the preset deviation threshold is three centimeters, which is set according to the positioning accuracy requirements of the parking garage.

[0119] If the vehicle's 3D spatial coordinates deviate from the kinematic model's predicted values ​​by more than three centimeters for five consecutive frames, it is considered parameter drift, and subsequent optimization steps continue. If there is no deviation or the deviation is less than three centimeters, optimization steps are not triggered, and real-time vehicle localization continues. The kinematic model's predicted values ​​are calculated based on the vehicle's coordinates and velocity in the previous frames. For example, the formula for predicting uniform linear motion is: ,

[0120] In the formula, Indicates the vehicle in the current frame in the world coordinate system. Predicted three-dimensional coordinates of the direction (in meters). Indicates the vehicle in the current frame in the world coordinate system. Predicted three-dimensional coordinates of the direction (in meters); This indicates that the vehicle in the previous frame is in the world coordinate system. The actual three-dimensional coordinates of the direction (in meters). This indicates that the vehicle in the previous frame is in the world coordinate system. The actual three-dimensional coordinates of the direction (in meters); Indicates that the vehicle is in The velocity of motion in the direction (in meters per second). Indicates that the vehicle is in The velocity of motion in the direction (in meters per second) is calculated from the coordinate differences of the previous few frames and the frame interval time, such as... ; This represents the time interval (in seconds) between two adjacent frames, and is equal to the reciprocal of the camera's frame rate (e.g., at a frame rate of 30 FPS). Second).

[0121] In one possible implementation, the preset frame count can be adjusted to 3-8 frames, and the preset deviation threshold can be adjusted to 2-5 centimeters, which can be flexibly set according to the positioning accuracy requirements of the actual application scenario.

[0122] Step 520: Identify fixed reference features in the integrated transportation hub parking garage scene using the YOLO object detection algorithm, extract the current pixel coordinates of the fixed reference features, input the current pixel coordinates into the full-resolution lookup table to obtain the current three-dimensional coordinates, and store the true three-dimensional coordinates of the fixed reference features synchronously during the initial joint calibration stage.

[0123] Among them, fixed reference features refer to objects in the parking garage that have stable geometric features and will not move, including column corners, wall corners, intersections of parking lines, etc. Their three-dimensional coordinates have been directly collected and stored by lidar in the initial joint calibration stage of step 200, and are used as the true three-dimensional coordinates for parameter optimization.

[0124] The YOLO object detection algorithm is used to identify fixed reference features in images and extract the current pixel coordinates of these features. The identification process is the same as that for vehicle bounding box extraction. Training samples of fixed reference features need to be labeled in advance (which can be merged with vehicle training samples to build a dataset) to train a specialized recognition model and ensure the accuracy of feature recognition.

[0125] Input the current pixel coordinates into the full-resolution lookup table generated in step 300 to obtain the current 3D coordinates. If the deviation between the current 3D coordinates of the fixed reference feature and the true 3D coordinates exceeds 1 cm, it is determined to be parameter drift, and the optimization process needs to be triggered in conjunction with the vehicle motion trajectory consistency judgment result.

[0126] In one possible implementation, there are no fewer than five fixed reference features, which are evenly distributed within the monitoring area of ​​the parking garage to ensure that sufficient constraints are provided during parameter optimization.

[0127] Step 530: Using the true three-dimensional coordinates of the fixed reference feature as a reference, construct an objective function to minimize the reprojection error, and iteratively correct the optical parameters and the relative positional relationship in three-dimensional space through the objective function.

[0128] The core of the calibration parameter iterative correction is to use the true 3D coordinates of a fixed reference feature as a benchmark, and iteratively optimize the optical parameters and their relative positional relationship in 3D space by minimizing the reprojection error. The objective function expression for minimizing the reprojection error is: , In the formula, The pixel coordinates of the current reference feature (dimensionless, in pixel count). The true three-dimensional coordinates (in meters) of the reference feature. This is the camera intrinsic parameter matrix (dimensionless, its core function is coordinate transformation). It is a three-dimensional rotation matrix (dimensionless). It is a translation vector (unit: meters). The projection function (based on the perspective projection equation in step 240, converts the three-dimensional coordinates into pixel coordinates to achieve dimensional unification).

[0129] The objective function is iteratively optimized using the gradient descent algorithm, and the optical parameters (intrinsic parameter matrix) are adjusted. (distortion coefficients) and extrinsic transformation matrix The parameter values ​​are set, with a learning rate of 0.001 and an upper limit of 1000 iterations during the iteration process, until the reprojection error is less than a preset threshold (e.g., 1 pixel) or the upper limit of the iteration count is reached. During the optimization process, the adjustment range of the parameters needs to be constrained (e.g., the adjustment range of the intrinsic parameter matrix parameters should not exceed ±10% of the initial value) to avoid the parameters deviating from the reasonable range due to over-optimization, which would affect the positioning accuracy.

[0130] Gradient descent is a common algorithm in the field of numerical optimization, and it is only used here as a general reference without further explanation.

[0131] Step 540: Substitute the optimized optical parameters and the relative positional relationship in three-dimensional space into the transformation model, and repeat the operations from step 310 to step 340 to generate a new full-resolution lookup table.

[0132] After the calibration parameters are optimized, a new full-resolution lookup table needs to be generated to adapt to the optimized parameters. The generation process of the new lookup table is the same as that of the original lookup table, ensuring that the optimized positioning accuracy can be quickly implemented and applied.

[0133] In practice, the optimized optical parameters and the relative positional relationship in three-dimensional space are substituted into the coordinate transformation model in step 300, and the operations in steps 310 to 340 are re-executed. All pixels are traversed row by row, the three-dimensional world coordinates corresponding to each pixel are calculated, and a new full-resolution lookup table is generated to replace the original lookup table for subsequent mapping of vehicle three-dimensional spatial coordinates.

[0134] In this embodiment, by verifying the consistency of vehicle motion trajectory and calibrating fixed reference features, dynamic optimization of calibration parameters is achieved, correcting parameter drift issues during long-term use, ensuring the stability of vehicle positioning accuracy, and further improving the practicality and reliability of the technical solution.

[0135] Furthermore, the iterative optimization process triggered in step 500 must meet two triggering conditions: the first triggering condition is that the consistency deviation of the vehicle's motion trajectory continuously exceeds the preset deviation threshold, and the second triggering condition is that the number of effectively identified fixed reference features is not less than three, and the iterative optimization process is executed asynchronously.

[0136] The dual triggering condition is set to ensure the necessity and effectiveness of parameter optimization. The requirement of having no fewer than three effectively identified fixed reference features is based on the technical principle of three-dimensional coordinate calibration. At least three non-collinear reference points are needed to construct an effective constraint equation, thus ensuring the effectiveness of parameter optimization.

[0137] The asynchronous execution mode means that the optimization process and the real-time vehicle positioning process are carried out in parallel. The optimization calculation occupies independent thread resources, which does not affect the calculation speed and timeliness of real-time positioning, and ensures that the vehicle positioning process is uninterrupted and without delay.

[0138] In this embodiment, by integrating the advantages of LiDAR and monocular camera, the spatial coordinate relationship between the two is established through a single calibration using a calibration board, compensating for the lack of depth information in monocular camera and improving vehicle positioning accuracy. It avoids the high cost of using a single LiDAR, achieving rapid coordinate mapping through a full-resolution lookup table while also meeting real-time positioning requirements. Adapting to the complex scenarios of integrated transportation hubs, it solves the problem of missing point clouds in complex lighting, dense vehicle traffic, and occlusion environments, improving the model's generalization ability. Through motion trajectory verification and iterative optimization of fixed reference features, parameter drift is corrected, ensuring long-term positioning stability. This effectively overcomes the limitations of single-device positioning accuracy, economy, and stability in hub scenarios, demonstrating significant practical value.

[0139] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0140] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A monocular vision perception enhancement method based on lidar, characterized in that, Includes the following steps: Step 100: Acquire vehicle video images of the parking lot scene of the integrated transportation hub using a monocular camera, and acquire three-dimensional point cloud data of the scene using a lidar. The three-dimensional point cloud data includes feature information of the diffuse reflection calibration plate. Step 200: Based on the vehicle video image and 3D point cloud data, perform intrinsic parameter calibration on the monocular camera and joint calibration of the camera and lidar to determine the optical parameters of the monocular camera and the 3D spatial relative position relationship between the monocular camera and the lidar deployment point. Step 300: Based on the optical parameters and the relative positional relationship in three-dimensional space, establish a transformation model from two-dimensional pixel coordinates to three-dimensional world coordinates and generate a full-resolution lookup table; Step 400: The YOLO object detection algorithm is used to extract the vehicle bounding box from the vehicle video image, and the vehicle's three-dimensional spatial coordinates are obtained by mapping through the full-resolution lookup table to achieve real-time vehicle positioning. Step 500: Based on the vehicle motion trajectory consistency verification and scene fixed reference features constructed from the vehicle's three-dimensional spatial coordinates, iteratively optimize the optical parameters and the relative positional relationship in three-dimensional space to correct parameter drift.

2. The monocular vision perception enhancement method based on lidar according to claim 1, characterized in that, Step 100 includes: Step 110: Deploy the lidar in an unobstructed position behind the monocular camera, and deploy the diffuse reflection calibration plate in the integrated transportation hub parking garage scene; Step 120: Control the monocular camera and lidar to synchronously record multiple sets of calibration data with different poses using ROS-docker script, and change the pose of the diffuse reflection calibration plate multiple times during the recording process; Step 130: In the low-light, occluded, and multi-directional driving environment of the integrated transportation hub parking garage scenario, vehicle images are acquired using the monocular camera. After filtering the acquired vehicle images, the labelimg tool is used to annotate the filtered vehicle images to construct a training dataset.

3. The monocular vision perception enhancement method based on lidar according to claim 1, characterized in that, Step 200 includes: Step 210: Extract clear frames of the calibration images from the calibration data, process the clear frames using MATLAB, and solve for the intrinsic parameter matrix and distortion coefficients of the monocular camera. The intrinsic parameter matrix and distortion coefficients together constitute the optical parameters. Step 220: Use CloudCompare to filter the three-dimensional point cloud data. Initially filter out irrelevant point clouds by limiting the coordinate range, and then accurately filter out point clouds related to the diffuse reflection calibration plate fixing device by calculating the minimum height of the z-axis, while retaining the effective point cloud of the diffuse reflection calibration plate. Step 230: Perform corner point extraction processing on the effective point cloud to obtain the three-dimensional corner coordinates of the diffuse reflection calibration plate; Step 240: Establish point-pair constraints between the three-dimensional corner coordinates and the pixel coordinates of the corresponding corner points in the calibration image; input the optical parameters to remove the distortion of the pixel coordinates and establish a camera imaging model; under the point-pair constraints, use the solvePnP algorithm to deduce the pose of the monocular camera by minimizing the reprojection error based on the perspective projection equation, and obtain the optimal rotation vector and translation vector; convert the rotation vector into a three-dimensional rotation matrix using the Rodrigues transform; merge the three-dimensional rotation matrix and the translation vector to form a homogeneous extrinsic transformation matrix; the extrinsic transformation matrix constitutes the relative positional relationship in three-dimensional space.

4. The monocular vision perception enhancement method based on lidar according to claim 3, characterized in that, The defined coordinate range is set according to the actual size of the integrated transportation hub parking garage scene, and the minimum height of the z-axis is determined by the actual installation height of the diffuse reflection calibration plate fixing device.

5. The monocular vision perception enhancement method based on lidar according to claim 3, characterized in that, Step 230 includes: Step 231: Detect the number of valid point clouds. If the number of valid point clouds is less than a preset threshold, output an abnormal prompt message indicating insufficient point cloud quantity and terminate subsequent operations. If the number of valid point clouds reaches or exceeds the preset threshold, proceed to step 232. Step 232: Perform planar segmentation on the effective point cloud using the random sampling consensus algorithm to identify the inner point cloud that conforms to the planar features of the diffuse reflection calibration plate; Step 233: Reduce the dimensionality of the interior point cloud to a two-dimensional planar coordinate system using the principal component analysis algorithm, and map the interior point cloud to the optimal fitting plane; Step 234: The reduced interior point cloud is automatically clustered using a density-based spatial clustering algorithm, with a preset optimal clustering radius parameter. Step 235: Extract the extreme points distributed along the main direction of each cluster as feature boundary endpoints, fit the feature boundary endpoints using the minimum bounding rectangle algorithm to obtain the two-dimensional rectangular parameters of the diffuse reflection calibration plate, and map the two-dimensional rectangular parameters back to the original three-dimensional spatial coordinate system using the inverse transformation function of principal component analysis to obtain the three-dimensional corner coordinates.

6. The monocular vision perception enhancement method based on lidar according to claim 1, characterized in that, Step 300 includes: Step 310: Read the intrinsic parameter matrix and distortion coefficients in the optical parameters. For the input image pixel coordinates, calculate the normalized coordinates of the image pixel coordinates according to the intrinsic parameter matrix, and then remap them back to the pixel coordinate system after distortion correction. Step 320: Expand the distortion-corrected pixel into homogeneous coordinate form, and transform the pixel in homogeneous coordinate form into the camera normalized coordinate system through the inverse matrix of the intrinsic parameter matrix to obtain the line-of-sight direction vector starting from the optical center of the monocular camera and passing through the pixel. Step 330: Using the three-dimensional rotation matrix in the relative positional relationship of the three-dimensional space, the viewing direction vector is transformed from the camera coordinate system to the world coordinate system. The viewing direction vector in the world coordinate system is described by the parametric equation as the world coordinate point being equal to the viewing direction vector multiplied by the scale parameter in the ray direction and the optical center coordinate of the monocular camera. The parametric equation is substituted into the ground plane equation to obtain the scale parameter. The three-dimensional world coordinates corresponding to the image pixel coordinate point are calculated based on the scale parameter. Step 340: Create a three-dimensional array structure based on the image resolution of the monocular camera. The dimensions of the three-dimensional array structure are the vertical pixel resolution, the horizontal pixel resolution, and the three-dimensional world coordinates of the corresponding pixels. Iterate through all pixels row by row and assign the three-dimensional world coordinates of each pixel to the corresponding position of the three-dimensional array structure according to the operations in steps 310 to 330. Generate and save a full-resolution lookup table. The full-resolution lookup table is saved in npy format.

7. The monocular vision perception enhancement method based on lidar according to claim 1, characterized in that, Step 400 includes: Step 410: Select a lightweight YOLO series model and train the model based on the training dataset to obtain a vehicle detection model; Step 420: Process the vehicle video image using the vehicle detection model to output the vehicle's identification information, bounding box coordinates, and pixel coordinates of the bottom border of the bounding box. Step 430: Input the bounding box coordinates and the pixel coordinates of the bottom border of the bounding box into the full-resolution lookup table, and obtain the three-dimensional spatial coordinates of the vehicle through index mapping.

8. The monocular vision perception enhancement method based on lidar according to claim 1, characterized in that, Step 500 includes: Step 510: Construct the vehicle motion trajectory based on the vehicle's three-dimensional spatial coordinates, and perform a consistency judgment on the vehicle motion trajectory using a kinematic model; if the vehicle's three-dimensional spatial coordinates deviate from the predicted value of the kinematic model for a consecutive preset number of frames by more than a preset deviation threshold, then continue to step 520; if the vehicle's three-dimensional spatial coordinates do not deviate from the predicted value of the kinematic model for a consecutive preset number of frames or the degree of deviation does not exceed the preset deviation threshold, then jump to step 420, and continue to process the vehicle video images acquired in real time by the monocular camera using a vehicle detection model; Step 520: Identify fixed reference features in the integrated transportation hub parking garage scene using the YOLO object detection algorithm, extract the current pixel coordinates of the fixed reference features, input the current pixel coordinates into the full-resolution lookup table to map and obtain the current three-dimensional coordinates, and store the true three-dimensional coordinates of the fixed reference features synchronously during the initial joint calibration stage; Step 530: Based on the true three-dimensional coordinates of the fixed reference feature, construct an objective function to minimize the reprojection error, and iteratively correct the optical parameters and the relative positional relationship in three-dimensional space through the objective function; Step 540: Substitute the optimized optical parameters and the relative positional relationship in three-dimensional space into the transformation model, and repeat the operations from step 310 to step 340 to generate a new full-resolution lookup table.

9. A monocular vision perception enhancement method based on lidar according to claim 8, characterized in that, The kinematic model includes equations for uniform linear motion and circular motion. The preset frame count is five frames, the preset deviation threshold is three centimeters, and the objective function for minimizing the reprojection error is... In the formula, The pixel coordinates of the current reference feature. The true three-dimensional coordinates of the reference feature. For the camera intrinsic parameter matrix, It is a three-dimensional rotation matrix. It is a translation vector. This is the projection function.

10. A monocular vision perception enhancement method based on lidar according to claim 8, characterized in that, The iterative optimization process triggered in step 500 must meet two triggering conditions: the first triggering condition is that the consistency deviation of the vehicle's motion trajectory continuously exceeds the preset deviation threshold, and the second triggering condition is that the number of effectively identified fixed reference features is not less than three, and the iterative optimization process is executed asynchronously.

Images