Vehicle posture visual recognition method and system for parking robot

Abstract

The present application relates to the technical field of intelligent parking and robot vision automation control, in particular to a vehicle posture vision recognition method and system for parking robots, comprising: acquiring multi-view two-dimensional images and three-dimensional point clouds with the same time stamp through multi-view vision and laser radar, and performing sampling and coordinate alignment; inputting the aligned data into a feature extraction network to extract and fuse the texture of the image and the depth information of the point cloud; performing weighted processing on the fused data of continuous multiple frames to generate a three-dimensional point set; combining the extracted environmental brightness of the image and a preset vehicle model library to compare and rigidly register the three-dimensional point set to obtain the initial pose of the vehicle; combining chassis odometer data to correct and exclude abnormalities of the initial pose, and outputting the final vehicle posture to the motion control bottom layer; the present application suppresses single-frame pose jumping, and guarantees the stability and continuity of the recognition result.

Description

Technical Field

[0001] This invention relates to the field of intelligent parking and robot vision automation control technology, specifically to a vehicle posture visual recognition method and system for parking robots. Background Technology

[0002] The vehicle posture recognition of parking robots is essentially to obtain information about the vehicle's surroundings through multi-view vision arrays and sensors such as LiDAR, in order to analyze the spatial position and orientation of the target vehicle, thereby serving the operation of vehicles to be moved or guided in the parking lot lane.

[0003] Existing vehicle pose recognition methods mostly rely on a single sensor or only on the vehicle's bounding box for contour determination. However, in narrow parking spaces and complex lighting environments, relying solely on the vehicle's bounding box is insufficient to meet the robot's grasping and guidance accuracy requirements. Furthermore, visual cameras are easily affected by vehicle reflections, leading to positional shifts, and lidar often exhibits discrete points of local depth due to wheel rim reflections. If the pose is calculated based solely on a single frame or single sensor result, it is highly susceptible to unstable vehicle boundary judgments and pose jumps. Currently, for high-precision parking robot operations, it is necessary to capture finer-grained data such as wheel center points and curvature change lines to assist in registration.

[0004] Therefore, how to effectively integrate image texture and 3D point cloud information, and overcome single-frame jump errors to automatically output stable and accurate vehicle poses, has become a problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a vehicle posture visual recognition method and system for parking robots, avoiding single-frame recognition jumps and data mismatches caused by occlusion and local reflective environments, and realizing continuous vehicle posture output, thereby improving the continuous operation capability of parking robots in complex environments.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A vehicle posture visual recognition method for parking robots includes the following steps:

[0008] The parking robot uses a multi-view vision array and LiDAR to acquire multi-view 2D images and 3D point clouds of the vehicle at the same time stamp, and performs point cloud sampling adjustment and coordinate alignment to obtain aligned image and point cloud data.

[0009] The aligned image and point cloud data are input into a feature extraction network containing image branches and point cloud branches for information extraction and stitching to obtain combined data containing texture information and depth information.

[0010] The combined data of multiple consecutive frames is cached, and multi-frame weighted processing is performed based on the combined data of multiple consecutive frames to obtain the processed three-dimensional point set;

[0011] Based on the extraction of ambient brightness values ​​from multi-view vehicle 2D images, the vehicle model comparison and rigid body registration are performed on the processed 3D point set by combining the preset vehicle model library and ambient brightness values ​​to obtain the initial vehicle pose.

[0012] The initial vehicle pose is combined with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal. The corrected vehicle pose is then output as the visual recognition result of the target vehicle posture to the motion control layer of the parking robot.

[0013] Furthermore, based on the multi-view vision array and lidar of the parking robot, multi-view 2D images and 3D point clouds of the vehicle at the same time stamp are acquired, and point cloud sampling adjustment and coordinate alignment are performed to obtain aligned image and point cloud data, including:

[0014] Based on the hardware trigger pulse sent by the hardware synchronous clock source, multi-view vehicle 2D images and vehicle 3D point clouds at the same timestamp are acquired.

[0015] In non-initial state, obtain the physical distance from the target vehicle to the parking robot in the previous frame; in initial state, use the preset default physical distance.

[0016] The point cloud sampling retention rate is calculated based on physical distance, and the vehicle's 3D point cloud is sampled and adjusted according to the point cloud sampling retention rate.

[0017] The static environmental features of the parking lot are extracted, the extrinsic parameter matrix between the multi-view vision array and the lidar is calculated, and the pixel coordinates of the multi-view vehicle 2D images are projected onto the coordinate system of the vehicle 3D point cloud based on the extrinsic parameter matrix to obtain the aligned image and point cloud data.

[0018] Furthermore, the point cloud sampling retention rate is calculated as follows:

[0019] Based on the physical distance from the target vehicle to the parking robot in the previous frame, the preset optimal operating distance, and the preset distance attenuation coefficient, the point cloud sampling retention rate is calculated, and the point cloud sampling retention rate is not lower than the preset minimum point cloud retention threshold.

[0020] Furthermore, the aligned image and point cloud data are input into a feature extraction network containing image and point cloud branches for information extraction and concatenation, resulting in combined data containing texture and depth information, including:

[0021] In the image branch of the feature extraction network, two-dimensional pixel information of wheel grounding points, the outermost edge of the vehicle's exterior rearview mirror, door handles, and charging ports or fuel tank caps are extracted from multi-view vehicle two-dimensional images.

[0022] In the point cloud branch of the feature extraction network, information on the curvature change lines of the front and rear bumpers of the vehicle and the cylindrical geometry information of the wheel area are extracted from the vehicle's 3D point cloud, and the position of the wheel center point is calculated based on the cylindrical geometry information.

[0023] Based on the pre-calibrated extrinsic matrix between the multi-view vision array and the lidar, the two-dimensional pixel information is stitched together with the curvature change line information of the front and rear bumpers, the cylindrical geometry information, and the position of the wheel center point in the channel dimension to obtain combined data containing texture information and depth information, and the information points in the combined data are defined as vehicle marker points.

[0024] Furthermore, combined data from multiple consecutive frames is cached, and multi-frame weighted processing is performed based on this combined data to obtain a processed set of 3D point locations, including:

[0025] Create a sliding window and cache consecutive windows. Combined data of frames, It is a positive integer greater than or equal to 5;

[0026] For continuous Add timestamps to vehicle markers in the combined data of frames;

[0027] Calculate the inter-frame weight between the current frame and historical frames based on the corresponding point matching score between the current frame and historical frames;

[0028] Based on inter-frame weights for continuous The combined data of the frames are weighted and summed to obtain the processed three-dimensional point set.

[0029] Furthermore, displacement compensation is performed on the wheel center point position of historical frames based on the chassis movement speed of the parking robot to obtain the predicted wheel center point position; the spatial distance deviation between the wheel center point position of the current frame and the predicted wheel center point position is extracted, and the inter-frame weight is determined by combining the corresponding point matching score between the current frame and the historical frames.

[0030] Furthermore, ambient brightness values ​​are extracted from multi-view vehicle 2D images. These values ​​are then combined with a pre-defined vehicle model database to perform vehicle model comparison and rigid body registration on the processed 3D point set, yielding the vehicle's initial pose, including:

[0031] The measured wheelbase and measured track width are calculated based on the wheel center point positions in the processed 3D point set, and then compared and retrieved in a preset vehicle model library to obtain the standard vehicle model of the target vehicle.

[0032] Calculate ambient brightness value based on average grayscale value of multi-view vehicle 2D images;

[0033] Calculate the difference or ratio between the ambient brightness value and the preset illumination threshold, and map it to a weighted ratio of visual weight and point cloud weight;

[0034] Vehicle pose determination is based on 3D point cloud computing LiDAR and vehicle pose determination is based on multi-view vehicle 2D images.

[0035] The initial pose of the vehicle in three-dimensional space is calculated by weighted summation of the pose determined by lidar and vision based on a weighted ratio. Then, the initial pose of the vehicle in three-dimensional space is fine-tuned by rigid body registration based on a standard vehicle model to obtain the initial pose of the vehicle.

[0036] Furthermore, the initial vehicle pose is combined with the chassis wheel odometer data of the parking robot for overall pose correction and abnormal pose removal. The corrected vehicle pose is then output as the target vehicle pose visual recognition result to the motion control layer of the parking robot, including:

[0037] The initial vehicle pose and chassis wheel odometer data are input into a preset pose correction program. The parking robot pose calculated based on the chassis wheel odometer data is used as the pose reference point. The processed three-dimensional point set is used as the reference road sign point for trajectory closed-loop correction to obtain the optimized vehicle pose.

[0038] Calculate the yaw angle change and lateral displacement change of the optimized vehicle pose between two adjacent frames.

[0039] If the change in yaw angle is greater than a preset threshold for yaw angle change, or the change in lateral displacement is greater than a preset threshold for lateral displacement change, the optimized vehicle pose of the current frame is discarded, and the Kalman filter is used to predict the pose of the next frame based on the historical pose state.

[0040] If the change in yaw angle is less than or equal to the preset threshold for yaw angle change, and the change in lateral displacement is less than or equal to the preset threshold for lateral displacement change, then the current optimized vehicle pose will be output as the target vehicle pose visual recognition result to the motion control layer of the parking robot.

[0041] A vehicle posture vision recognition system for parking robots includes:

[0042] The acquisition and alignment module is used to acquire multi-view 2D images and 3D point clouds of the vehicle at the same time stamp based on the multi-view vision array and lidar of the parking robot, and to perform point cloud sampling adjustment and coordinate alignment to obtain aligned image and point cloud data.

[0043] The information processing and stitching module is used to input the aligned image and point cloud data into a feature extraction network containing image branches and point cloud branches for information extraction and stitching, to obtain combined data containing texture information and depth information.

[0044] The weighted processing and recognition module is used to cache the combined data of multiple consecutive frames and perform multi-frame weighted processing based on the combined data of multiple consecutive frames to obtain the processed three-dimensional point set; the ambient brightness value is extracted based on the multi-view vehicle two-dimensional image, and the vehicle model comparison and rigid body registration are performed on the processed three-dimensional point set in combination with the preset vehicle model library and the ambient brightness value to obtain the initial pose of the vehicle.

[0045] The correction and output module is used to combine the initial vehicle pose with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal, and output the corrected vehicle pose as the target vehicle posture visual recognition result to the motion control layer of the parking robot.

[0046] The beneficial effects of this invention are:

[0047] 1. This invention calculates the point cloud sampling retention rate by extracting the physical distance from the center point of the target vehicle to the parking robot, and dynamically adjusts the vehicle's 3D point cloud accordingly. This solution changes the traditional fixed full point cloud processing mode and can adaptively adjust the point cloud density according to the working distance. While ensuring the vehicle contour recognition requirements, it reduces the amount of redundant data processing, thereby effectively reducing the processing load of the computing unit and the system response delay.

[0048] 2. This invention establishes a sliding window to cache combined data from multiple consecutive frames, and calculates inter-frame weights by combining the chassis movement speed, distance deviation, and corresponding point matching scores of the parking robot, and then performs a weighted summation of the data from multiple consecutive frames; this mechanism introduces temporal continuity and physical motion constraints, effectively overcomes feature matching inaccuracies caused by reflections on the vehicle surface or partial field of view obstruction, smooths the recognition trajectory, significantly suppresses pose jumps in a single frame, and ensures the stability and continuity of the recognition results;

[0049] 3. This invention performs deep stitching of local two-dimensional pixel information extracted from image branches and three-dimensional geometric information extracted from point cloud branches in the channel dimension, and innovatively extracts ambient brightness values ​​to dynamically map visual weights and point cloud weights for pose fusion calculation; combined with chassis wheel odometer data for closed-loop correction and abnormal pose removal, this solution reduces the interference of complex physical lighting changes in parking lots on a single sensor, and comprehensively improves the accuracy and robustness of target vehicle posture visual recognition. Attached Figure Description

[0050] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0051] Figure 1 This is a flowchart illustrating the steps of the vehicle posture visual recognition method for parking robots according to the present invention.

[0052] Figure 2 This diagram illustrates a system block diagram of a vehicle posture visual recognition system for parking robots according to an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1 A vehicle posture visual recognition method for parking robots includes the following steps: acquiring multi-view two-dimensional images and three-dimensional point clouds of the vehicle at the same time stamp based on the multi-view visual array and lidar of the parking robot, and performing point cloud sampling adjustment and coordinate alignment to obtain aligned image and point cloud data;

[0055] The aligned image and point cloud data are input into a feature extraction network containing image branches and point cloud branches for information extraction and stitching to obtain combined data containing texture information and depth information.

[0056] The combined data of multiple consecutive frames is cached, and multi-frame weighted processing is performed based on the combined data of multiple consecutive frames to obtain the processed three-dimensional point set;

[0057] Based on the extraction of ambient brightness values ​​from multi-view vehicle 2D images, the vehicle model comparison and rigid body registration are performed on the processed 3D point set by combining the preset vehicle model library and ambient brightness values ​​to obtain the initial vehicle pose.

[0058] The initial vehicle pose is combined with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal. The corrected vehicle pose is then output as the visual recognition result of the target vehicle pose to the motion control layer of the parking robot.

[0059] Based on a multi-view vision array and LiDAR used in a parking robot, multi-view 2D images and 3D point clouds of the vehicle are acquired at the same time stamp. Point cloud sampling, adjustment, and coordinate alignment are then performed to obtain aligned image and point cloud data, including:

[0060] Based on the hardware trigger pulse sent by the hardware synchronous clock source, multi-view vehicle 2D images and vehicle 3D point clouds at the same timestamp are acquired.

[0061] In non-initial state, obtain the physical distance from the target vehicle to the parking robot in the previous frame; in initial state, use the preset default physical distance.

[0062] The point cloud sampling retention rate is calculated based on physical distance, and the vehicle's 3D point cloud is sampled and adjusted according to the point cloud sampling retention rate.

[0063] Extract the static environmental features of the parking lot, calculate the extrinsic parameter matrix between the multi-view vision array and the lidar, and project the pixel coordinates of the multi-view vehicle 2D images onto the coordinate system of the vehicle 3D point cloud based on the extrinsic parameter matrix to obtain the aligned image and point cloud data.

[0064] The point cloud sampling retention rate is calculated as follows: based on the physical distance from the target vehicle to the parking robot in the previous frame, the preset optimal operating distance, and the preset distance attenuation coefficient, the point cloud sampling retention rate is calculated, and the point cloud sampling retention rate is not lower than the preset minimum point cloud retention threshold.

[0065] This embodiment is applicable to the parking robot performing posture recognition on vehicles to be moved or guided in parking lot lanes; the parking robot body is equipped with a multi-view vision array, LiDAR, hardware synchronization clock source and computing processing unit;

[0066] The multi-view vision array consists of at least multiple cameras distributed on the front and sides of the robot to acquire two-dimensional images of the vehicle from different directions; the lidar is used to output a three-dimensional point cloud of the vehicle surface.

[0067] A hardware synchronization clock source sends a unified trigger pulse to the camera and LiDAR, enabling each sensor to output data with the same timestamp within the same acquisition cycle; the computing and processing unit is used to perform subsequent sampling adjustment, coordinate alignment, feature extraction, multi-frame processing, and pose output.

[0068] At the start of an acquisition cycle, the hardware synchronization clock source emits a hardware trigger pulse, the multi-view vision array outputs a multi-view 2D image of the vehicle, and the LiDAR outputs a 3D point cloud of the vehicle; the same timestamp indicates that the image and point cloud both correspond to the trigger results within the same acquisition cycle.

[0069] After receiving the image and point cloud, the computing and processing unit first determines whether it is in the initial state; the initial state is the state in which the current recognition process has not yet obtained the position of the center point of the target vehicle in the previous frame.

[0070] In the initial state, the calculation and processing unit uses the preset default physical distance as the distance input; in the non-initial state, the calculation and processing unit reads the coordinates of the target vehicle's center point that have been saved in the previous frame, and calculates the physical distance from the target vehicle's center point to the parking robot in the current coordinate system, based on the parking robot's current position; the physical distance is the actual spatial distance between the parking robot and the target vehicle.

[0071] After obtaining the physical distance, the computing unit begins to perform point cloud sampling adjustment; the point cloud sampling retention rate is the proportion of points that need to be retained in the currently acquired vehicle 3D point cloud.

[0072] When the distance increases, the number of points should be appropriately reduced to reduce the computational burden, while maintaining the minimum density required for vehicle contour recognition. In specific calculations, the difference between the physical distance from the center point of the target vehicle in the previous frame to the parking robot and the preset optimal working distance of the parking robot should be calculated first, and the larger value between this difference and zero should be taken as the distance deviation.

[0073] When the vehicle is within the optimal operating distance, the distance deviation is zero and no additional attenuation is applied; when the physical distance between the target vehicle and the parking robot exceeds the preset optimal operating distance, the distance deviation increases as the difference between the two increases.

[0074] The calculation and processing unit subtracts the product of the preset distance attenuation coefficient and the distance deviation from 1.0 to obtain the attenuation retention rate. The distance attenuation coefficient is calibrated by the divergence angle of the lidar beam and the commonly used operating distance range of the parking robot, and is used to reflect the degree of decrease in point cloud density utilization as the distance increases.

[0075] Then compare the attenuation retention rate with the preset minimum point cloud retention threshold, and take the larger value of the two as the point cloud sampling retention rate;

[0076] The optimal working distance is set to the center value corresponding to the preset grasping or guiding working distance range of the parking robot. The minimum point cloud retention threshold can be set to the lower limit ratio that can guarantee contour recognition. The distance attenuation coefficient can be determined by the calibration results of the contour recognition success rate at different distances.

[0077] After the computing and processing unit samples and adjusts the current vehicle 3D point cloud according to the above retention rate, it continues to perform coordinate alignment between the image and the point cloud;

[0078] The extrinsic parameter matrix represents the coordinate transformation relationship between the multi-view array and the lidar, and is used to map the pixel observation results in the multi-view array to the coordinate system of the lidar.

[0079] The computational processing unit extracts static environmental features from the parking lot, such as structural features that do not change with vehicle movement, like the edges of load-bearing columns and ground parking lines.

[0080] Since these features can be observed in both images and point clouds, the computational processing unit can use the same set of static features to establish the correspondence between the visual coordinate system and the radar coordinate system and calculate the extrinsic parameter matrix.

[0081] The pixel coordinates in the multi-view vehicle 2D image are projected onto the coordinate system of the vehicle 3D point cloud according to the extrinsic parameter matrix; after this projection, each image feature can correspond to the point cloud contour under a unified spatial reference, forming aligned image and point cloud data.

[0082] After obtaining the aligned image and point cloud data, the computational processing unit inputs them into the feature extraction network. The feature extraction network includes an image branch and a point cloud branch. The image branch is responsible for extracting texture and edge information from multi-view vehicle 2D images, and the point cloud branch is responsible for extracting spatial geometry and depth information from vehicle 3D point clouds.

[0083] The results of the image branch output and the point cloud branch output are then unified to the same coordinate reference, and the information is stitched together in the channel dimension to obtain combined data containing texture information and depth information.

[0084] The feature extraction network is a jointly trained two-stream neural network; the image branch uses a pre-trained fifty-layer residual network as its backbone, with the input being... The color image tensor is output as a feature map that is unfolded into a dimension of . Two-dimensional feature vectors;

[0085] The point cloud branch employs a multi-level point cloud neural network based on metric space sampling and grouping, such as the PointNet++ network architecture, with the input being... The 3D point cloud coordinate matrix, after multi-level local feature extraction, outputs a dimension of... 3D feature vectors;

[0086] The network is trained in a supervised manner using a training set containing manually annotated vehicle 3D bounding boxes and key points. Its loss function is a weighted combination of key point pixel-level smoothing L1 loss and 3D point cloud chamfer distance loss.

[0087] The combined data carries both vehicle exterior texture and spatial depth information, serving as input for multi-frame weighting and pose solving;

[0088] After generating the combined data for the current frame, the computational processing unit writes the combined data into the buffer. The buffer uses a sliding method to store the combined data of multiple consecutive frames, thereby providing timing input for subsequent multi-frame weighted processing.

[0089] In a specific application, when a parking robot approaches a parked vehicle along the aisle, the robot first uses a hardware-synchronized clock source to synchronously trigger a multi-view vision array and LiDAR to acquire vehicle images and point clouds.

[0090] Before historical vehicle center point records are formed, the computing unit calls the default physical distance to complete the initial sampling. As subsequent collection cycles continue, the system switches to calling the physical distance between the vehicle center point and the robot in the previous frame to dynamically update the point cloud retention rate.

[0091] If the difference between the physical distance between the vehicle and the robot and the optimal working distance is less than a preset distance threshold, then the point cloud is retained at a ratio greater than the preset point cloud retention rate threshold for identifying tire and bumper edges.

[0092] If the distance increases, the retention rate is reduced but not lower than the minimum threshold to control the processing burden; then the extrinsic parameter matrix is ​​updated using the stable column edges and ground markings in the parking lot to complete the unified coordinate alignment between the multi-view image and the point cloud, and the alignment result is sent into the feature extraction network.

[0093] When static environmental features are partially obscured during a certain acquisition cycle, the computational processing unit can continue to calculate the extrinsic parameter matrix using the column edges or ground markings that can still be stably identified.

[0094] If the number of static environmental features available for calculation is insufficient, the extrinsic parameter matrix obtained in the previous acquisition cycle is retained in the current projection, and updated after sufficient static features are recovered in subsequent cycles. This avoids sudden mismatch between the image and the point cloud due to occlusion in a single acquisition cycle, thereby maintaining the continuity and availability of subsequent combined data.

[0095] In a preferred embodiment of the present invention, the aligned image and point cloud data are input into a feature extraction network containing image branches and point cloud branches for information extraction and splicing to obtain combined data containing texture information and depth information, including:

[0096] In the image branch of the feature extraction network, two-dimensional pixel information of wheel grounding points, the outermost edge of the vehicle's exterior rearview mirror, door handles, and charging ports or fuel tank caps are extracted from multi-view vehicle two-dimensional images.

[0097] In the point cloud branch of the feature extraction network, information on the curvature change lines of the front and rear bumpers of the vehicle and the cylindrical geometry information of the wheel area are extracted from the vehicle's 3D point cloud, and the position of the wheel center point is calculated based on the cylindrical geometry information.

[0098] Based on the pre-calibrated extrinsic matrix between the multi-view vision array and the lidar, the two-dimensional pixel information is stitched together with the curvature change line information of the front and rear bumpers of the vehicle, the cylindrical geometry information and the position of the wheel center point in the channel dimension to obtain combined data containing texture information and depth information, and the information points in the combined data are defined as vehicle marker points.

[0099] Cache combined data from multiple consecutive frames, and perform multi-frame weighted processing based on the combined data to obtain a processed set of 3D points, including:

[0100] Create a sliding window and cache consecutive windows. Combined data of frames, Positive integers greater than or equal to 5; for consecutive... Add timestamps to vehicle markers in the combined data of frames;

[0101] Calculate the inter-frame weight between the current frame and historical frames based on the corresponding point matching score between the current frame and historical frames; calculate the inter-frame weight based on the continuous inter-frame weight pairs. The combined data of the frames are weighted and summed to obtain the processed three-dimensional point set.

[0102] The displacement compensation of the wheel center point position in historical frames is performed based on the chassis movement speed of the parking robot to obtain the predicted wheel center point position; the spatial distance deviation between the wheel center point position in the current frame and the predicted wheel center point position is extracted, and the inter-frame weight is determined by combining the corresponding point matching score between the current frame and the historical frames.

[0103] This implementation focuses on how to extract parking operation-related features from aligned images and point clouds, and how to use continuous multi-frame data to suppress single-frame jumps; the execution entity is still the computing and processing unit of the parking robot, and the input data is image and point cloud data that are already under a unified coordinate reference;

[0104] Before feature extraction begins, the computational processing unit first sends the multi-view vehicle 2D images from the same acquisition cycle into the image branch of the feature extraction network, and sends the corresponding vehicle 3D point cloud into the point cloud branch; the image branch is the network path for processing vehicle surface texture, edges and local component appearance.

[0105] Point cloud branches refer to network paths that specifically process the three-dimensional contours and local geometry of vehicles. Since parking robots work in narrow parking spaces and complex lighting environments, relying solely on the vehicle bounding box is insufficient to meet the requirements for grasping and guidance accuracy. Therefore, this implementation does not use the vehicle contour as the only recognition result, but extracts features from local parts of the vehicle that are correlated with the posture judgment based on preset parameters.

[0106] The image branch performs component-level recognition on multi-view vehicle 2D images, extracting 2D pixel information of wheel contact points, the outermost edge of the vehicle's exterior rearview mirror, door handles, and charging ports or fuel tank caps; the wheel contact point is the pixel area corresponding to the position where the tire contacts the ground in the image;

[0107] The outermost edge of the exterior rearview mirror reflects the vehicle's outermost lateral expansion area; door handles and charging ports or fuel tank caps can serve as stable texture markers on the side of the vehicle body.

[0108] After the image branch outputs the two-dimensional pixel coordinates corresponding to these features, the computational processing unit temporarily stores them as a two-dimensional feature set for the current frame.

[0109] In parallel, the point cloud branch performs geometric feature extraction on the vehicle's 3D point cloud; it identifies curvature change lines from the front and rear areas of the vehicle body where the curvature of the bumper shape exceeds a preset change threshold.

[0110] Curvature change line information refers to the spatial broken line or boundary line formed when the surface of the vehicle body transitions from a flat area to a protruding or inward area;

[0111] Extract cylindrical geometric information in the area where the wheel is located; since the outer contour of the tire usually appears as a part of an approximate cylindrical surface in the point cloud, the computational processing unit can use this geometric feature to determine the range of the wheel and calculate the position of the wheel center point based on this.

[0112] Specifically, the computational processing unit performs spatial cylindrical surface fitting on the three-dimensional point cloud of the vehicle in the area where the wheel is located, obtains the central axis of the fitted cylinder, and takes the midpoint of the central axis as the position of the wheel center point.

[0113] The position of the wheel center point is a key three-dimensional reference point for subsequent wheelbase, wheel track, inter-frame consistency judgment and initial pose estimation.

[0114] After obtaining two-dimensional pixel information and three-dimensional geometric information in the image branch and point cloud branch respectively, the computational processing unit calls the pre-calibrated extrinsic matrix between the multi-view vision array and the lidar to convert the two-dimensional pixel information to the point cloud coordinate reference, so that image features such as wheel grounding point, rearview mirror edge point, door handle, and charging port or fuel tank cap can correspond to the bumper curvature change line, wheel cylindrical geometric information and wheel center point position under the same spatial reference.

[0115] The above information is stitched together in the channel dimension to obtain combined data containing both texture and depth information. In this embodiment, the information points in the combined data are defined as vehicle marker points. Vehicle marker points are a set of feature points that complete the fusion of the image and point cloud and reflect the vehicle's posture and contour boundary.

[0116] This set does not require all vehicles to have exactly the same number of features, as long as it contains enough key points to support subsequent pose calculations.

[0117] After obtaining the vehicle marker points of the current frame, the computational processing unit establishes a sliding window buffer continuum. The combined data of frames, where It is a positive integer greater than or equal to 5;

[0118] Each time the sliding window receives a new frame of combined data, it removes the oldest frame from the buffer, thus keeping the window length constant.

[0119] For each frame of combined data within the sliding window, the computing unit adds a timestamp to the vehicle markers within it;

[0120] The purpose of timestamp marking is to clarify the acquisition period to which each marker belongs, so that the current frame can be distinguished from the historical frame when comparing the current frame and the historical frame, and to perform position compensation in conjunction with the chassis movement speed;

[0121] The computational processing unit calculates the inter-frame weights between the current frame and each historical frame based on the preset corresponding point matching score and spatial position change constraints.

[0122] In a set of consecutive frames, let the current frame be the [number]th frame. Frame, a certain historical frame is the first The processing unit first reads the chassis motion velocity vector of the parking robot, and then combines the chassis motion velocity vector with the frame interval number. Multiplying the displacement compensation vector by the preset inter-frame time interval yields the displacement compensation vector.

[0123] The displacement compensation amount is the change in relative position caused by the movement of the parking robot itself, under the assumption that the target vehicle is stationary.

[0124] The computing processing unit will... The predicted wheel center point position is obtained by adding the displacement compensation amount to the wheel center point position recorded in the first frame; the predicted wheel center point position is then calculated. The squared Euclidean distance between the position of the wheel center point in the frame and the predicted position of the wheel center point is used as the squared distance deviation.

[0125] After obtaining the squared distance deviation, the computational processing unit calls the natural constant. The exponential function of is multiplied by the corresponding point matching score to obtain the th . Frame and the Inter-frame weights between frames;

[0126] The negative exponent term in the exponential function is the square of the distance deviation divided by twice the square of the preset ranging noise standard deviation; the corresponding point matching score represents the degree of matching between the current frame and the vehicle markers of the same type in the historical frames.

[0127] In the specific calculation, the computing processing unit counts the total number of valid vehicle markers extracted from the current frame and historical frames under the same type of features, as well as the number of overlapping markers that can be paired after chassis motion compensation.

[0128] The number of overlapping marker points is divided by the total number of valid vehicle marker points to obtain a quantitative ratio value between 0 and 1, which is used as the corresponding point matching score; the standard deviation of ranging noise comes from the offline calibration results of the fusion sensor and is used to represent the normal ranging fluctuation range of the system.

[0129] Based on chassis movement speed and inter-frame time interval The displacement compensation amount is calculated as follows: Based on the predicted wheel center point position and the The center point of the wheel in the frame Calculate the square of the distance deviation:

[0130]

[0131] Combine corresponding points to match scores and the standard deviation of ranging noise Calculate the inter-frame weights:

[0132]

[0133] After the computational processing unit calculates the inter-frame weights between each historical frame and the current frame within the sliding window, it assigns weights to consecutive frames. The frame-combined data is weighted and summed to output a processed set of 3D points.

[0134] The processed 3D point set is the result of 3D marker points obtained after multi-frame temporal correlation and weighted fusion;

[0135] In a specific application, when the parking robot moves along the side of the vehicle and gradually adjusts the gripping or lifting position, the camera may be affected by the reflection of the vehicle body during certain acquisition cycles, causing the identification position of the wheel contact point in the image to shift. The lidar may also have discrete points in the local depth due to the reflection of the wheel rim. If the attitude is calculated based on only the result of this single frame, it may cause the judgment of the vehicle's lateral boundary to be unstable.

[0136] In this embodiment, the calculation and processing unit takes the acquisition period as the current frame, retrieves the vehicle marker points of the previous few frames from the sliding window, performs displacement compensation on the historical wheel center point position according to the chassis movement speed, and then calculates the weight by combining the corresponding point matching score of each frame with the current frame.

[0137] When the distance deviation between the current observation point and the compensated historical point is greater than the preset distance difference threshold, the corresponding inter-frame weight is reduced; the historical observation points with better consistency have higher weights, and finally a smooth three-dimensional point set is obtained after weighted summation; in this way, the input data used for subsequent vehicle model matching and pose estimation are more stable.

[0138] When some vehicle markers are temporarily missing in a certain acquisition cycle, the calculation and processing unit can still use the same type of markers that already exist in the other historical frames within the window and have higher weights to participate in the weighting, so as to maintain the continuous output of the processed three-dimensional point set.

[0139] As long as a sufficient number of valid vehicle markers remain within the sliding window, subsequent vehicle model comparison and pose calculation can continue, thereby improving the parking robot's ability to operate continuously in complex environments.

[0140] In a preferred embodiment of the present invention, ambient brightness values ​​are extracted based on multi-view vehicle 2D images. The processed 3D point set is then compared with the vehicle model database and rigid body registration is performed using the ambient brightness values ​​to obtain the initial vehicle pose, including:

[0141] The measured wheelbase and measured track width are calculated based on the wheel center point positions in the processed 3D point set, and then compared and retrieved in a preset vehicle model database to obtain the standard vehicle model of the target vehicle; the ambient brightness value is calculated based on the average gray value of the multi-view vehicle 2D image.

[0142] Calculate the difference or ratio between the ambient brightness value and the preset illumination threshold, and map it to a weighted ratio of visual weight and point cloud weight;

[0143] Vehicle pose determination is based on 3D point cloud computing LiDAR and vehicle pose determination is based on multi-view vehicle 2D images.

[0144] The initial pose of the vehicle in three-dimensional space is calculated by weighted summation of the pose determined by lidar and vision based on a weighted ratio. The initial pose of the vehicle in three-dimensional space is then fine-tuned by rigid body registration based on a standard vehicle model to obtain the initial pose of the vehicle.

[0145] The initial vehicle pose is combined with the chassis wheel odometer data of the parking robot for overall pose correction and abnormal pose removal. The corrected vehicle pose is then output as the target vehicle posture visual recognition result to the motion control layer of the parking robot, including:

[0146] The initial vehicle pose and chassis wheel odometer data are input into a preset pose correction program. The parking robot pose calculated based on the chassis wheel odometer data is used as the pose reference point. The processed three-dimensional point set is used as the reference road sign point for trajectory closed-loop correction to obtain the optimized vehicle pose.

[0147] Calculate the yaw angle change and lateral displacement change of the optimized vehicle pose between two adjacent frames; if the yaw angle change is greater than the preset yaw angle change threshold or the lateral displacement change is greater than the preset lateral displacement change threshold, discard the optimized vehicle pose of the current frame and use the Kalman filter to predict the pose of the next frame based on the historical pose state.

[0148] If the change in yaw angle is less than or equal to the preset threshold for yaw angle change, and the change in lateral displacement is less than or equal to the preset threshold for lateral displacement change, then the current optimized vehicle pose will be output as the visual recognition result of the target vehicle pose to the motion control layer of the parking robot.

[0149] Please see Figure 2 A vehicle posture visual recognition system for parking robots includes: an acquisition and alignment module, which is used to acquire multi-view two-dimensional images and three-dimensional point clouds of the vehicle at the same time stamp based on the multi-view visual array and lidar of the parking robot, and to perform point cloud sampling adjustment and coordinate alignment to obtain aligned image and point cloud data.

[0150] The information processing and stitching module is used to input the aligned image and point cloud data into a feature extraction network containing image branches and point cloud branches for information extraction and stitching, to obtain combined data containing texture information and depth information.

[0151] The weighted processing and recognition module is used to cache the combined data of multiple consecutive frames and perform multi-frame weighted processing based on the combined data of multiple consecutive frames to obtain the processed three-dimensional point set; the ambient brightness value is extracted based on the multi-view vehicle two-dimensional image, and the vehicle model comparison and rigid body registration are performed on the processed three-dimensional point set in combination with the preset vehicle model library and the ambient brightness value to obtain the initial pose of the vehicle.

[0152] The correction and output module is used to combine the initial vehicle pose with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal, and output the corrected vehicle pose as the target vehicle posture visual recognition result to the motion control layer of the parking robot.

[0153] This implementation focuses on how to further complete vehicle model comparison, initial pose solving, overall pose correction and result output after obtaining the processed three-dimensional point set, and explains the collaborative relationship between system modules;

[0154] The main components still include the parking robot's multi-view vision array, lidar, chassis wheel-type odometer, and computing processing unit;

[0155] After the processed 3D point set is generated, the calculation processing unit extracts the wheel center point position from it; the measured wheelbase is the actual distance between the center points of the front and rear wheels on the same side;

[0156] The measured wheelbase is the actual distance between the center points of the left and right wheels on the same axle; the calculation and processing unit calculates the measured wheelbase and measured track of the current vehicle based on the wheel center point positions in the processed three-dimensional point set.

[0157] Since the three-dimensional point set has undergone weighted processing over multiple consecutive frames, the points used to calculate the wheelbase and track width are relatively stable, which can reduce the impact of single-frame noise on vehicle model judgment.

[0158] After calculating the measured wheelbase and measured track width, the calculation and processing unit performs a search and comparison in the preset vehicle model database. The preset vehicle model database is a data set that pre-stores various standard structural parameters and standard three-dimensional topological relationships of vehicles, including the standard wheelbase, standard track width, rearview mirror unfolding or folding width, and feature point layout of each vehicle model.

[0159] The calculation and processing unit compares the measured wheelbase and measured track width with the standard parameters in the vehicle model library one by one, filters out the vehicle models that match the current observation results, and determines the standard vehicle model of the target vehicle. This standard vehicle model is used to complete the other side contour or body boundary information that is missing due to the limitation of the viewing angle during subsequent attitude fine-tuning.

[0160] After completing vehicle model matching, the computing and processing unit extracts the ambient brightness value from the multi-view vehicle 2D images. The ambient brightness value is a light characterization quantity calculated from the average gray value of the multi-view vehicle 2D images. The computing and processing unit can calculate the average gray value of each view image separately, and then take the comprehensive result as the ambient brightness value of the current acquisition cycle.

[0161] The ambient brightness value is compared with a preset illumination threshold, the difference or ratio is calculated, and it is mapped to a weighted ratio of visual weight and point cloud weight.

[0162] The illumination threshold is a preset illumination boundary value, which is pre-calibrated based on the image test indicators of the camera under different illumination conditions.

[0163] When the ambient brightness value is less than the illumination threshold, increase the point cloud weighting ratio and decrease the visual weighting ratio; when the ambient brightness value is in a higher range, the visual texture features are clearer, and the visual weight can be appropriately increased.

[0164] Calculate ambient brightness value With light threshold Ambient brightness difference:

[0165]

[0166] If the difference in ambient brightness Then set the visual weight:

[0167]

[0168] Set point cloud weights ,in The preset adjustment constant, It is a natural constant; if the ambient brightness difference is... Then set the visual weight. And set point cloud weights ;

[0169] After obtaining the weighting ratio, the computing unit determines the pose based on the vehicle's three-dimensional point cloud computing LiDAR and calculates the pose based on the multi-view vehicle two-dimensional image.

[0170] LiDAR determines the vehicle's spatial position and orientation by calculating the contour and geometric features in a 3D point cloud.

[0171] Visual pose determination refers to the determination of a vehicle’s spatial position and orientation using texture marks such as wheel contact points, rearview mirror edge points, door handles, and charging ports or fuel tank caps in an image.

[0172] The computational processing unit performs a weighted sum of the pose determined by the lidar and the pose determined by vision based on the aforementioned visual weights and point cloud weights to obtain the initial pose of the vehicle in three-dimensional space.

[0173] After the initial pose of the vehicle in three-dimensional space is obtained, the computing unit retrieves the previously matched standard vehicle model and performs rigid body registration fine-tuning on the initial pose.

[0174] Rigid body registration fine-tuning employs an iterative nearest-point algorithm for spatial alignment, setting the key feature point set of the standard vehicle model as follows: The current set of measured and processed 3D points is as follows: Iteratively solve the rotation matrix Translation vector To minimize the objective error function:

[0175]

[0176] in, For point set Zhongyu The point with the closest Euclidean distance is used to output the fine-tuned initial pose of the vehicle when the error converges.

[0177] After obtaining the initial vehicle pose, the calculation and processing unit continues to input the initial vehicle pose and the data from the chassis wheel odometer of the parking robot into a preset pose correction program.

[0178] The chassis wheel odometer data consists of displacement and steering data output by the chassis walking mechanism of the parking robot, which is used to calculate the parking robot's own pose changes.

[0179] The pose correction program is used to comprehensively correct the vehicle's pose by combining the initial vehicle pose, the robot's own pose, and the processed 3D point set with reference landmarks.

[0180] In specific processing, the computing unit uses the parking robot pose calculated based on the chassis wheel odometer data as the pose reference point, and the processed three-dimensional point set as the reference road sign point to correct the trajectory relationship in the continuous acquisition cycle, so as to obtain the optimized vehicle pose.

[0181] In practice, the preset pose correction program is built based on the pose graph optimization algorithm. The system uses the odometer-measured pose of the parking robot at each moment and the three-dimensional global coordinates of the vehicle feature points as graph nodes, and the adjacent relative pose change matrix output by the odometer and the relative measurement values ​​of feature points calculated by visual radar fusion as edge constraints of the graph.

[0182] A least-squares nonlinear error function is constructed, and the global error is iteratively minimized using the Levonberg-Marquardt algorithm to output the optimized vehicle pose.

[0183] After obtaining the optimized vehicle pose, the computational processing unit further performs abnormal pose removal; specifically, it calculates the yaw angle change and lateral displacement change of the optimized vehicle pose between two adjacent frames.

[0184] The yaw angle change is the degree of change in the vehicle's heading angle between two adjacent frames; the lateral displacement change is the degree of change in the vehicle's position in the lateral direction.

[0185] Therefore, the calculation and processing unit compares the current yaw angle change with the preset yaw angle change threshold, and at the same time compares the lateral displacement change with the preset lateral displacement change threshold.

[0186] The threshold for yaw angle change is a parameter determined by multiplying the maximum allowable turning angular velocity of the parking robot by the time interval of a single acquisition cycle and superimposing the maximum angle measurement error of the sensor.

[0187] The threshold for lateral displacement change is determined by multiplying the maximum permissible lateral speed of the parking robot by the time interval of a single acquisition cycle, and then adding the maximum ranging error of the sensor.

[0188] If the change in yaw angle exceeds the threshold for the change in yaw angle, or the change in lateral displacement exceeds the threshold for the change in lateral displacement, it means that the identified state transition has exceeded the limit of the physical motion characteristics of the system. Therefore, the optimized vehicle pose of the current frame is determined to be abnormal and is discarded. The Kalman filter is then used to predict the pose of the next frame based on the historical pose state.

[0189] If neither of them exceeds the threshold, the current optimized vehicle pose will be output as the target vehicle pose visual recognition result to the motion control layer of the parking robot.

[0190] The specific processing logic for predicting the next frame's pose based on historical pose states using a Kalman filter is as follows: Establish a state vector containing the vehicle's center position and yaw angle. ;

[0191] When the current frame is determined to be abnormal and unreliable, the filter measurement and update step is skipped, and only the state transition equation is used:

[0192]

[0193] Perform time-updated predictions, where the state transition matrix... Based on the construction of a two-dimensional planar uniform linear motion and uniform turning model, the predicted state is extracted. In The component is used as the predicted smooth attitude output for the next acquisition cycle;

[0194] In terms of system implementation, the acquisition and alignment module, information processing and splicing module, weighted processing and recognition module, and correction and output module can all be deployed in the same computing unit, or they can be allocated and executed in communicable processing resources according to their functions.

[0195] The acquisition and alignment module is responsible for receiving synchronous data from the multi-view vision array and LiDAR, and completing point cloud sampling adjustment and coordinate alignment; the information processing and stitching module is responsible for extracting features from the image branch and point cloud branch and generating combined data.

[0196] The weighted processing and recognition module is responsible for buffering continuous multi-frame combined data, calculating inter-frame weights, multi-frame weighted processing, extracting ambient brightness values, searching the vehicle model database, and rigid body registration to obtain the initial vehicle pose; the correction and output module is responsible for combining chassis wheel odometer data to perform overall pose correction, eliminate abnormal poses, and send the corrected vehicle pose to the motion control layer of the parking robot.

[0197] In a specific application, after the parking robot enters the parking space area where the vehicle to be operated is located, the aforementioned modules are executed sequentially.

[0198] The acquisition and alignment module outputs aligned image and point cloud data. The information processing and stitching module extracts vehicle marker points such as wheel ground contact points, wheel center points, and bumper curvature change lines. The weighted processing and recognition module weights the results of multiple consecutive frames to obtain a stable set of three-dimensional points.

[0199] This module calculates the measured wheelbase and measured track width based on the position of the wheel center point, and selects the most matching standard vehicle model from the vehicle model library;

[0200] Then, the ambient brightness value is obtained based on the average gray value of the multi-view images. The visual weight and point cloud weight are determined according to the relationship between the ambient brightness value and the illumination threshold. The weighted sum of the LiDAR-measured pose and the visual-measured pose is completed. Rigid body registration is then performed with the help of a standard vehicle model to obtain the initial pose of the vehicle.

[0201] The correction and output module then sends the initial pose and chassis wheel odometer data into the pose correction program to obtain the optimized vehicle pose, and judges the yaw angle change and lateral displacement change in adjacent frames; if the current result is normal, it is output to the motion control layer.

[0202] If an abnormal transition occurs, the current result is discarded and the Kalman filter is invoked to output the predicted attitude, thereby maintaining the continuity of the control input.

[0203] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A vehicle posture visual recognition method for parking robots, characterized in that, Includes the following steps: The parking robot uses a multi-view vision array and LiDAR to acquire multi-view 2D images and 3D point clouds of the vehicle at the same time stamp, and performs point cloud sampling adjustment and coordinate alignment to obtain aligned image and point cloud data. The aligned image and point cloud data are input into a feature extraction network containing image branches and point cloud branches for information extraction and splicing to obtain combined data containing texture information and depth information. The combined data of multiple consecutive frames is cached, and multi-frame weighted processing is performed based on the combined data of multiple consecutive frames to obtain a processed three-dimensional point set. Based on the multi-view vehicle 2D image, the ambient brightness value is extracted. The vehicle model is then compared and rigid body registration is performed on the processed 3D point set using the preset vehicle model library and the ambient brightness value to obtain the initial vehicle pose. The initial vehicle pose is combined with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal. The corrected vehicle pose is then output as the target vehicle posture visual recognition result to the motion control layer of the parking robot. The combined data from multiple consecutive frames is cached, and multi-frame weighted processing is performed based on the combined data to obtain a processed set of three-dimensional points, including: Create a sliding window and cache consecutive windows. Combined data of frames, It is a positive integer greater than or equal to 5; For the continuous Add timestamps to vehicle markers in the combined data of frames; Calculate the inter-frame weight between the current frame and historical frames based on the corresponding point matching score between the current frame and historical frames; Based on the inter-frame weights, the continuous The combined frame data is weighted and summed to obtain the processed 3D point set; Based on the multi-view vehicle 2D images, ambient brightness values ​​are extracted. Then, the processed 3D point set is compared with the vehicle model database and the ambient brightness values, and rigid body registration is performed to obtain the initial vehicle pose, including: Based on the wheel center point position in the processed three-dimensional point set, the measured wheelbase and measured wheel track are calculated, and then searched and compared in the preset vehicle model library to obtain the standard vehicle model of the target vehicle. The ambient brightness value is calculated based on the average grayscale value of the multi-view vehicle two-dimensional image; Calculate the difference or ratio between the ambient brightness value and the preset illumination threshold, and map it to a weighted ratio of visual weight and point cloud weight; The vehicle pose is determined by a three-dimensional point cloud computing LiDAR and by a multi-view vehicle two-dimensional image computational vision. The initial pose of the vehicle in three-dimensional space is calculated by weighting and summing the pose measured by the lidar and the pose measured by the vision based on the weighting ratio, and then by performing rigid body registration fine-tuning on the initial pose of the vehicle in three-dimensional space based on the standard vehicle model to obtain the initial pose of the vehicle.

2. The vehicle posture visual recognition method for parking robots according to claim 1, characterized in that, Based on a multi-view vision array and LiDAR used in a parking robot, multi-view 2D images and 3D point clouds of the vehicle are acquired at the same time stamp. Point cloud sampling, adjustment, and coordinate alignment are then performed to obtain aligned image and point cloud data, including: Based on the hardware trigger pulse sent by the hardware synchronous clock source, the multi-view vehicle 2D image and the vehicle 3D point cloud at the same timestamp are acquired. In the non-initial state, the physical distance from the target vehicle to the parking robot in the previous frame is obtained; in the initial state, a preset default physical distance is used. The point cloud sampling retention rate is calculated based on the physical distance, and the vehicle's 3D point cloud is sampled and adjusted according to the point cloud sampling retention rate. Extract the static environmental features of the parking lot, calculate the extrinsic parameter matrix between the multi-view vision array and the lidar, and project the pixel coordinates of the multi-view vehicle 2D image onto the coordinate system of the vehicle 3D point cloud based on the extrinsic parameter matrix to obtain the aligned image and point cloud data.

3. The vehicle posture visual recognition method for parking robots according to claim 2, characterized in that, The point cloud sampling retention rate is calculated as follows: Based on the physical distance from the target vehicle to the parking robot in the previous frame, the preset optimal operating distance, and the preset distance attenuation coefficient, the point cloud sampling retention rate is calculated, and the point cloud sampling retention rate is not lower than the preset minimum point cloud retention threshold.

4. The vehicle posture visual recognition method for parking robots according to claim 1, characterized in that, The aligned image and point cloud data are input into a feature extraction network containing image branches and point cloud branches for information extraction and concatenation, resulting in combined data containing texture and depth information, including: In the image branch of the feature extraction network, two-dimensional pixel information of the wheel ground contact point, the outermost edge point of the vehicle's exterior rearview mirror, the door handle, and the charging port or fuel tank cap is extracted from the multi-view vehicle two-dimensional image. In the point cloud branch of the feature extraction network, the curvature change line information of the front and rear bumpers of the vehicle and the cylindrical geometric information of the wheel area are extracted from the three-dimensional point cloud of the vehicle, and the position of the wheel center point is calculated based on the cylindrical geometric information. Based on the pre-calibrated extrinsic matrix between the multi-view vision array and the lidar, the two-dimensional pixel information is spliced ​​with the curvature change line information of the front and rear bumpers of the vehicle, the cylindrical geometry information, and the position of the wheel center point in the channel dimension to obtain the combined data containing texture information and depth information, and the information points in the combined data are defined as vehicle marker points.

5. The vehicle posture visual recognition method for parking robots according to claim 1, characterized in that, Based on the chassis movement speed of the parking robot, displacement compensation is performed on the wheel center point position of the historical frame to obtain the predicted wheel center point position; the spatial distance deviation between the wheel center point position of the current frame and the predicted wheel center point position is extracted, and the inter-frame weight is determined by combining the corresponding point matching score between the current frame and the historical frame.

6. The vehicle posture visual recognition method for parking robots according to claim 1, characterized in that, The initial vehicle pose is combined with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal. The corrected vehicle pose is then output as the target vehicle pose visual recognition result to the motion control layer of the parking robot, including: The initial vehicle pose and the chassis wheel odometer data are input into a preset pose correction program. The parking robot pose calculated based on the chassis wheel odometer data is used as the pose reference point. The processed three-dimensional point set is used as the reference road sign point for trajectory closed-loop correction to obtain the optimized vehicle pose. Calculate the yaw angle change and lateral displacement change of the optimized vehicle pose between two adjacent frames. If the change in yaw angle is greater than a preset threshold for yaw angle change, or the change in lateral displacement is greater than a preset threshold for lateral displacement change, then the optimized vehicle pose of the current frame is discarded, and the pose of the next frame is predicted based on the historical pose state using a Kalman filter; if the change in yaw angle is less than or equal to a preset threshold for yaw angle change, and the change in lateral displacement is less than or equal to a preset threshold for lateral displacement change, then the current optimized vehicle pose is output as the target vehicle pose visual recognition result to the motion control layer of the parking robot.

7. A vehicle posture visual recognition system for parking robots, used to implement the vehicle posture visual recognition method for parking robots as described in any one of claims 1-6, characterized in that, include: The acquisition and alignment module is used to acquire multi-view 2D images and 3D point clouds of the vehicle at the same time stamp based on the multi-view vision array and lidar of the parking robot, and to perform point cloud sampling adjustment and coordinate alignment to obtain aligned image and point cloud data. The information processing and stitching module is used to input the aligned image and point cloud data into a feature extraction network containing image branches and point cloud branches for information extraction and stitching, to obtain combined data containing texture information and depth information. The weighted processing and recognition module is used to cache the combined data of multiple consecutive frames and perform multi-frame weighted processing based on the combined data of multiple consecutive frames to obtain a processed three-dimensional point set; the ambient brightness value is extracted based on the multi-view vehicle two-dimensional image, and the vehicle model comparison and rigid body registration are performed on the processed three-dimensional point set in combination with the preset vehicle model library and the ambient brightness value to obtain the initial pose of the vehicle; The correction and output module is used to combine the initial vehicle pose with the chassis wheel odometer data of the parking robot to perform overall pose correction and abnormal pose removal, and output the corrected vehicle pose as the target vehicle posture visual recognition result to the motion control layer of the parking robot.

Images