A computer vision-based mobile robot pallet recognition method

Abstract

The application discloses a kind of based on computer vision's mobile robot pallet identification method, it is related to robot vision perception technical field, including, obtain camera inside participate camera to robot base external parameter, collect color image and depth image and carry out distortion removal and alignment processing, generate near-ground mask based on depth image fitting ground plane, construct structure dimension candidate area in combination with depth gradient and brightness gradient, extract effective structure and weightedly solve pallet center pose and pose covariance, further calculate docking deviation and construct error uncertainty domain to complete docking determination;Through space constraint and multi-source information fusion, improve the accuracy of structure identification, through observation quality weighting and uncertainty quantification, enhance the stability of pose estimation, and realize the conservative evaluation of docking risk, improve the safety and reliability of docking.

Description

Technical Field

[0001] This invention relates to the field of robot vision perception technology, and in particular to a method for recognizing mobile robot pallets based on computer vision. Background Technology

[0002] With the development of intelligent logistics and warehouse automation, mobile robots are increasingly widely used in cargo handling, pallet docking, and autonomous operations. Current technologies for pallet recognition typically employ methods based on 2D image feature extraction, combined with edge detection, shape matching, or feature point matching to achieve target localization. Some solutions further introduce depth sensors, using RGB-D data fusion to obtain the pallet's 3D pose information. In engineering practice, steps such as image preprocessing, candidate region selection, and geometric model fitting are often performed to estimate the position and attitude parameters of the pallet's fork holes, thus providing pose information for the robot's docking control.

[0003] However, the aforementioned conventional methods still have certain limitations in complex operating environments: First, under conditions of occlusion, changes in lighting, or strong ground interference, relying solely on the detection of local edges or feature points can easily lead to fluctuations in structural visibility, thus affecting the stability of pose estimation; Second, in the pose solution process, most methods do not explicitly model and quantify the observation uncertainty, making it difficult to comprehensively consider error propagation and safety margins in the docking determination stage, thereby limiting their adaptability to high-precision docking scenarios. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a computer vision-based method for recognizing mobile robot pallets to address the problems of large fluctuations in structural visibility due to environmental influences and the lack of quantitative analysis of pose uncertainty in existing technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a computer vision-based method for recognizing mobile robot pallets, comprising: acquiring camera intrinsic parameters and camera-to-robot extrinsic parameters; acquiring color and depth images, and performing distortion correction and alignment processing; fitting a ground plane based on the depth image to generate a near-ground mask; constructing candidate regions for structural dimensions of the pallet's left fork, right fork, and leading edge guide edge based on the near-ground mask and combining depth and brightness gradients; generating visibility probability and boundary segment confidence within each candidate region, extracting observable boundary segments, calculating structural visibility probability and boundary coverage, and determining the effective structure set. The process involves: calling the pallet structure template, performing normalized soft correspondence matching on the observation boundary points of the effective structure, and constructing a geometric residual observation set; calculating the weighted solution of the center pose from the pallet to the camera based on the structural visibility probability and boundary coverage, and calculating the pose covariance of the pallet relative to the camera; transforming the center pose to the robot base coordinate system, and calculating the lateral deviation, height deviation, and yaw angle deviation between the fork hole centerline and the fork centerline; based on the lateral deviation, height deviation, and yaw angle deviation, combined with the pose covariance, constructing the error uncertainty domain, calculating the most unfavorable docking margin, and outputting the robot's docking decision for the pallet.

[0008] As a preferred embodiment of the computer vision-based mobile robot pallet recognition method of the present invention, the step of obtaining camera intrinsic parameters and camera-to-robot base extrinsic parameters includes: setting up a planar calibration board within the robot's observable area, acquiring multiple frames of calibration images of the planar calibration board at different distances, different viewpoints, and different postures; extracting the pixel coordinates of feature points in each frame of calibration images, and establishing the three-dimensional coordinates of the feature points in the calibration board coordinate system; constructing the reprojection error between the three-dimensional coordinates and the pixel coordinates, and solving the camera focal length parameters, principal point parameters, and distortion parameters by minimizing the reprojection error to obtain the camera intrinsic parameter matrix; under the condition of the planar calibration board's posture relative to the robot base, solving the posture of the calibration board relative to the camera using the PnP algorithm based on the three-dimensional coordinates of the feature points and the corresponding pixel coordinates; and performing matrix operations on the posture from the calibration board to the base and the solved posture from the calibration board to the camera according to the coordinate transformation link relationship to obtain the camera-to-robot base extrinsic parameter matrix.

[0009] As a preferred embodiment of the computer vision-based mobile robot pallet recognition method of the present invention, the distortion correction and alignment processing includes: establishing a distortion correction lookup table based on the intrinsic parameters and distortion parameters of the color channel and depth channel; resampling the color image and depth image using the distortion correction lookup table to obtain a distortion-corrected image; back-projecting the effective depth pixels in the depth image into three-dimensional points, and projecting them onto the color image coordinate system according to the extrinsic parameter relationship between the depth camera and the color camera to perform spatial alignment of the depth image and the color image.

[0010] As a preferred embodiment of the computer vision-based mobile robot pallet recognition method of the present invention, the generation of the near-ground mask includes: selecting candidate pixels in the lower half of the depth image; back-projecting the candidate pixels into a three-dimensional point set; performing plane fitting using a random sampling consistency method, selecting the plane with the most interior points as the initial ground plane; performing least-squares refinement on the initial ground plane, and performing orientation constraint verification based on the angle between the planar direction and the vertical direction; and generating the near-ground mask according to the distance threshold from the depth point to the ground.

[0011] As a preferred embodiment of the mobile robot pallet recognition method based on computer vision described in this invention, the construction of candidate regions for the structural dimensions of the left fork, right fork, and leading edge guide of the pallet includes: calculating the depth gradient magnitude and the brightness gradient magnitude within the set of effective pixels near the ground; identifying pixels with a depth gradient magnitude greater than or equal to a depth threshold as depth boundary bands; identifying pixels with a brightness gradient magnitude greater than or equal to a brightness threshold as texture boundary bands; taking the intersection of the depth boundary band and the texture boundary band as a gradient consistency candidate band; and constructing geometric windows for the left fork, right fork, and leading edge guide based on the gradient consistency candidate band, and taking the intersection to form candidate regions for each structural dimension.

[0012] As a preferred embodiment of the mobile robot pallet recognition method based on computer vision described in this invention, the generation of visibility probability and boundary segment confidence includes: cropping candidate regions of structural dimension into fixed-resolution image blocks; inputting the resolution image blocks and corresponding mask blocks into a joint feature network sharing a backbone network and a dual-output head structure; generating a pixel-level visibility probability field through the visibility output head; and generating a boundary segment confidence field with multiple discrete directions through the boundary output head.

[0013] As a preferred embodiment of the computer vision-based mobile robot pallet recognition method of the present invention, the construction of the geometric residual observation set includes: uniformly sampling the observable boundary segments of the effective structure to obtain observation boundary points; mapping the observation boundary points to normalized candidate region coordinates, calculating the Euclidean distance between the observed normalized boundary points and the template normalized boundary points and converting it into similarity weights; normalizing the similarity weights to form a soft correspondence weight matrix; and constructing the geometric residual set based on the projection positions of the template 3D structural points and the distance between the soft correspondence target points.

[0014] As a preferred embodiment of the mobile robot pallet recognition method based on computer vision described in this invention, the weighted solution of the center pose from the pallet to the camera includes: constructing a weighted objective function containing structural weights and a robust cost function by using a Lie group parameterization of the pose with six-dimensional minimum parameters; optimizing the weighted objective function by using an iterative reweighted least squares algorithm; after iterative convergence, constructing an information matrix based on the Jacobian matrix and the weight matrix; and inverting the information matrix to obtain the pose covariance matrix.

[0015] As a preferred embodiment of the mobile robot pallet recognition method based on computer vision described in this invention, the calculation of the lateral deviation, height deviation, and yaw angle deviation between the fork hole centerline and the fork centerline includes: converting the pallet center pose to the base coordinate system through the extrinsic parameters from the camera to the robot base; constructing the fork hole centerline and the fork centerline in the base coordinate system respectively, and calculating the offset of the fork hole centerline relative to the fork centerline in the horizontal lateral direction as the lateral deviation, calculating the difference between the fork hole center height and the fork height reference as the height deviation, and calculating the angle between the two centerlines in the horizontal plane as the yaw angle deviation.

[0016] As a preferred embodiment of the mobile robot pallet recognition method based on computer vision described in this invention, the output of the robot's docking determination for the pallet includes: mapping the pose covariance to a docking error covariance matrix based on the first-order linearization of the center pose parameters; constructing a three-dimensional ellipsoidal error uncertainty domain at a confidence level, and calculating the upper bound of the most unfavorable absolute values ​​of the lateral deviation, height deviation, and yaw angle deviation within the three-dimensional ellipsoidal error uncertainty domain; comparing the upper bound of the most unfavorable absolute values ​​with the corresponding allowable tolerances, and taking the minimum value of the remaining margin ratio as the most unfavorable docking margin; and outputting the docking determination based on the most unfavorable docking margin.

[0017] The beneficial effects of this invention are as follows: by fitting the ground plane and generating a near-ground mask, spatial constraints on candidate regions are achieved, thereby improving the accuracy of structure positioning; by fusing depth gradients and brightness gradients to construct candidate regions for structural dimensions, multi-source information synergistic enhancement is achieved, thereby improving the robustness of structure recognition; by performing weighted pose solving based on visibility probability and boundary coverage, adaptive adjustment of observation quality is achieved, thereby improving the stability of pose estimation; by constructing a pose covariance matrix, a quantitative expression of pose uncertainty is achieved, thereby enhancing the reliability of docking decisions; by establishing an error uncertainty domain and calculating the most unfavorable docking margin, a conservative assessment of docking risk is achieved, thereby improving docking safety. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a computer vision-based method for recognizing pallets on mobile robots.

[0020] Figure 2 A flowchart for camera calibration and RGB-D alignment.

[0021] Figure 3 A flowchart for constructing candidate regions for near-Earth mask and structural dimensions.

[0022] Figure 4 The flowchart is for effective structure, pose solving, and docking determination.

[0023] Figure 5 This is a diagram showing the relationship between the most unfavorable docking margin and the residual RMSE.

[0024] Figure 6 This is a residual RMSE distribution plot. Detailed Implementation

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0028] Reference Figures 1-6 This is one embodiment of the present invention, which provides a mobile robot pallet recognition method based on computer vision, including the following steps:

[0029] S1. Obtain the camera intrinsic parameters and the camera-to-robot extrinsic parameters.

[0030] Furthermore, a planar calibration plate with known geometric dimensions is fixed within the robot's observable area, wherein the calibration plate is a checkerboard pattern and the distance between adjacent corner points of the calibration plate is a known constant.

[0031] The camera is controlled to acquire color images at different distances, from different angles, and in different planar orientations to obtain a calibration image sequence.

[0032] For each frame of the calibration image, extract the set of pixel coordinates of the corner points of the calibration board.

[0033] At the same time, based on the geometric dimensions of the calibration plate, a set of planar three-dimensional coordinates of the geometric dimensions in the calibration plate coordinate system is established.

[0034] Furthermore, let the camera intrinsic parameters be... The distortion parameters are .

[0035] First transform the 3D points in the calibration plate coordinate system to the camera coordinate system to obtain Then normalize to obtain pixel coordinates , is represented as:

[0036] ;

[0037] ;

[0038] in, , , This represents the coordinates of a 3D point in the camera coordinate system. , Represents the normalized imaging plane coordinates.

[0039] Distortion-corrected normalized coordinates , is represented as:

[0040] ;

[0041] ;

[0042] ;

[0043] in, This represents the squared radial distance to the optical axis. , , Represents the radial distortion coefficient. , Indicates the tangential distortion coefficient. , This represents the normalized coordinates after distortion correction.

[0044] Pixel coordinate prediction value , is represented as:

[0045] ;

[0046] ;

[0047] in, , Represents the predicted pixel coordinates. , Indicates focal length. , This represents the pixel coordinates of the principal point.

[0048] The intrinsic and distortion parameters are solved by minimizing the reprojection error, i.e., minimizing the objective function, which is expressed as:

[0049] ;

[0050] in, Indicates the first The rotation matrix of the calibration board relative to the camera in the image. Indicates the first The translation vector of the calibration board relative to the camera in the image. Indicates the total number of calibration images. This indicates the number of feature points detected in each image. , Indicates the first The first image The true pixel coordinates of each feature point , Indicates the first The first image Predicted pixel coordinates of each feature point This represents the Euclidean norm.

[0051] After optimization, the camera intrinsic parameter matrix is ​​obtained. , is represented as:

[0052] ;

[0053] in, This represents the camera intrinsic parameter matrix.

[0054] Furthermore, a calibration plate for the camera's intrinsic parameters is fixedly mounted on the robot base, and the pose of the calibration plate relative to the robot base is a known constant, denoted as:

[0055] ;

[0056] in, Indicates from the calibration plate coordinate system to base coordinate system The homogeneous transformation matrix, Indicates from the calibration plate coordinate system to base coordinate system The rotation matrix, Indicates from the calibration plate coordinate system to base coordinate system The translation vector.

[0057] Images are acquired when the robot is stationary or scanning at low speed. The pixel coordinates of feature points on the calibration board are detected and combined with the obtained camera intrinsic parameters. Solve for the pose of the calibration board relative to the camera. .

[0058] in, Indicates from the calibration plate coordinate system to base coordinate system The homogeneous transformation matrix is ​​obtained by solving PnP, which involves finding the matrix that minimizes the reprojection error based on the 3D coordinates of the feature points on the calibration board in the calibration board coordinate system and the 2D coordinates in the image. and .

[0059] Furthermore, the extrinsic parameters from the camera to the robot base are defined as follows: The extrinsic parameters from the camera to the robot base are corrected for consistency via the coordinate link, as shown below:

[0060] ;

[0061] in, This represents the extrinsic parameter matrix from the camera to the base that we are looking for. This represents the transformation matrix from the calibration board to the camera obtained through PnP.

[0062] By rearranging the coordinate link consistency formula, we obtain the extrinsic parameter matrix, which is expressed as:

[0063] ;

[0064] in, The inverse of the transformation matrix from calibration board to camera is represented as:

[0065] ;

[0066] in, This represents the transpose of a rotation matrix. This indicates a reverse translation.

[0067] S2. Acquire color and depth images, and perform distortion correction and alignment processing.

[0068] Furthermore, when the robot reaches the sampling time in each control cycle, it simultaneously triggers the color imaging channel and the depth imaging channel to acquire one frame of data, obtaining one frame of color image and one frame of depth image respectively, and records a hardware timestamp for each frame of data.

[0069] Furthermore, the camera intrinsic parameters and distortion parameters of the color channel are read, the pixel coordinates of the color image are converted to the normalized imaging plane, and then the normalized coordinates are corrected according to radial and tangential distortion. Finally, the corrected normalized coordinates are remapped back to the pixel plane, thereby obtaining the position mapping relationship of the pixels falling on the corrected image.

[0070] The positional mapping relationship of the color image is generated as a distortion correction lookup table for the color channels. The original color image is then resampled and the difference is calculated using the distortion correction lookup table to output the corrected color image.

[0071] Furthermore, the camera intrinsic parameters and distortion parameters of the depth channel are read, and the depth image is resampled using a lookup table method that is isomorphic to the color image to remove distortion, resulting in a distortion-removed depth map.

[0072] Invalid depth values ​​in the depth map are processed by marking pixels with zero depth or those exceeding the range as invalid. For invalid pixels, if there is a valid depth value in the local neighborhood, the median of the valid depth in the neighborhood is used to fill the invalid pixel, thus obtaining a repaired depth map for 3D reconstruction.

[0073] Furthermore, the extrinsic parameter relationship between the depth camera and the color camera is read. For each valid depth pixel in the repaired depth map, the pixel and pixel depth value are first calculated into a 3D point in the depth camera coordinate system based on the depth camera intrinsic parameters. Then, the 3D point is transformed to the color camera coordinate system using the extrinsic parameters. Based on the color camera intrinsic parameters, the 3D point is projected back to the pixel plane of the color image to obtain the corresponding pixel position of the 3D point in the color image, and the depth value of the 3D point is written into the pixel position, thus obtaining the depth alignment map.

[0074] When multiple depth points are projected onto the same color pixel, in order to avoid incorrect depth due to occlusion, this embodiment retains the depth value closer to the color camera as the effective depth of the pixel.

[0075] For pixels with depth holes that still exist after alignment, if there is a valid depth point in the neighborhood, the neighborhood median is used to fill the hole.

[0076] Finally, the corrected color image and the aligned depth image are bound together at the same timestamp to form a frame of corrected and aligned RGB-D observation data.

[0077] S3. Based on the depth image, fit the ground plane to generate a near-ground mask.

[0078] Furthermore, the mobile robot acquires a depth image aligned to the color coordinate system at each sampling time and reads the camera intrinsic parameters.

[0079] To reduce interference from non-ground objects on the fitting, ground candidate pixel regions are defined in the depth image.

[0080] The candidate pixel region is defined as the set of pixels in the lower half of the image.

[0081] Furthermore, each pixel within the candidate pixel region is back-projected into a 3D point in the camera coordinate system based on the camera intrinsic parameters and the pixel's depth value, resulting in a 3D point set for ground fitting.

[0082] Robust plane fitting is performed on the 3D point set. Three points are randomly selected from the 3D point set to construct a candidate plane, and the point-to-plane distance between the candidate plane and each point in the point set is calculated. When the distance from a point to the candidate plane is less than the distance threshold, the current constructed point is determined to be an interior point of the candidate plane. The random sampling and interior point statistics are repeated multiple times, and the candidate plane with the most interior points is selected as the initial ground plane.

[0083] In a preferred embodiment, to improve the fitting accuracy, the initial ground plane can be refined by least squares based on all interior points of the initial ground plane to obtain the final initial ground plane.

[0084] Furthermore, to avoid misidentifying walls, tabletops, and other large flat surfaces as the ground, after obtaining the initial ground plane, an orientation constraint is applied to the initial ground plane. Specifically, the vertical direction vector in the camera coordinate system is determined, and the angle between the fitting plane normal and the vertical direction is calculated. When the angle is greater than the maximum tilt angle threshold, the fitting plane is determined not to meet the ground orientation condition, triggering refitting. When the angle is not greater than the maximum tilt angle threshold, the current plane is confirmed as the ground plane.

[0085] Furthermore, after determining the ground plane, for each effective depth pixel in the depth image, the distance from the effective depth pixel to the ground is calculated using the 3D points obtained by back-projection of the effective depth pixel. When the distance from the effective depth pixel to the ground is less than or equal to the near-ground thickness threshold, the current effective depth pixel is marked as a near-ground pixel; otherwise, it is marked as a non-near-ground pixel, and a near-ground mask is obtained.

[0086] Finally, morphological cleaning is performed on the near-ground mask. Specifically, an opening operation is first performed on the near-ground mask to remove isolated noise points, and then a closing operation is performed to fill small holes, so that the near-ground mask forms a continuous and coherent near-ground region.

[0087] It should be noted that the distance threshold is determined by calibration based on the standard deviation of depth measurements taken by the depth camera within the working distance range, and is usually twice the standard deviation of the depth measurement value, with a value range of [5,30]. The maximum tilt angle threshold is determined by offline calibration based on the robot's allowable ground slope and installation posture error, with a value range of [5,15]. The near-zone thickness threshold is determined by comprehensive calibration based on the fork insertion height tolerance and the tolerance of local ground undulations, with a value range of [10,50].

[0088] S4. Based on the near-ground mask, and combined with the depth gradient and brightness gradient, construct the structural dimension candidate regions of the left fork hole, right fork hole, and leading edge guide edge of the stack.

[0089] Furthermore, the corrected color image, aligned depth image, and near-ground mask are read at the same time.

[0090] To ensure that subsequent gradient calculations are performed only in the vicinity of the dockable region, the set of effective pixels near the ground is limited.

[0091] Specifically, the set of effective near-ground pixels is limited. Only pixels that are marked as near-ground by the near-ground mask and whose depth values ​​are within the effective depth range are retained as effective near-ground pixels.

[0092] Furthermore, the gradient vector of the depth image is calculated within the set of effective near-ground pixels, and the Euclidean norm of the gradient vector of the depth image is defined as the depth gradient magnitude.

[0093] Pixels with a depth gradient magnitude greater than or equal to the depth threshold are designated as depth boundary bands.

[0094] The corrected color image is first converted to grayscale, then the gradient vector is calculated within the set of effective pixels near the ground, and the Euclidean norm of the gradient vector of the color image is defined as the brightness gradient magnitude.

[0095] Pixels with a brightness gradient magnitude greater than or equal to the brightness threshold are designated as texture boundary bands.

[0096] It should be noted that the depth threshold is determined by statistically analyzing the depth gradient magnitude of structural boundary pixels and the magnitude distribution of non-boundary pixels on near-ground samples, and selecting the magnitude at the intersection of the two distributions as the depth threshold. The value range is usually [0.02, 0.15]. The brightness threshold is determined by statistically analyzing the brightness gradient magnitude of structural boundary pixels and the magnitude distribution of background pixels on different lighting samples, and selecting the magnitude that controls the background false trigger rate, such as being lower than a given proportion, as the brightness threshold. The value range is usually [10, 60].

[0097] Furthermore, the intersection of the depth boundary band and the texture boundary band is taken to obtain the gradient consistency candidate band of the image, so as to ensure that the candidate structure has both geometric and texture evidence.

[0098] After obtaining the gradient consistency candidate band, denoising processing is performed on the gradient consistency candidate band. First, isolated noise points are removed by morphological opening operation, and then connected component filtering is performed to retain only connected regions whose area meets the minimum pixel count requirement, so as to obtain continuous and observable near-ground candidate bands.

[0099] It should be noted that the minimum number of pixels required is determined by offline statistical analysis of the minimum imaging area of ​​the target, and the value range is usually [50, 500].

[0100] Furthermore, the horizontal midline of the image is defined as half the number of pixels of the image width, and the average of the abscissas of all pixels in the near-ground candidate zone is defined as the centroid abscissa of the near-ground candidate zone.

[0101] Using the centroid abscissa of the near-Earth candidate zone as the adaptive center, construct the left fork hole geometric window, the right fork hole geometric window, and the leading edge guide edge geometric window.

[0102] Specifically, the centroid of the near-Earth candidate band is located to the left of its horizontal coordinate, with a horizontal width equal to the width of the first pixel and a vertical range of [missing information]. The rectangular window is used as the geometric window for the left fork.

[0103] The centroid of the near-Earth candidate zone is located to the right of its horizontal coordinates, with a horizontal width of the second pixel and a vertical range of [missing information]. The rectangular window is used as the geometric window for the right fork.

[0104] The region around the centroid of the near-Earth candidate zone is defined by its horizontal coordinates, with a horizontal width of the third pixel and a vertical range of [missing information]. The rectangular window is used as the leading edge guide geometry window.

[0105] It should be noted that, These represent the row interval thresholds. Based on the camera installation height, pitch angle, and ground plane fitting results, the ground height band within the expected docking distance range is projected onto the image row coordinates. The corresponding row coordinate range is then used as the threshold for each row interval. The value range is usually [0.55, 0.70). The value range is usually [0.70, 0.85). The value range is usually [0.85, 0.98], where, This indicates the image height in pixels.

[0106] The pixel width is calculated by converting the nominal dimensions of the pallet fork width, fork spacing, and leading edge width, as well as the camera imaging scale (determined by intrinsic parameters and typical docking distance), and then fine-tuning it on the sample using low false detection constraints. The values ​​of the first and second pixel widths are typically [40, 180], and the values ​​of the third pixel width are typically [80, 320].

[0107] Furthermore, candidate regions for structural dimensions are constructed for the left fork hole, the right fork hole, and the leading edge guide edge, respectively. Specifically, for each type of structure, the intersection of the near-ground mask region, the gradient consistency candidate band, and the geometric window of the corresponding structure is taken to obtain the candidate regions for structural dimensions of the left fork hole, the right fork hole, and the leading edge guide edge, respectively.

[0108] Perform hole filling on each candidate region of each structural dimension and perform minimum area verification.

[0109] Specifically, the minimum area check marks the current structural dimension candidate region as invalid if its area is less than the minimum area threshold, and valid otherwise.

[0110] It should be noted that the minimum area threshold is determined by collecting samples containing candidate regions of the true stack structure dimension and non-stack noise regions during the calibration phase, statistically analyzing the pixel area distribution of the true stack candidate regions, and using the low quantile value of the pixel area of ​​the true stack candidate regions, such as the 5th percentile, as the minimum area threshold. The value range is usually [300, 3000].

[0111] S5. Within each candidate region of the structural dimension, generate visibility probability and boundary segment confidence, extract observable boundary segments, calculate structural visibility probability and boundary coverage, and determine the effective structural set.

[0112] Furthermore, for each structural dimension candidate region, the minimum bounding rectangle of the structural dimension candidate region is first calculated, and the corresponding structural dimension image patch is obtained by cropping from the corrected color image based on the minimum bounding rectangle.

[0113] The structural dimension image patch is scaled to a fixed resolution, while candidate region mask patches aligned with the structural dimension image patch are generated.

[0114] The structural dimension image patch and the candidate region mask patch are input into the joint feature network, which outputs the visibility probability field and boundary segment confidence field of each structure in each structural dimension candidate region.

[0115] The joint feature network employs a lightweight structure combining a shared network backbone with dual output heads. Specifically, it takes structural dimension image blocks and candidate region mask blocks as inputs, extracts multi-scale features through a shared network backbone composed of several levels of convolutional downsampling and depth-separable convolutional blocks, and fuses them from top to bottom through a feature pyramid to form high-resolution semantic features. Two parallel output heads are set after the backbone: one is a visibility head, which uses several convolutional layers followed by a sigmoid function to output a pixel-level visibility probability field; the other is a boundary head, which uses several convolutional layers to output a boundary segment confidence field with a fixed set of discrete directions, such as 0°, 45°, 90°, and 135°. The boundary segment confidence field is a confidence map normalized by a sigmoid function for each discrete direction.

[0116] Furthermore, based on the visibility probability field output by the joint feature network, the visibility probability in the visibility probability field is averaged within the candidate region of the structural dimension to obtain the visibility probability of each structure.

[0117] Based on the boundary segment confidence field output by the joint feature network, the boundary segment confidence field is comprehensively processed to obtain the set of observable boundary segments for each structure.

[0118] Specifically, the comprehensive processing involves, based on the boundary segment confidence field output by the joint feature network, for each candidate region of the structural dimension, selecting the direction with the highest confidence from multiple discrete directions as the optimal boundary direction for each pixel within the candidate region of the structural dimension, and taking the maximum confidence of the optimal boundary direction as the boundary strength of the pixel; setting a boundary confidence threshold, and selecting pixels within the candidate region of the structural dimension whose boundary strength is not lower than the boundary confidence threshold as the boundary seed point set; performing a neighborhood connectivity-based aggregation process on the boundary seed point set, aggregating interconnected boundary seed points into several boundary pixel chains; and performing a refinement process and fitting a polyline segment on each boundary pixel chain to obtain several boundary segments.

[0119] It should be noted that the training process of the joint feature network specifically involves offline supervised training using an RGB-D image dataset containing annotations of left fork holes, right fork holes, and leading edge guide structures. Each sample simultaneously provides pixel-level structural visibility labels and directional boundary labels. Visibility branches and boundary direction branches are set on the basis of a shared backbone feature extraction network, and binary cross-entropy loss and multi-directional cross-entropy loss are used for joint optimization. The samples are also subjected to viewpoint perturbation, occlusion simulation, and brightness change enhancement to improve robustness. During training, a fixed learning rate decay strategy is used until the validation set loss converges, thereby obtaining a joint feature network model that can simultaneously output the visibility probability field and the boundary segment confidence field within the candidate region.

[0120] It should be noted that the boundary confidence threshold is determined by the output probability distribution of the offline statistical network on the validation set, and the value range is usually [0.5, 0.9].

[0121] Furthermore, to ensure that the acquired boundary segments belong to the real structural boundaries that can be reliably observed, a direction consistency screening is performed on each boundary segment. Specifically, the optimal boundary direction of each pixel in the current boundary segment is counted. If the proportion of the direction with the highest proportion in the current boundary segment reaches the consistency threshold, the current boundary segment is retained. If the consistency does not reach the consistency threshold, the current boundary segment is identified as a noise or texture interference boundary and is removed. Finally, the set of observable boundary segments of each structure is obtained.

[0122] It should be noted that the consistency threshold is determined offline by statistically analyzing the distribution of the directional consistency ratio in the real structural boundary segments under the calibration scenario and combining it with the false detection ratio of the noisy boundary segments. This ensures that the consistency can cover the vast majority of real boundaries while effectively suppressing random noise. The value range is usually [0.6, 0.85].

[0123] Furthermore, after obtaining the set of observable boundary segments for each structure, the boundary coverage is calculated.

[0124] Specifically, the boundary curve length of the candidate region mask block is obtained, the sum of the lengths of the observable boundary segments in the observable boundary segment set is calculated, and the ratio of the two is used as the boundary coverage rate.

[0125] Furthermore, visibility thresholds and boundary coverage thresholds are set for each structure.

[0126] For each structural dimension, if the visibility probability of the structure is not lower than the visibility threshold of the corresponding structure, and the boundary coverage is not lower than the boundary coverage threshold of the corresponding structure, then the current structure is determined to be a valid structure and added to the set of valid structures; otherwise, the current structure is determined to be an invalid structure and no geometric solution is performed.

[0127] It should be noted that when the set of valid structures is empty, a flag indicating that the current frame structure is unavailable is output for the robot controller to perform a re-observation.

[0128] It should be noted that the visibility threshold is determined by the average visibility ratio of visible structures in offline statistics, and the value range is usually [0.3, 0.8]; the boundary coverage threshold is determined by the proportion of effective matching boundary lengths in offline statistics, and the value range is usually [0.4, 0.9].

[0129] S6. Call the stack structure template, perform normalized soft correspondence matching on the observation boundary points of the effective structure, and construct the geometric residual observation set.

[0130] Furthermore, the set of valid structures at the current moment is obtained, the set of observable boundary segments is read for each valid structure dimension, and uniform sampling is performed along each boundary segment at fixed arc length intervals to obtain the set of observation boundary points corresponding to each structure dimension. The observation boundary points are represented in the form of pixel coordinates.

[0131] At the same time, template data corresponding to each structural dimension is called from the stack structure template library. The template data includes at least two parts: one is the set of template boundary points of the structural dimension in the template normalized coordinate system, and the other is the set of template three-dimensional structural points associated with the structural dimension. The template three-dimensional structural points are represented in the stack coordinate system.

[0132] Furthermore, the observation boundary points of each structural dimension are mapped from pixel coordinates to the normalized coordinates of the candidate region of the structural dimension. Specifically, the minimum bounding rectangle of the candidate region of the structural dimension is first determined, and the coordinates of the upper left corner, width and height of the bounding rectangle are obtained. The observation boundary points are translated and scaled so that the horizontal and vertical coordinates of the observation boundary points are mapped to the interval between zero and one, thereby obtaining the set of observation normalized boundary points.

[0133] Furthermore, to address the non-one-to-one correspondence issues caused by boundary loss, occlusion, and reflection, a soft correspondence approach is adopted to construct the correspondence between observed normalized boundary points and template normalized boundary points. Specifically, for each observed normalized boundary point, the Euclidean distance between the observed normalized boundary point and all template normalized boundary points is calculated, and the Euclidean distance is converted into a similarity score. The closer the Euclidean distance, the higher the similarity score. All similarities for the same observed point are normalized so that the sum of the weights of the observed point to all template points is one, thereby obtaining the soft correspondence weight matrix from the observed point to the template point.

[0134] It should be noted that the Euclidean distance is defined as follows:

[0135] ;

[0136] in, Indicates the first in the same structural dimension The observed normalized boundary point and the first Euclidean distance between template normalized boundary points Indicates the first in the same structural dimension The coordinates of the observed normalized boundary point, Indicates the first in the same structural dimension The template normalizes the boundary point coordinates.

[0137] Based on a fixed matching scale parameter, the Euclidean distance is converted into a similarity score, expressed as:

[0138] ;

[0139] in, Indicates the first under the same structural dimension The observed normalized boundary point and the first Unnormalized similarity of template-normalized boundary points This indicates a fixed matching scale parameter.

[0140] For each observation normalized boundary point, normalize it on the template normalized boundary dimension to obtain the soft correspondence weight matrix.

[0141] A fixed matching scale parameter is used. During the calibration phase, the average Euclidean distance from each point in the template normalization boundary point set of the same structural dimension to its nearest neighbor is statistically analyzed, and this average Euclidean distance is used as the matching scale parameter. Furthermore, since the soft correspondence weight matrix essentially describes the weight allocation of each observation point to each template point, in order to obtain a definite pixel domain reference for each template point during residual construction, the soft correspondence weight matrix undergoes orientation unification processing. Specifically, the soft correspondence weights are re-normalized according to the template point dimension, so that for each template point, a set of normalized weights can be formed from all observation points; the soft correspondence weights are re-normalized according to the template point dimension, so that for each template point, a set of normalized weights can be formed from all observation points.

[0142] Furthermore, the template's 3D structural points are transformed to the camera coordinate system under the assumed pose of the stack, and the template's 3D structural points are projected onto the pixel plane using the camera's intrinsic parameters to obtain the pixel prediction point corresponding to each template's 3D structural point. The distance between the pixel prediction point and the template-guided observation target point is calculated, and the distance between the pixel prediction point and the template-guided observation target point is used as a geometric residual. The geometric residual calculation process is repeated for all template points in all effective structural dimensions to obtain a geometric residual observation set composed of multiple residuals.

[0143] S7. Based on the structural visibility probability and boundary coverage, calculate the weighted center pose of the pallet to the camera, and calculate the pose covariance of the pallet relative to the camera.

[0144] Furthermore, the set of valid structures is read, and the structural visibility probability and boundary coverage corresponding to each valid structure are read. At the same time, the constructed geometric residual observation data is read, where the residual observation data reflects the deviation between the projection position of the template's 3D structural points and the soft corresponding pixel target points.

[0145] Subsequently, the pose from the stack to the camera is parameterized using six-dimensional minimum parameters. The first three parameters are used to characterize the rotation increment, and the last three parameters are used to characterize the translation component. The pose is then reduced by the six-dimensional parameters to form a rotation matrix and a translation vector through exponential mapping, thus obtaining the homogeneous pose transformation.

[0146] Finally, the structural weight is calculated for each structural dimension. The structural weight is obtained by multiplying the visibility probability of the structure and the boundary coverage, and the value range is usually [0,1].

[0147] Furthermore, to obtain the center pose, a weighted robust optimization objective is first established. For all effective structural dimensions, the geometric residuals corresponding to each effective structural dimension are accumulated. At the same time, the accumulated residual term of each structural dimension is multiplied by the structural weight of the structural dimension. During the residual accumulation process, a robust cost function is used to transform the residuals so that the influence of large residuals is automatically compressed.

[0148] After completing the construction of the objective function, set the initial pose value, using the pose obtained in the previous frame as the initial value.

[0149] It should be noted that when the previous frame is unavailable, the pose obtained based on coarse localization of the candidate region is used as the initial value.

[0150] Furthermore, the pose is solved using an iterative reweighted least squares approach. Specifically, the iterative process of calculating residuals, calculating Jacobian, constructing weights, and solving for incremental pose updates is executed repeatedly until convergence.

[0151] In each iteration, all geometric residuals are recalculated under the current pose, and all residuals are arranged into a residual vector in a fixed order. The derivative of the residual vector with respect to the pose parameters is calculated to obtain the Jacobian matrix, which is used to characterize the effect of small changes in pose on the residuals.

[0152] Furthermore, a combined weight is constructed for each residual. The combined weight contains two parts: the first part is the structural weight, which reflects the reliability of the structural dimension to which the residual belongs; the second part is the point-level robust weight, such as Huber. The robust weight is determined by the derivative of the robust cost function with respect to the current residual, so that abnormal residuals are automatically reduced in weight during iteration. The value range is usually [0,1].

[0153] Meanwhile, a normal equation is constructed based on the Jacobian matrix, combined weights, and residual vectors, and the pose increment is solved.

[0154] Finally, the pose increment is applied to the current pose according to the Lie group increment update rule to obtain a new pose estimate.

[0155] If the norm of the pose increment is less than the convergence threshold or the number of iterations reaches the upper limit, the iteration stops, and the finally converged pose is output as the center pose from the stack to the camera.

[0156] It should be noted that the convergence threshold is determined offline by statistically analyzing the mean and standard deviation of the pose update amount in two consecutive iterations of multiple sets of stable docking data in the calibration scenario. The convergence threshold is not higher than the equivalent pose fluctuation amplitude caused by measurement noise, and the value range is usually [0.1, 1].

[0157] Furthermore, after the center pose is solved, in order to quantify the uncertainty of the center pose result, an information matrix is ​​constructed using the Jacobian matrix and weight matrix at the convergence time. The information matrix is ​​used to characterize the constraint strength of the current observation on the six-dimensional pose parameters. An invertibility check is performed on the information matrix. When the information matrix becomes close to singular due to structural missing or observation degradation, a regularization term proportional to the identity matrix is ​​added to the information matrix to improve numerical stability and ensure invertibility.

[0158] Finally, the inverse of the processed information matrix is ​​used to obtain the pose covariance matrix, which is used to characterize the estimation uncertainty of the center pose in each degree of freedom direction.

[0159] S8. Transform the center pose to the robot base coordinate system and calculate the lateral deviation, height deviation, and yaw angle deviation between the center line of the fork hole and the center line of the fork.

[0160] Furthermore, the center pose of the pallet relative to the camera and the extrinsic parameters of the camera relative to the robot base are read; at the same time, the geometric calibration parameters of the forks are read, and the geometric definition of the center line of the fork hole in the pallet structure template in the pallet coordinate system is read.

[0161] Furthermore, the center pose from the stack to the camera is obtained by performing a chained coordinate transformation using the extrinsic parameters from the camera to the base.

[0162] After the transformation is completed, the position and orientation of the pallet in space are represented in the robot base coordinate system.

[0163] Based on the geometric parameters of the fork centerline defined in the pallet structure template, the reference point and direction representation of the fork centerline are obtained in the pallet coordinate system. Using the obtained pallet-to-base pose, the fork centerline is transformed as a whole into the base coordinate system to obtain the fork centerline in the base coordinate system.

[0164] Meanwhile, based on the fork geometric calibration parameters, the reference point and direction of the fork centerline are directly given in the base coordinate system. The fork centerline is usually consistent with the robot's forward direction or determined according to the fork installation posture.

[0165] At this point, both the centerline of the fork hole and the centerline of the fork are constructed in the same base coordinate system.

[0166] Furthermore, in order to obtain the lateral deviation in the sense of docking, the directional relationship between the two center lines is horizontalized according to the vertical direction of the base, that is, only the directional component in the horizontal plane of the ground is considered to avoid the interference of the height component on the lateral calculation.

[0167] Using the horizontal forward direction of the fork centerline as a reference, construct a lateral direction that is orthogonal to the fork centerline in the horizontal plane.

[0168] Finally, the offset of the fork hole centerline relative to the fork centerline in the lateral direction is calculated, and the absolute value of the offset is taken as the lateral deviation.

[0169] Among them, lateral deviation is used to measure whether the fork will deviate from the center of the fork hole when the fork is inserted.

[0170] Furthermore, the height of the reference point of the fork hole centerline in the vertical direction of the base is taken as the center height of the fork hole; the current or target insertion height of the fork is taken as the fork height reference; the difference between the two is the height deviation.

[0171] Among them, the height deviation is used to measure whether the fork insertion height is consistent with the center height of the fork hole.

[0172] When the height deviation is zero, it means that the fork height and the center height of the fork hole match, and the pallet can be inserted smoothly.

[0173] Furthermore, the centerline direction of the fork opening and the centerline direction of the fork are projected onto the horizontal plane and normalized. Based on the normalized centerline, the angle between the two in the horizontal plane is calculated, and the angle is used as the yaw angle deviation.

[0174] Among them, the yaw angle deviation is used to measure whether the forward direction of the forks is in the same direction as the axial direction of the fork hole.

[0175] The smaller the yaw angle deviation, the more aligned the forks are with the fork holes axially, resulting in less insertion resistance and a higher success rate.

[0176] Furthermore, the lateral deviation, altitude deviation, and yaw angle deviation are combined in a fixed order to form a docking error vector.

[0177] S9. Based on lateral deviation, height deviation, and yaw angle deviation, combined with pose covariance, an error uncertainty domain is constructed, the most unfavorable docking margin is calculated, and the robot's docking determination on the pallet is output.

[0178] Furthermore, the docking error vector is considered as a function of the center pose parameters.

[0179] To transfer the uncertainty of the pose to the uncertainty of the docking error, an approximate mapping is performed using a first-order linearization method near the center pose. Specifically, the docking error vector is recalculated by applying small positive and negative perturbations to the pose parameters dimension by dimension and the difference is taken to obtain the sensitivity matrix of the docking error to the pose parameters. Then, the sensitivity matrix and the pose covariance are used to perform matrix mapping operations to obtain the docking error covariance.

[0180] Among them, the error covariance is used to characterize the fluctuation range and intercorrelation of lateral deviation, altitude deviation and yaw angle deviation under the current observation conditions.

[0181] Furthermore, based on the center value of the docking error vector and the docking error covariance, an error uncertainty region is constructed at the confidence level.

[0182] The error uncertainty domain is represented in ellipsoidal form to describe the range of possible values ​​of docking error at a given confidence level.

[0183] By using the ellipsoidal uncertainty domain, not only is the fluctuation of each error considered, but also the correlation between the three errors is considered, thus upgrading the docking determination from whether the central error is qualified to whether it is still qualified under the most unfavorable conditions within the confidence region.

[0184] It should be noted that the uncertainty region of the error is constructed as an ellipsoid set based on the confidence level, and is represented as follows:

[0185] ;

[0186] in, Indicates time The error uncertainty region, This represents any possible docking error vector, with dimensions 3×1. Indicates time The center docking error vector, This represents the docking error covariance matrix, with dimensions 3×3. This represents the inverse matrix of the docking error covariance matrix. Indicates the confidence level, with a value range of (0,1). This represents a chi-square distribution with 3 degrees of freedom at a confidence level of 100%. The quantiles below.

[0187] It should be noted that the confidence level is determined by statistically analyzing the historical distribution data of docking errors under the calibration environment and combining it with the equipment's safe offline setting. This ensures that the confidence level covers a high confidence probability interval of no less than the expected error distribution, and the value range is usually [0.90, 0.99].

[0188] Furthermore, to avoid complex global searches within the ellipsoidal uncertainty domain, a closed-loop, real-time computable conservative upper bound method is adopted to obtain the most unfavorable error. Specifically, for lateral deviation, altitude deviation, and yaw angle deviation, the maximum possible offset amplitude of the components within the confidence domain is calculated based on the variance information and confidence level in the error covariance. The maximum offset amplitude is then superimposed on the absolute value of the corresponding error center value to obtain the most unfavorable absolute upper bound of the error component within the confidence domain.

[0189] Among them, the upper bound of the absolute value of the most unfavorable condition is used to represent the maximum error that may occur under the most unfavorable working conditions, ensuring that the docking determination has safety and engineering conservatism.

[0190] It should be noted that, regarding the first One error component The upper bound of the maximum offset amplitude within the ellipsoidal domain is taken as:

[0191] ;

[0192] in, Indicates the first The upper bound of the maximum offset magnitude of each error component within the confidence ellipsoid domain. The first element of the error covariance matrix is... One diagonal element.

[0193] No. The upper bound of the most unfavorable absolute value of each error component within the confidence region is expressed as:

[0194] ;

[0195] in, Indicates the first The error components represent the upper bound of the most unfavorable absolute value within the confidence uncertainty region. Represents the center error vector The One error component.

[0196] Furthermore, the upper bound of the most unfavorable absolute value of the three errors is compared with the corresponding allowable tolerance, and the remaining margin ratio of each is calculated. The minimum value among the three margin ratios is taken as the most unfavorable docking margin, which reflects the safety margin of the weakest constraint dimension under the current docking conditions.

[0197] Subsequently, based on whether the worst-case docking margin is still not less than zero, the robot's docking determination result for the pallet is output. That is, when the worst-case docking margin is not less than zero, it is determined that docking is possible; when the worst-case docking margin is less than zero, it is determined that docking is not possible.

[0198] It should be noted that the worst-case docking margin is expressed as:

[0199] ;

[0200] in, , , .

[0201] in, Indicates time The worst-case docking margin, This represents the upper bound of the most unfavorable absolute value of the lateral deviation. This represents the upper bound of the most unfavorable absolute value of the height deviation. This represents the upper bound of the most unfavorable absolute value of the yaw angle deviation. Indicates the allowable tolerance in the lateral direction. Indicates the maximum allowable tolerance. This indicates the allowable tolerance for the yaw angle.

[0202] It should also be noted that, such as Figure 5 As shown, this invention establishes a quantifiable mapping relationship between observation consistency and docking safety margin. With increasing residual RMSE, the most unfavorable docking margin generally decreases, indicating that when the geometric consistency between the template projection and the soft-corresponding observations deteriorates, the algorithm can simultaneously reflect a reduction in the docking safety margin, thus making the judgment criteria verifiable and traceable. Meanwhile, Figure 5 The trend also indicates that uncertainty propagation and the worst-case upper bound mechanism can automatically become conservative under observation degradation conditions such as occlusion, reflection, boundary loss, and deep voids, avoiding the output of aggressive docking conclusions when structural information is insufficient, thus improving docking security. In addition, the scattered points still have reasonable dispersion near the same RMSE, reflecting the moderating effect of structural weights composed of structural visibility and boundary coverage on the results. That is, reliable structures contribute more and unreliable structures are automatically downweighted, so that the multi-structure fusion solution remains stable when some structures are missing, and reflects the reliability of the current frame through margin changes, thus improving overall robustness and engineering usability.

[0203] It should also be noted that, such as Figure 6As shown, the statistical distribution of residual RMSE across the entire sample range is presented. The residual RMSE exhibits a typical characteristic of concentrated main components and long tails. On the one hand, the distribution peaks are concentrated in the lower RMSE range, indicating that under most normal observation conditions, such as effective depth, clear texture and geometric boundaries, and minimal occlusion, this invention can stably obtain a small geometric consistency error, thus providing a reliable observation basis for pose solving and docking determination. On the other hand, the distribution has a certain long tail, reflecting that a small number of extremely degraded frames will still have large residuals, which is consistent with situations such as reflections, local occlusions, missing boundaries, and sudden depth holes in the field. The statistical characteristics indirectly prove the necessity of structural weights and robust cost functions. That is, in the face of long-tailed anomalies, the algorithm can automatically reduce the weights of large residuals, avoid a small number of abnormal frames dominating the overall estimation and causing misjudgments, and improve the safety and availability of the system under complex working conditions.

[0204] In summary, this invention improves the accuracy of structure localization by fitting the ground plane and generating a near-ground mask to achieve spatial constraints on candidate regions; it enhances the robustness of structure recognition by fusing depth gradients and brightness gradients to construct candidate regions for structural dimensions, thereby achieving synergistic enhancement of multi-source information; it improves the stability of pose estimation by performing weighted pose solving based on visibility probability and boundary coverage, thereby achieving adaptive adjustment of observation quality; it enhances the reliability of docking decisions by constructing a pose covariance matrix, thereby achieving quantitative expression of pose uncertainty; and it improves docking safety by establishing an error uncertainty domain and calculating the most unfavorable docking margin, thereby achieving conservative assessment of docking risks.

[0205] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for recognizing pallets in mobile robots based on computer vision, characterized in that, include: Obtain the intrinsic parameters of the color camera and the depth camera respectively, and obtain the extrinsic parameters from the camera to the robot base; Acquire color and depth images, and perform distortion correction and alignment processing; A near-ground mask is generated by fitting a ground plane to a depth image. The generation of the near-ground mask includes selecting candidate pixels in the lower half of the depth image; Backproject the candidate pixels into a 3D point set; By using the random sampling consistency method for plane fitting, the plane with the most interior points is selected as the initial ground plane. The initial ground plane is refined using least squares, and orientation constraints are checked based on the angle between the plane normal and the vertical direction. A near-ground mask is generated based on a threshold distance from the depth point to the ground. Based on the near-ground mask, and combined with the depth gradient and brightness gradient, candidate regions for the structural dimensions of the left fork, right fork, and leading edge guide edge of the stack are constructed. Within each candidate region of structural dimension, visibility probability and boundary segment confidence are generated, observable boundary segments are extracted, and structural visibility probability and boundary coverage are calculated to determine the effective set of structures. The visibility probability is the pixel-level visibility probability within each candidate region of the structural dimension. The structural visibility probability is obtained by averaging all visibility probabilities within the candidate region of the structural dimension. Call the stack structure template, perform normalized soft correspondence matching on the observation boundary points of the effective structure, and construct the geometric residual observation set; Based on the structural visibility probability and boundary coverage, the center pose of the pallet to the camera is calculated using a weighted method, and the pose covariance of the pallet relative to the camera is calculated. The structural weights in the weighted solution are obtained by multiplying the visibility probability of the structure and the boundary coverage. The weighted solution of the center pose from the stack to the camera includes establishing a weighted robust optimization objective, accumulating the geometric residuals corresponding to the effective structural dimensions, multiplying the residual accumulation term of each structural dimension by the structural weight of the structural dimension, and transforming the residuals through a robust cost function during the residual accumulation process. Transform the center pose to the robot base coordinate system and calculate the lateral deviation, height deviation and yaw angle deviation between the center line of the fork hole and the center line of the fork. Based on lateral deviation, height deviation, and yaw angle deviation, and combined with pose covariance, an error uncertainty domain is constructed, the most unfavorable docking margin is calculated, and the robot's docking determination on the pallet is output.

2. The mobile robot pallet recognition method based on computer vision as described in claim 1, characterized in that, The specific steps for obtaining the intrinsic parameters of the color camera and the depth camera, and obtaining the extrinsic parameters from the camera to the robot base, are as follows: A planar calibration board is set up within the robot's observable area, and multiple frames of calibration images of the planar calibration board are collected at different distances, different viewpoints, and different postures. Extract the pixel coordinates of feature points in each frame of the calibration image and establish the three-dimensional coordinates of the feature points in the calibration plate coordinate system; Construct the reprojection error between 3D coordinates and pixel coordinates, and solve for the camera focal length parameters, principal point parameters, and distortion parameters by minimizing the reprojection error to obtain the camera intrinsic parameter matrix; Given the pose of the planar calibration plate relative to the robot base, the PnP algorithm is used to solve the pose of the calibration plate relative to the camera based on the three-dimensional coordinates of the feature points and the corresponding pixel coordinates. Based on the coordinate transformation link relationship, matrix operations are performed on the pose from the calibration plate to the base and the solved pose from the calibration plate to the camera to obtain the extrinsic parameter matrix from the camera to the robot base.

3. The mobile robot pallet recognition method based on computer vision as described in claim 1, characterized in that, The distortion correction and alignment process includes: A distortion lookup table is established based on the intrinsic parameters and distortion parameters of the color channel and depth channel; The color image and depth image are resampled using a distortion correction lookup table to obtain the distortion correction image; The effective depth pixels in the depth image are back-projected into 3D points, and then projected onto the color image coordinate system according to the extrinsic relationship between the depth camera and the color camera to achieve spatial alignment between the depth image and the color image.

4. The mobile robot pallet recognition method based on computer vision as described in claim 1, characterized in that, The structural dimension candidate regions for the left fork, right fork, and leading edge guide of the construction stack include: Within the set of effective near-ground pixels, calculate the depth gradient magnitude and the brightness gradient magnitude respectively; The set of effective near-ground pixels is limited, and only pixels that are marked as near-ground by the near-ground mask and whose depth values ​​are within the effective depth range are retained as effective near-ground pixels; Within the set of effective pixels near the ground, the gradient vector of the depth image is calculated, and the Euclidean norm of the gradient vector of the depth image is defined as the depth gradient magnitude. The corrected color image is first converted to grayscale, then the gradient vector is calculated within the set of effective pixels near the ground, and the Euclidean norm of the gradient vector of the color image is defined as the brightness gradient magnitude. Pixels with depth gradient magnitudes greater than or equal to the depth threshold are defined as depth boundary bands; Pixels with a brightness gradient magnitude greater than or equal to a brightness threshold are identified as texture boundary bands; The intersection of the depth boundary band and the texture boundary band is taken as the gradient consistency candidate band; Based on the gradient consistency candidate band, geometric windows for the left fork hole, right fork hole, and leading edge guide are constructed. The intersection of the near-ground mask region, the gradient consistency candidate band, and the geometric window of the corresponding structure is taken to form the candidate region for each structural dimension.

5. The mobile robot pallet recognition method based on computer vision as described in claim 4, characterized in that, The generated visibility probability and boundary segment confidence include: Crop candidate regions based on structural dimensions into fixed-resolution image patches; The resolution image patch and the corresponding mask patch are input into the joint feature network of the shared backbone network and the dual-output head structure; A pixel-level visibility probability field is generated using the visibility output head; The boundary segment confidence field with multiple discrete directions is generated through the boundary output head.

6. The mobile robot pallet recognition method based on computer vision as described in claim 5, characterized in that, The construction of the geometric residual observation set includes: Uniform sampling is performed on the observable boundary segments of the effective structure to obtain the observation boundary points; The template data corresponding to each structural dimension is called from the stack structure template library. The template data includes the template boundary point set of the structural dimension in the template normalized coordinate system and the template three-dimensional structural point set associated with the structural dimension. The template three-dimensional structural points are represented in the stack coordinate system. Map the observed boundary points to the normalized candidate region coordinates, calculate the Euclidean distance between the observed normalized boundary points and the template normalized boundary points, and convert it into similarity weights. The similarity weights are normalized to form a soft correspondence weight matrix; The soft correspondence weights are re-normalized according to the template point dimension, so that for each template point, a set of normalized weights is formed from all observation points. The template's 3D structure points are transformed to the camera coordinate system under the assumed pose of the stack, and the template's 3D structure points are projected onto the pixel plane using the camera's intrinsic parameters to obtain the pixel prediction points corresponding to each template's 3D structure point. Calculate the distance between the pixel prediction point and the template-guided observation target point, and use the distance between the pixel prediction point and the template-guided observation target point as a geometric residual; Repeat the geometric residual calculation process for all template points of all effective structural dimensions to obtain a geometric residual observation set composed of multiple residuals.

7. The mobile robot pallet recognition method based on computer vision as described in claim 6, characterized in that, The weighted solution for the center pose from the stack to the camera includes: By using a six-dimensional minimum parameter to parametrically represent the pose using a Lie group, a weighted objective function containing structural weights and a robust cost function is constructed. The weighted objective function is optimized and solved by using an iterative reweighted least squares algorithm. After iterative convergence, an information matrix is ​​constructed based on the Jacobian matrix and the weight matrix; The Jacobian matrix is ​​obtained by recalculating all geometric residuals in the current pose in each iteration, forming a residual vector from all residuals in a fixed order, and calculating the derivative of the residual vector with respect to the pose parameters. The weight matrix refers to constructing a combined weight for each residual. The combined weight includes structural weights and point-level robust weights. The robust weights are determined by the derivative of the robust cost function with respect to the current residual, so that abnormal residuals are automatically reduced in weight during iteration. Invert the information matrix to obtain the pose covariance matrix.

8. The mobile robot pallet recognition method based on computer vision as described in claim 7, characterized in that, The calculation of the lateral deviation, height deviation, and yaw angle deviation between the centerline of the fork opening and the centerline of the fork includes: The pose of the center of the pallet is transformed to the base coordinate system using the extrinsic parameters from the camera to the robot base. Construct the center lines of the fork opening and the fork itself in the base coordinate system. Calculate the offset of the center line of the fork opening relative to the center line of the fork in the horizontal lateral direction as the lateral deviation. Calculate the difference between the center height of the fork opening and the fork height reference as the height deviation. Calculate the angle between the two center lines in the horizontal plane as the yaw angle deviation.

9. The mobile robot pallet recognition method based on computer vision as described in claim 8, characterized in that, The docking determination of the output robot with the pallet includes: Based on the first-order linearization of the center pose parameters, the pose covariance is mapped to the docking error covariance matrix; At the confidence level, a three-dimensional ellipsoidal error uncertainty domain is constructed, and the upper bounds of the most unfavorable absolute values ​​of the lateral deviation, altitude deviation, and yaw angle deviation within the three-dimensional ellipsoidal error uncertainty domain are calculated respectively. Compare the upper bound of the absolute value of the most unfavorable condition with the corresponding allowable tolerance, and take the minimum value of the remaining margin ratio as the most unfavorable docking margin. Based on the worst-case docking margin, output the docking decision.

Images