Alignment guiding method and device, equipment and storage medium

Abstract

The invention discloses an alignment guiding method, an alignment guiding device, equipment and a medium, and relates to the technical field of safety control of high-altitude operation equipment, the alignment guiding method comprises the steps that three-dimensional depth information of an operation area is collected, and the operation area comprises fork teeth of a pallet fork in the operation equipment and pallet fork holes in a goods shelf; based on the three-dimensional depth information, determining the spatial position of the inner edge of the fork tooth and the spatial position of the fork hole of the tray; calculating a three-dimensional relative pose deviation between the inner edge of the fork tooth and the space position of the tray fork hole; and generating guide information based on the three-dimensional relative pose deviation and outputting the guide information so as to guide and adjust alignment of the prongs and the pallet fork holes. According to the invention, the alignment precision and reliability of the operation equipment can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aerial work equipment safety control, in particular to a positioning guiding method, a positioning guiding device, equipment and a medium. BACKGROUND

[0002] In the modern logistics and warehousing system, the telescopic forklift as the core equipment of multi-layer warehouse goods handling undertakes the key responsibility of efficient transfer of goods, and its operation scene covers multiple fields such as industrial production, e-commerce warehousing, cold chain logistics, etc. With the increasing demand for warehousing stereoscopic and high frequency of goods turnover, the telescopic forklift needs to complete the accurate positioning of the fork teeth and the pallet fork hole in the high-altitude environment of dozens of meters, and the operation precision directly affects the efficiency of goods transfer and the utilization rate of warehouse space, which has become an important part of logistics automation upgrade.

[0003] However, the particularity of the high-altitude operation environment brings significant challenges to the positioning operation of the fork teeth and the pallet fork hole. On the one hand, the operator in the cab is affected by factors such as visual obstruction, high-altitude light changes, and is difficult to intuitively judge the three-dimensional relative position of the fork teeth and the pallet fork hole; on the other hand, high-altitude positioning requires high precision, and manual operation is prone to long positioning time and low operation efficiency due to visual errors and operation fatigue. More importantly, if there is an operation error in the manual positioning process, it may cause safety accidents such as goods falling and equipment collision, causing loss of personnel safety and property.

[0004] To solve the above problems, the existing technology usually adopts a simple visual auxiliary scheme, which relies on two-dimensional image acquisition and analysis, lacks accurate perception ability of three-dimensional space position, and cannot effectively deal with complex situations such as attitude change and space offset of the fork teeth and the pallet in high-altitude operation. The positioning accuracy and reliability cannot meet the actual demand. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a positioning guiding method, a positioning guiding device, equipment and a medium to solve the problem that the positioning accuracy and reliability of the existing positioning guiding method cannot meet the actual demand.

[0006] To achieve the above purpose, the first aspect of the present application provides a positioning guiding method, the method comprising: acquiring three-dimensional depth information of an operation area, the operation area including fork teeth of a fork of an operation equipment and a pallet fork hole on a goods shelf; determining a spatial position of an inner edge of the fork teeth and a spatial position of the pallet fork hole based on the three-dimensional depth information; calculating a three-dimensional relative pose deviation between the spatial position of the inner edge of the fork teeth and the spatial position of the pallet fork hole; generate guiding information based on the three-dimensional relative pose deviation and output the guiding information to guide adjustment of the alignment of the tine and the pallet fork hole.

[0007] The second aspect of the present application provides a positioning guiding device for aerial work equipment, the positioning guiding device comprising: a collection module configured to collect three-dimensional depth information of a work area, the work area including a tine of a fork in the aerial work equipment and a pallet fork hole on a pallet rack; a determination module configured to determine spatial positions of an inner edge of the tine and the pallet fork hole based on the three-dimensional depth information; a calculation module configured to calculate a three-dimensional relative pose deviation between the spatial positions of the inner edge of the tine and the pallet fork hole; an output module configured to generate guiding information based on the three-dimensional relative pose deviation and output the guiding information to guide adjustment of the alignment of the tine and the pallet fork hole.

[0008] The third aspect of the present application provides aerial work equipment, comprising: a positioning guiding device, the device comprising: a depth camera mounted at a tip of a fork and configured to collect three-dimensional depth information of a work area; a processor in communication connection with the depth camera and configured to execute the method according to any one of the first aspect; a human-computer interaction device in communication connection with the processor and configured to output the guiding information.

[0009] The fourth aspect of the present application provides a machine readable storage medium, the machine readable storage medium having instructions stored thereon, the instructions being used to cause a machine to execute the positioning guiding method according to the first aspect.

[0010] The fifth aspect of the present application provides a computer program product, instructions in the computer program product being executed by a processor of an electronic device to cause the electronic device to execute the positioning guiding method.

[0011] In the present application, by collecting three-dimensional depth information of a work area, compared with the prior art scheme relying on two-dimensional image analysis, the three-dimensional spatial positions of a tine and a pallet fork hole can be accurately perceived, and complex scenarios such as attitude changes and spatial shifts of the two in aerial work can be effectively coped with; based on the three-dimensional depth information, spatial positions of an inner edge of the tine and the pallet fork hole are determined and a three-dimensional relative pose deviation is calculated, thereby ensuring the accuracy of the determination of the positioning deviation; and then guiding information is generated according to the deviation, which can effectively improve the accuracy and reliability of the alignment of the tine and the pallet fork hole, solve the problem that the prior art cannot meet the actual aerial positioning needs, and improve work efficiency and reduce operation risks through guiding adjustment.

[0012] Other features and advantages of the embodiments of the present application will be described in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the embodiments of the present application together with the following detailed description, but do not limit the embodiments of the present application. In the drawings: Figure 1 A flowchart of a positioning guide method according to an embodiment of the present application is schematically shown; Figure 2 An installation diagram of a depth camera according to an embodiment of the present application is schematically shown; Figure 3 A camera field of view diagram applied to a shelf pallet fork taking scene according to an embodiment of the present application is schematically shown; Figure 4 A diagram of a pallet fork coordinate system and a camera coordinate system according to an embodiment of the present application is schematically shown; Figure 5 A diagram of a pallet front view angle according to an embodiment of the present application is schematically shown; Figure 6 A structural diagram of a positioning guide device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0014] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the specific embodiments described herein are only used to explain and illustrate the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0015] It should be noted that the acquisition, transmission, storage, use, processing and the like of data in the technical solutions of the present application comply with relevant provisions of laws and regulations. In the embodiments of the present application, some industry existing solutions such as software, components, models and the like may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but does not mean that the applicant has or will necessarily use the solutions.

[0016] It should be noted that if the embodiments of the present application involve directionality indication (such as up, down, left, right, front, back, etc.), the directionality indication is only used to explain the relative position relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directionality indication also changes accordingly.

[0017] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0018] It should be noted that the alignment guiding method provided in the subsequent embodiments of the present application can be applied to any work equipment. For the sake of clear description of the technical solutions, the alignment guiding method is taken as an example of being applied to a forklift truck, and the embodiments are described.

[0019] Figure 1 The flowchart of the alignment guiding method according to the embodiments of the present application is schematically shown. As shown in Figure 1 The alignment guiding method provided in the embodiments of the present application can include the following steps.

[0020] Step 101, collecting three-dimensional depth information of a work area, the work area including tines of a fork in a work equipment and a pallet fork hole on a shelf; In the present embodiment, the work equipment is specifically a telescopic forklift truck, and the work area refers to a space range in which the tines need to be connected with the pallet fork hole on the shelf when the telescopic forklift truck performs a cargo handling operation, covering the insertion end of the tines, the pallet fork hole and the associated pallet structure around it, such as pallet legs, bottom crossbeams, etc., and is suitable for high-altitude cargo transfer scenarios of multi-layer storage in industrial production, e-commerce warehousing, cold-chain logistics, etc.

[0021] The measurement equipment for collecting three-dimensional depth information is a depth camera, such as Figure 2 and Figure 3 As shown in Figure 2 is an installation schematic diagram of the depth camera, Figure 3is a camera field of view schematic diagram applied to a fork truck pallet fork picking scene. The depth camera needs to be fixedly installed at the end of the pallet fork of the fork truck. The preferred installation position is the center of symmetry of the pallet fork, the installation height is denoted as H, and the pitch angle is denoted as θ. During installation, the following conditions should be ensured: 1) the depth camera can follow the lifting, translation and pitching movement of the pallet fork; 2) the end of the pallet fork is located at about one third below the image in the field of view of the depth camera; and 3) the angle between the camera optical axis and the plane of the pallet fork (i.e. the plane in which the left and right tines are located) is less than 30 degrees. This installation mode ensures that the camera can continuously and stably capture the complete working area including the front end of the pallet fork tine and the target pallet fork hole on the high-altitude shelf during the working process.

[0022] The collected three-dimensional depth information specifically includes two types of data: color images (RGB images) and depth images; wherein the color images are used to provide the color and texture features of the scene, and assist in identifying the two-dimensional contours of the inner edges of the tines, the pallet and other targets; and the depth images are used to provide the distance from each pixel point in the image to the depth camera.

[0023] Step 102, based on the three-dimensional depth information, determining the spatial position of the inner edge of the tine and the spatial position of the pallet fork hole; In this embodiment, first, a line detection algorithm such as Hough transform is used on the RGB image to identify the positions of the inner edges of the left and right tines in the image. Then, by combining the depth image collected synchronously, the two-dimensional image coordinates are mapped into the depth camera coordinate system through three-dimensional coordinate conversion, forming a three-dimensional point cloud set representing the inner edges of the left and right tines.

[0024] Then, the above point cloud is converted from the depth camera coordinate system to the pallet fork coordinate system by using the coordinate conversion relationship obtained through calibration in advance. As shown in Figure 4 , Figure 4 is a schematic diagram of the pallet fork coordinate system and the camera coordinate system, and the pallet fork coordinate system is defined as follows: the origin O F is located at the center point of the tine root (the center of symmetry of the left and right tines); the Z F axis points to the tine tip along the extension direction of the tine; the X F axis is horizontal to the right (perpendicular to the forward direction of the fork truck); and the Y F axis is vertical downward (consistent with the right-hand rule).

[0025] Finally, in the pallet fork coordinate system, the coordinate values of the point cloud sets of the left and right tines in the X F axis and Y F axis directions are respectively calculated as arithmetic mean values, to obtain the average spatial coordinates of the inner edges of the left and right tines in the pallet fork coordinate system, which are taken as the spatial positions of the inner edges of the tines.

[0026] For determining the spatial position of the pallet fork hole, first, a pre-trained visual model is used to quickly locate the pallet area in the RGB image. At the same time, the original three-dimensional point cloud is filtered and denoised, and the point cloud data mainly belonging to the pallet is screened according to the image positioning result. Then, the bearing plane of the pallet is identified and optimized from the screened point cloud, and the plane normal vector is obtained. Further, the boundary feature points are extracted near the pallet plane, and the feature lines representing the bottom edge (horizontal) and leg edge (vertical) of the pallet are obtained through clustering and straight line fitting, and are screened according to their direction attributes and the number of three-dimensional points they depend on. Finally, the intersection of the screened horizontal and vertical feature lines is calculated, and this intersection point is the key reference point of the pallet fork hole. Combined with the three-dimensional coordinates of this intersection point and the plane normal vector of the pallet, the spatial position of the pallet fork hole can be completely determined.

[0027] Step 103, calculating the three-dimensional relative pose deviation between the inner edge of the fork tooth and the spatial position of the pallet fork hole; In this embodiment, all calculations are carried out in the fork coordinate system, mainly including: Horizontal deviation (ΔX): the center deviation of the fork tooth and the fork hole in the horizontal direction.

[0028] Vertical deviation (ΔY): the deviation of the fork tooth and the fork hole in the height direction.

[0029] Yaw angle (ΔYaw): the angle between the direction of the fork extension and the direction of the pallet plane normal vector in the horizontal plane, reflecting the angular deviation of the two directions.

[0030] Step 104, generating guidance information based on the three-dimensional relative pose deviation and outputting to guide the adjustment of the alignment of the fork tooth and the pallet fork hole.

[0031] In this embodiment, first, it is judged whether the above deviation meets the preset alignment success condition. If it meets, a prompt allowing insertion is generated; if it does not meet, specific adjustment instructions such as "move left", "raise", "rotate clockwise" are generated according to the type and size of the deviation. These guidance information are output in real time through various ways such as the graphical interface of the vehicle-mounted display screen, text and voice broadcast, etc., directly guiding the operator to adjust the forklift until accurate alignment.

[0032] In the embodiment, by collecting three-dimensional depth information of the working area, compared with the prior art relying on two-dimensional image analysis, the three-dimensional spatial position of the tine and the pallet fork hole can be accurately perceived, and the complex scenes such as posture change and spatial offset of the two in high-altitude work can be effectively coped with; the spatial position of the tine inner edge and the pallet fork hole is determined based on the three-dimensional depth information, and the three-dimensional relative pose deviation is calculated, thereby ensuring the accuracy of the alignment deviation determination; and then the guide information is generated according to the deviation, which can effectively improve the accuracy and reliability of the alignment of the tine and the pallet fork hole, solve the problem that the prior art cannot meet the actual high-altitude alignment requirements, and improve the work efficiency and reduce the operation risk through the guide adjustment.

[0033] In an embodiment of the present application, the three-dimensional depth information includes a color image and a depth image. The spatial position of the tine inner edge is determined by the following steps: Based on the depth image, straight line detection and three-dimensional coordinate conversion are performed on the color image to obtain a point cloud set of the tine inner edge in the coordinate system of the depth camera, and the depth camera is arranged on the fork for collecting the color image and the depth image. The point cloud set is converted to a preset fork coordinate system. The average coordinates of the point cloud set in the preset direction after conversion in the fork coordinate system are calculated to obtain the spatial position of the tine inner edge.

[0034] In the embodiment, the three-dimensional depth information includes a color image and a depth image. A frame of color image and a frame of depth image collected by the depth camera installed at the end of the fork can be synchronously obtained. The color image provides rich texture and edge information, and the depth image provides distance information of each pixel point.

[0035] Based on the depth image, straight line detection and three-dimensional coordinate conversion are performed on the color image. Specifically, after the color image is preprocessed, the Hough transform algorithm is used to detect the two-dimensional straight line profile of the tine inner edge to obtain the pixel coordinates of the pixel points on the profile; then, the depth values of the corresponding pixel points in the depth image are combined, the two-dimensional pixel coordinates are converted into three-dimensional coordinates in the coordinate system of the measuring device through the measuring device internal parameters, and all three-dimensional coordinates are summarized to obtain the point cloud set of the tine inner edge.

[0036] The point cloud set of the tine inner edge in the coordinate system of the measuring device is converted to the preset fork coordinate system by using the pre-calibrated conversion parameters, including the rotation matrix R and the translation vector T. The average coordinates of the point cloud set in the preset direction are calculated to obtain the spatial position of the tine inner edge; through the mean value calculation, the local deviation caused by vibration and load in high-altitude work can be offset, and the obtained spatial position is stable and reliable, thereby providing an accurate own reference for subsequent relative pose deviation calculation.

[0037] Additionally, it should be noted that calibration is required during system installation to achieve the transformation from the camera coordinate system {C} to the fork coordinate system {F}. By acquiring the corresponding data of the camera and fork in multiple different poses, the fixed rigid body transformation relationship between them can be calculated. Specifically: After calibrating the depth camera and forks, the transformation relationship from the camera coordinate system {C} to the fork coordinate system {F} is obtained as follows: P F =R P C +T; Where R is the rotation matrix (the camera's orientation relative to the fork); T is the translation vector (the coordinates of the camera origin in the fork coordinate system). The fork coordinate system {F} is defined as follows, with the origin O... F : Center point of the root of the fork tooth (i.e., the center of symmetry between the left and right fork teeth). Z F Axis: Along the direction of the fork teeth (pointing towards the fork tip). X F Axis: Horizontal to the right (perpendicular to the forklift's direction of travel). Y F Axis: Vertically downward (following the right-hand rule). The depth camera coordinate system {C} is defined as follows, with the origin O... C Camera optical center. Z C Axis: The direction of the camera's optical axis (directly in front). X C Axis: Horizontal to the right, Y C Axis: Vertically downwards.

[0038] Given the camera mounting height H, the camera pitch angle θ, and the fork length L, we can obtain: Pitch angle θ: Camera optical axis Z C The angle between the camera and the horizontal plane (XFZF plane) is positive when the camera is tilted upwards; the height H of the camera from the center point of the fork is the optical center O of the camera. C To the center of the root of the fork tooth O F vertical distance (along) Y F Direction). Horizontal distance D: Camera optical center O C To O F Along Z F The distance between the axes, obtained from geometric relations: Then the origin O in the camera coordinate system C Translation vector in the coordinate system {F} Rotation matrix R C→F Let the rotation of the camera coordinate system {C} relative to {F} be a rotation θ about the XF axis. Therefore, the coordinates of any point PC in the camera coordinate system in the forklift coordinate system are: Expand into component form: .

[0039] In this embodiment, by accurately converting the fork tooth edge three-dimensional point cloud detected by fusing the depth image and the color image to the fork coordinate system and calculating its average coordinates, the spatial position of the fork tooth is stably and accurately extracted. This method effectively overcomes the problem of unstable detection caused by changes in viewing angle and light interference in high-altitude operations, significantly improving the robustness and accuracy of detection.

[0040] In an embodiment of the present application, the spatial position of the pallet fork hole is determined by the following steps: A pre-trained pallet detection model is used to determine the position of the pallet in the color image, obtaining a two-dimensional positioning area corresponding to the pallet; The original point cloud data generated by the depth image is subjected to interference point rejection processing, and effective point clouds within a preset distance range in the depth camera coordinate system are retained; The two-dimensional positioning area is converted to a spatial positioning range in the depth camera coordinate system, and effective point clouds falling within the spatial positioning range are screened out; The screened point clouds are subjected to plane fitting to determine the plane normal vector corresponding to the pallet plane; The boundary feature points of the screened point clouds are extracted, and the spatial position of the pallet fork hole is determined using the boundary feature points and the plane normal vector.

[0041] In this embodiment, a pre-trained pallet detection model can be used to process the color image to identify the two-dimensional positioning area of the pallet, locking the approximate range of the pallet in the image. The pallet detection model can be a deep learning model such as YOLO or SSD, and the two-dimensional positioning area can be a rectangular box.

[0042] A straight-through filtering algorithm can be used to perform interference point rejection processing on the original point cloud data generated by the depth image, retaining effective point clouds with Z values within a preset distance range (0.5m-2m) in the measurement device coordinate system, and rejecting irrelevant interference points such as the ground and distant shelves. Subsequently, the effective point clouds are subjected to downsampling processing to reduce the computational complexity of subsequent calculations.

[0043] The two-dimensional positioning area of the pallet obtained in the color image is converted to a spatial positioning range in the measurement device coordinate system through the coordinate conversion relationship obtained above, and only the effective point clouds falling within this spatial range are retained, achieving accurate focusing on the pallet area point cloud.

[0044] The screened pallet area point clouds are subjected to plane fitting, and the pallet plane and the corresponding plane normal vector are determined through centroid score calculation. This normal vector can directly reflect the overall tilt posture of the pallet, providing a directional reference for subsequent feature line screening.

[0045] The boundary feature points correspond to the outer edges of the tray support legs, the bottom transverse beam edges and the like structures. The boundary feature points of the screened point cloud are extracted, the boundary feature points are clustered and fitted to obtain feature straight lines in different directions; based on the angle relationship between the direction vector of the feature straight line and the normal vector of the tray plane, combined with the number of three-dimensional points contained by the feature straight line, the transverse feature line and the longitudinal feature line are screened; the intersection of the transverse feature line and the longitudinal feature line is calculated to obtain the positioning reference point of the tray fork hole, that is, the physical connection position of the tray support leg and the bottom transverse beam.

[0046] Based on the three-dimensional coordinates of the positioning reference point, combined with the normal vector of the tray plane, the spatial position of the tray fork hole is determined, which can include the center coordinates of the fork hole, the opening direction, the vertical height range and the like, and the position can accurately reflect the real spatial posture of the fork hole, providing a target reference for the insertion of the fork teeth.

[0047] In the embodiment, the pre-trained model is used to quickly and roughly position the tray, then the point cloud filtering and plane fitting are used to accurately determine the posture of the tray, and finally the boundary feature extraction and structure reasoning are used to accurately calculate the position of the fork hole, so that the efficient, robust and accurate three-dimensional positioning of the tray fork hole in a complex high-altitude environment is realized.

[0048] In an embodiment of the present application, the screened point cloud is fitted to determine the tray plane, including: a plurality of candidate planes are fitted from the screened point cloud; for each candidate plane, a comprehensive score of each candidate plane is calculated based on the centroid height, the normal vector direction and the number of three-dimensional points contained; the candidate plane with the highest comprehensive score is selected as the tray plane.

[0049] In the embodiment, the screened point cloud is the tray area point cloud screened after the straight-through filtering interference removal, downsampling and spatial range screening, which contains the point cloud data of the tray bearing surface, the connection area between the support leg top and the tray, and ensures that the point cloud data is strongly related to the main structure of the tray. A robust plane fitting algorithm can be used to fit the input point cloud multiple times, for example, based on the RANSAC algorithm or its improved variants.

[0050] The specific centroid score calculation is as follows: let a plane , the point cloud of which is , and each point , then the centroid thereof is: , a comprehensive score function is designed, which can be as follows:

[0051] wherein, the centroid height of the plane , H is the height of the tray, Let be the normal vector of the plane. Let be the unit vector in the forklift coordinate system. Let i be the number of points in plane i. The total number of points on the plane. We retain only the plane that yields the highest score as the tray plane and obtain the corresponding tray plane normal vector. .

[0052] In this embodiment, by introducing a multi-dimensional comprehensive scoring mechanism based on centroid height, normal vector direction, and point cloud scale, the accuracy and robustness of identifying the true pallet bearing plane in complex point cloud scenarios are significantly improved.

[0053] In one embodiment of this application, determining the spatial position of the pallet fork hole using the boundary feature points and the plane normal vector includes: Clustering of boundary feature points and fitting them yields feature lines in different directions; Based on the direction vector of the feature line and the number of three-dimensional points it contains, horizontal and vertical feature lines are selected from the feature lines. The intersection of the selected transverse and longitudinal feature lines is calculated to obtain the positioning reference point of the pallet fork hole; Based on the three-dimensional coordinates and plane normal vector of the positioning reference point, the spatial position of the pallet fork hole is determined.

[0054] In this embodiment, the extracted boundary feature points (i.e., boundary points of the point cloud in the x-direction (horizontal direction) and y-direction (vertical direction), including point cloud edges, gradient abrupt change locations, or edge filter results) are clustered, and feature lines in different directions are fitted. The specific process is as follows: First, the KdTree algorithm is used to construct a spatial index for boundary feature points, which quickly realizes neighborhood search between point clouds and provides efficient data query support for subsequent clustering. Subsequently, based on the Euclidean distance clustering algorithm, a Euclidean distance threshold ε is set, and adjacent boundary feature points with a distance less than ε are clustered into a cluster. Each cluster is regarded as a "line segment candidate", thereby realizing the grouping of boundary feature points and separating boundary points corresponding to different structures. Finally, for each cluster-formed "line segment candidate", a random sampling consensus algorithm is used to fit a straight line to obtain the feature line corresponding to each candidate. The line equation is expressed in parametric form, and the set of three-dimensional points corresponding to each feature line is recorded.

[0055] Given the direction vector of the feature line and the number of 3D points it contains, we select the transverse and longitudinal feature lines related to the pallet fork hole structure from all the fitted feature lines. The specific selection logic is as follows: First, for each characteristic straight line, calculate its direction vector v in the forklift coordinate system, and determine the line type through directionality analysis. If the direction vector is closer to the horizontal direction, it is determined to be a horizontal candidate line; if the direction vector is closer to the vertical direction, it is determined to be a vertical candidate line.

[0056] Next, a point count filtering threshold is set, where the point count threshold for horizontal feature lines is greater than that for vertical feature lines. Specifically, the thresholds are set as follows: horizontal feature lines must contain more than 150 three-dimensional points, and vertical feature lines must contain more than 8 three-dimensional points. This threshold design is adapted to the characteristics of the pallet structure; horizontal feature lines correspond to the bottom edge (long straight edge) of the pallet top, requiring more points; vertical feature lines correspond to the outer boundary (short straight edge) of the pallet's central pillars, requiring relatively fewer points. Finally, all the horizontal and vertical candidate lines that have passed the above two steps are sorted by distance from the center point of the point cloud. The two horizontal candidate lines closest to the center are selected and retained. F The line with the higher direction is selected as the final horizontal feature line; at the same time, the two vertical candidate lines closest to the center are selected to enhance the robustness of the screening, eliminate background noise or boundary interference, and focus on the key structural lines in the visual center region.

[0057] The specific process for calculating the intersection points of the horizontal feature lines obtained after filtering with each vertical feature line is as follows: Based on the parametric equations of the horizontal and vertical feature lines, the spatial coordinates of the intersection point of the two lines are solved using analytical geometry methods; for example... Figure 5 As shown, since the horizontal feature line corresponds to the bottom line of the pallet top and the vertical feature line corresponds to the outer boundary of the pallet's central pillar, the intersection of the two is the intersection point of the pallet's "T-shaped" structure, which is the connection point between the outer edge of the pallet leg and the bottom edge. This intersection point serves as the positioning reference point for the pallet fork holes. Ultimately, two positioning reference points A and B are obtained, corresponding to the positions of the two fork holes on the pallet, respectively.

[0058] Based on the three-dimensional coordinates of the positioning reference point and the pallet plane normal vector, the complete spatial position of the pallet fork hole is determined comprehensively, including: Based on the three-dimensional coordinates of two positioning reference points A and B, and combined with the standard dimensions of the pallet fork holes, the center three-dimensional coordinates of each fork hole are calculated. Combined with the pallet plane normal vector, the relative relationship between the extension direction of the fork hole and the pallet plane is determined to ensure that the extension direction of the fork hole is consistent with the insertion direction of the fork teeth. Based on the center coordinates of the fork hole, the extension direction, and the standard hole diameter, the effective range of the fork hole in three-dimensional space is defined, and the spatial position information of the fork hole is fully described, including position, direction, and size parameters.

[0059] In this embodiment, feature lines are obtained by clustering and fitting boundary points. Based on the direction and number of points, the horizontal and vertical feature lines representing the bottom of the pallet and the legs are selected. Their intersection points are calculated as precise reference points. Combined with the plane normal vector, robust and accurate positioning of the spatial pose of the pallet fork hole is achieved, effectively overcoming the problems of inaccurate positioning and susceptibility to interference in traditional methods under complex environments.

[0060] In one embodiment of this application, the step of filtering out horizontal and vertical feature lines from the feature lines based on the direction vector of the feature lines and the number of three-dimensional points they contain includes: For each characteristic line, determine the direction vector of each characteristic line in the forklift coordinate system; Compare the absolute value of the dot product between the direction vector and the X-axis unit vector of the fork coordinate system, and the absolute value of the dot product between the direction vector and the Y-axis unit vector of the fork coordinate system; If the absolute value of the dot product between the direction vector and the X-axis unit vector is large, the corresponding feature line is determined to be a horizontal candidate line; if the absolute value of the dot product between the direction vector and the Y-axis unit vector is large, the corresponding feature line is determined to be a vertical candidate line. From the horizontal candidate lines, feature lines containing more than a first number threshold are selected as the horizontal feature lines. From the longitudinal candidate lines, feature lines containing more than a second number threshold are selected as the longitudinal feature lines. Wherein, the first quantity threshold is greater than the second quantity threshold.

[0061] In this embodiment, each fitted straight line has a direction vector. .

[0062] By comparing the absolute value of the dot product between the unit direction vector and the unit vector of the forklift coordinate system axis, the feature line is determined to be either a horizontal or vertical candidate line. The specific logic is as follows: Determine the X coordinate system of the forklift F Axial unit vector Y F Axial unit vector ; Calculate X respectively F The absolute value of the dot product of the unit vector of the axis and the direction vector, and Y F The absolute value of the dot product of the axial unit vector and the direction vector; like This indicates that the main direction of the characteristic line is related to X. F If the line aligns with the horizontal axis, it is considered a horizontal candidate line; otherwise, it indicates that the main direction of the characteristic line is aligned with the Y-axis. FIf the axis (vertical direction) is consistent, it is determined to be a vertical candidate line.

[0063] Different threshold values ​​are set to perform secondary filtering on horizontal and vertical candidate lines, retaining valid feature lines that conform to the pallet structure characteristics. The specific setting and filtering logic is as follows: Based on the characteristics of the pallet structure, the horizontal feature lines correspond to the bottom line of the pallet top (long straight edge) and contain a larger number of three-dimensional points; the vertical feature lines correspond to the outer boundary of the pallet's central pillars (short straight edge) and contain a relatively smaller number of three-dimensional points. Therefore, a first quantity threshold is set to 150, and a second quantity threshold is set to 8, with the first quantity threshold being greater than the second quantity threshold. Count the number of points N in the three-dimensional point set corresponding to each horizontal and vertical candidate line; If the number of points N for the horizontal candidate line is greater than 150, the line is retained as the final horizontal feature line; if the number of points N for the vertical candidate line is greater than 8, the line is retained as the final vertical feature line; if the number of points does not reach the corresponding threshold, the line is judged as noise and discarded.

[0064] To further improve the robustness of feature line selection and eliminate background interference, the selected horizontal and vertical feature lines are optimized: Calculate the coordinates of the center point of the point cloud based on the 3D coordinates of the filtered point cloud; Calculate the distance from each horizontal and vertical feature line to the center point of the point cloud. Sort the horizontal feature lines in ascending order of distance, retain the two closest lines, and then select the Y-coordinate system of the forklift from them. F The line with the higher directional (height) coordinate is selected as the core horizontal feature line; the vertical feature lines are sorted in ascending order of distance, and the two closest lines are retained as the core vertical feature lines to ensure that the selected feature lines focus on the key structures in the center area of ​​the tray.

[0065] In this embodiment, the horizontal / vertical direction is accurately distinguished by the dot product relationship with the coordinate axis, and a threshold for the number of differences is set, thereby accurately identifying the feature lines representing the key structure of the pallet, providing a data basis for the subsequent accurate positioning of the fork holes.

[0066] In one embodiment of this application, calculating the three-dimensional relative pose deviation between the inner edge of the fork tooth and the spatial position of the pallet fork hole includes: Based on the spatial position of the inner edge of the fork tooth and the spatial position of the pallet fork hole, calculate the lateral offset, vertical offset, and yaw angle; Wherein, the lateral offset is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the X-axis direction of the fork coordinate system; The vertical offset is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the Y-axis direction of the fork coordinate system. The yaw angle is the angle calculated based on the extension direction of the fork teeth and the normal direction of the pallet plane.

[0067] In this embodiment, the basic data for calculation includes: the average position coordinates of the inner edge of the fork teeth in the fork coordinate system, and the three-dimensional coordinates of the positioning reference point of the pallet fork hole, while extracting the pallet plane normal vector.

[0068] Lateral offset calculation: The lateral offset is the difference between the coordinate of the inner edge of the fork tooth on the X-axis of the fork coordinate system and the X-axis coordinate of the pallet fork hole, reflecting the horizontal positional deviation between the two.

[0069] Vertical offset calculation: The vertical offset is the difference between the coordinate of the inner edge of the fork tooth on the Y-axis of the fork coordinate system and the Y-axis coordinate of the pallet fork hole, reflecting the positional deviation of the two in the height direction.

[0070] Yaw angle calculation: fork tooth extension direction and fork coordinate system Z F With the axis aligned, take Z. F The unit vector of the axis; combined with the normal vector of the pallet plane, the angle between the two is obtained through the formula for calculating the angle between the vectors, which is the yaw angle, reflecting the attitude deviation between the fork teeth and the pallet plane.

[0071] In this embodiment, by calculating the above-mentioned deviations, the position and attitude deviations in three-dimensional space can be fully covered, providing complete data for precise guidance and adjustment.

[0072] In one embodiment of this application, generating and outputting guidance information based on the three-dimensional relative pose deviation includes: Determine whether the three-dimensional relative pose deviation meets the preset alignment conditions; If the conditions are met, a message indicating successful alignment will be generated. If not satisfied, then based on the type and value of the three-dimensional relative pose deviation, an operation instruction is generated to guide the adjustment of the forks; The prompts or operation instructions are output in at least one human-computer interaction mode.

[0073] In this embodiment, based on the accuracy requirements of the telescopic boom forklift alignment operation and the matching dimensions of the pallet fork holes and fork teeth, the alignment success conditions are preset. The specific parameters can be as follows: Lateral offset threshold: ΔX≤±30mm, where ΔX is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the XF axis direction (horizontal lateral) of the fork coordinate system; Vertical offset threshold: ΔY≤±30mm, where ΔY is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the YF axis direction (vertical height) of the fork coordinate system; Yaw angle threshold: ΔYaw≤±1.5°, where ΔYaw is the angle between the direction of the fork teeth and the direction of the normal vector of the pallet plane; The alignment condition is deemed met only if the lateral offset, vertical offset, and yaw angle in the three-dimensional relative pose deviation all meet the corresponding threshold requirements mentioned above; if any deviation exceeds the corresponding threshold, the alignment condition is deemed not met.

[0074] If all three deviation parameters meet the threshold requirements, output a "successful alignment" status result; if any one deviation parameter exceeds the threshold requirements, output a "failed alignment" status result, and record the deviation type and corresponding actual value that exceeded the threshold.

[0075] Successful alignment prompt message generation: If the alignment is determined to be "successful", a clear alignment success prompt message is generated. The core content is "position is aligned, one-click insertion is possible", which also includes a confirmation mark, prompting the operator to start the one-click insertion function and control the forks to automatically insert into the pallet fork holes. Operation command generation when alignment fails: If the alignment is determined to be "alignment failed", a targeted operation command is generated based on the deviation type exceeding the threshold and the actual value. The specific rules are as follows: Yaw angle deviation (ΔYaw actual): If ΔYaw actual > +1.5°, generate the command "Slightly adjust vehicle direction clockwise"; if ΔYaw actual < -1.5°, generate the command "Slightly adjust vehicle direction counterclockwise". Vertical offset deviation (ΔY actual): If ΔY actual > +30mm, generate the command "Raise fork teeth"; if ΔY actual < -30mm, generate the command "Lower fork teeth". Lateral offset deviation (ΔX actual): If ΔX actual > +30mm, generate the command "Slightly adjust forklift loading to the right"; if ΔX actual < -30mm, generate the command "Slightly adjust forklift loading to the left". The generated prompts or operation instructions will be output through at least one human-computer interaction method to ensure that operators can obtain guidance information in a timely and accurate manner. The specific output methods are as follows: Graphical prompts (vehicle-mounted display / external display device): Real-time display of the relative position of the fork tines and the pallet, including: the pallet outline representing the target position, a fork tin diagram representing the current fork tin posture, arrow indicators representing the adjustment direction (such as left, right, raise, lower, rotate, etc.), and real-time error information; when the deviation value exceeds the preset threshold, it is highlighted in red, and when the insertion conditions are met, it is highlighted in green. When all error values ​​meet the threshold, it switches to a green prompt box and displays "Aligned, press and hold the one-click insertion button"; Text prompts: Display prompts or operation instructions in concise text in a prominent area of ​​the display screen, such as "Position aligned, insert with one click", "Slightly adjust the vehicle direction clockwise", "Raise the fork", etc., to clearly guide the operator to make adjustments; Voice prompts: A loudspeaker is installed in the control room to output voice information corresponding to the text prompts. The voice broadcast is clear and concise, such as "Position is aligned, insert with one click" and "Slightly adjust the forklift to the right for loading", ensuring that the operator can still get effective guidance when the view is inconvenient or the attention is distracted.

[0076] In one embodiment, based on the type and value of the three-dimensional relative pose deviation, an operation instruction for guiding the adjustment of the forks is generated, including: In the case of multiple deviations in the three-dimensional relative pose deviation, operation instructions are generated sequentially for the deviations that do not meet the conditions, according to a preset priority order. The priority order is as follows: the priority of the adjustment command for vertical offset is higher than that for lateral offset, and lower than that for yaw angle adjustment.

[0077] In this embodiment, when multiple deviations in the three-dimensional relative pose deviation, such as lateral offset, vertical offset, and yaw angle, fail to meet preset conditions, the system generates corresponding operation commands sequentially according to a preset priority order. The priority order is explicitly set as follows: yaw angle adjustment commands have the highest priority, followed by vertical offset adjustment commands, and lateral offset adjustment commands have the lowest priority. This prioritization method ensures that the operator can focus on key deviations in a clear order, reducing invalid operations, improving the efficiency and accuracy of alignment adjustments, and guaranteeing the safe and smooth operation of high-altitude alignment work.

[0078] Figure 6 A schematic diagram of the alignment guide device provided in another embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0079] Reference Figure 6 The alignment guide device 600 may include: The acquisition module 601 is used to acquire three-dimensional depth information of the work area, which includes the fork teeth of the forks in the work equipment and the pallet fork holes on the shelf. The determining module 602 is used to determine the spatial position of the inner edge of the fork tooth and the spatial position of the pallet fork hole based on the three-dimensional depth information. Calculation module 603 is used to calculate the three-dimensional relative pose deviation between the inner edge of the fork tooth and the spatial position of the pallet fork hole; Output module 604 is used to generate and output guidance information based on the three-dimensional relative pose deviation to guide the alignment adjustment of the fork teeth and the pallet fork hole.

[0080] Optionally, module 602 includes: The first transformation submodule is used to perform line detection and three-dimensional coordinate transformation on the color image based on the depth image to obtain a set of point clouds of the inner edge of the fork tooth in the coordinate system of the depth camera. The depth camera is set on the fork and is used to acquire color images and depth images. The second transformation submodule is used to transform the point cloud set to a preset fork coordinate system; The calculation submodule is used to calculate the average coordinates of the point cloud set in the preset direction after transformation in the fork coordinate system, so as to obtain the spatial position of the inner edge of the fork tooth.

[0081] Optionally, module 602 includes: The first determining submodule is used to determine the position of the tray in the color image using a pre-trained tray detection model, and obtain the two-dimensional positioning area corresponding to the tray; The processing submodule is used to perform interference point removal processing on the original point cloud data generated by the depth image, and retain the effective point cloud within a preset distance range in the depth camera coordinate system. The filtering submodule is used to convert the two-dimensional positioning area into a spatial positioning range in the depth camera coordinate system, and filter out the valid point cloud that falls within the spatial positioning range; The fitting submodule is used to perform plane fitting on the filtered point cloud and determine the plane normal vector corresponding to the tray plane. The extraction submodule is used to extract the boundary feature points of the filtered point cloud and use the boundary feature points and the plane normal vector to determine the spatial position of the pallet fork hole.

[0082] Optionally, the fitting submodule includes: The fitting unit is used to fit the filtered point cloud to obtain multiple candidate planes; The first calculation unit is used to calculate the comprehensive score of each candidate plane based on the centroid height, normal vector direction and the number of three-dimensional points contained therein. The selection unit is used to select the candidate plane with the highest comprehensive score as the tray plane.

[0083] Optionally, extract submodules, including: Clustering units are used to cluster boundary feature points and fit feature lines in different directions; A filtering unit is used to filter out horizontal and vertical feature lines from the feature lines based on the direction vector of the feature lines and the number of three-dimensional points contained therein. The second calculation unit is used to calculate the intersection of the selected transverse feature lines and longitudinal feature lines to obtain the positioning reference point of the pallet fork hole. The first determining unit is used to determine the spatial position of the pallet fork hole based on the three-dimensional coordinates and plane normal vector of the positioning reference point.

[0084] Optionally, the filtering unit includes: Determine the sub-unit, which is used to determine the direction vector of each characteristic line in the forklift coordinate system for each characteristic line; The comparison subunit is used to compare the absolute value of the dot product between the direction vector and the X-axis unit vector of the fork coordinate system, and the absolute value of the dot product between the direction vector and the Y-axis unit vector of the fork coordinate system. The determination subunit is used to determine the corresponding feature line as a horizontal candidate line if the absolute value of the dot product between the direction vector and the X-axis unit vector is large; and to determine the corresponding feature line as a vertical candidate line if the absolute value of the dot product between the direction vector and the Y-axis unit vector is large. The first filtering subunit is used to filter out the feature lines from the horizontal candidate lines that contain more than a first number threshold as the horizontal feature lines. The second filtering subunit is used to filter out the feature lines from the longitudinal candidate lines that contain more than a second number threshold as the longitudinal feature lines. Wherein, the first quantity threshold is greater than the second quantity threshold.

[0085] Optionally, the calculation module 603 is specifically used for: Based on the spatial position of the inner edge of the fork tooth and the spatial position of the pallet fork hole, calculate the lateral offset, vertical offset, and yaw angle; Wherein, the lateral offset is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the X-axis direction of the fork coordinate system; The vertical offset is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the Y-axis direction of the fork coordinate system. The yaw angle is the angle calculated based on the extension direction of the fork teeth and the normal direction of the pallet plane.

[0086] Optionally, the output module 604 includes: The determination submodule is used to determine whether the three-dimensional relative pose deviation meets the preset alignment conditions; The first generation submodule is used to generate a prompt message indicating successful alignment if the conditions are met. The second generation submodule is used to generate operation instructions to guide the adjustment of the forks based on the type and value of the three-dimensional relative pose deviation if the conditions are not met. The output submodule is used to output the prompt information or operation instructions in at least one human-computer interaction mode.

[0087] Optionally, the second generation submodule is specifically used for: In the case of multiple deviations in the three-dimensional relative pose deviation, operation instructions are generated sequentially for the deviations that do not meet the conditions, according to a preset priority order. The priority order is as follows: the priority of the adjustment command for vertical offset is higher than that for lateral offset, and lower than that for yaw angle adjustment.

[0088] Furthermore, in conjunction with the methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores program instructions; when these program instructions are executed by a processor, they implement any of the methods in the above embodiments.

[0089] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0090] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0091] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0092] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0093] The functional modules shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on machine-readable media or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer grids such as the Internet, intranets, etc.

[0094] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0095] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0096] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A positioning guidance method, characterized in that, The method includes: Collect three-dimensional depth information of the work area, which includes the fork teeth of the forks in the work equipment and the pallet fork holes on the shelf; Based on the three-dimensional depth information, the spatial position of the inner edge of the fork tooth and the spatial position of the pallet fork hole are determined. Calculate the three-dimensional relative pose deviation between the inner edge of the fork tooth and the spatial position of the pallet fork hole; Based on the three-dimensional relative pose deviation, guidance information is generated and output to guide the alignment adjustment of the fork teeth with the pallet fork holes.

2. The method as described in claim 1, characterized in that, The three-dimensional depth information includes color images and depth images; The spatial position of the inner edge of the fork tooth is determined by the following steps: Based on the depth image, line detection and three-dimensional coordinate transformation are performed on the color image to obtain a set of point clouds of the inner edge of the fork tooth in the coordinate system of the depth camera. The depth camera is set on the fork and is used to acquire color images and depth images. The point cloud set is converted to a preset forklift coordinate system; The average coordinates of the point cloud set in the preset direction after transformation are calculated to obtain the spatial position of the inner edge of the fork tooth.

3. The method as described in claim 1, characterized in that, The three-dimensional depth information includes color images and depth images; The spatial position of the pallet fork hole is determined by the following steps: The position of the tray in the color image is determined using a pre-trained tray detection model, and the two-dimensional positioning region corresponding to the tray is obtained. The original point cloud data generated from the depth image is processed to remove interference points, and the effective point cloud within a preset distance range in the depth camera coordinate system is retained; The two-dimensional positioning area is converted into a spatial positioning range in the depth camera coordinate system, and valid point clouds falling within the spatial positioning range are selected. Perform plane fitting on the filtered point cloud to determine the plane normal vector corresponding to the tray plane; Extract the boundary feature points of the filtered point cloud, and use the boundary feature points and the plane normal vector to determine the spatial position of the pallet fork hole.

4. The method as described in claim 3, characterized in that, The step of performing planar fitting on the filtered point cloud to determine the tray plane includes: Multiple candidate planes are obtained by fitting the filtered point cloud; For each candidate plane, a comprehensive score is calculated based on the centroid height, normal vector direction, and the number of 3D points it contains. The candidate plane with the highest overall score is selected as the tray plane.

5. The method as described in claim 3, characterized in that, Determining the spatial position of the pallet fork hole using the boundary feature points and the plane normal vector includes: Clustering of boundary feature points and fitting them yields feature lines in different directions; Based on the direction vector of the feature line and the number of three-dimensional points it contains, horizontal and vertical feature lines are selected from the feature lines. The intersection of the selected transverse and longitudinal feature lines is calculated to obtain the positioning reference point of the pallet fork hole; Based on the three-dimensional coordinates and plane normal vector of the positioning reference point, the spatial position of the pallet fork hole is determined.

6. The method as described in claim 5, characterized in that, The step of filtering out horizontal and vertical feature lines from the feature lines based on the direction vector of the feature lines and the number of three-dimensional points they contain includes: For each characteristic line, determine the direction vector of each characteristic line in the forklift coordinate system; Compare the absolute value of the dot product between the direction vector and the X-axis unit vector of the fork coordinate system, and the absolute value of the dot product between the direction vector and the Y-axis unit vector of the fork coordinate system; If the absolute value of the dot product between the direction vector and the X-axis unit vector is large, the corresponding feature line is determined to be a horizontal candidate line; if the absolute value of the dot product between the direction vector and the Y-axis unit vector is large, the corresponding feature line is determined to be a vertical candidate line. From the horizontal candidate lines, feature lines containing more than a first number threshold are selected as the horizontal feature lines. From the longitudinal candidate lines, feature lines containing more than a second number threshold are selected as the longitudinal feature lines. Wherein, the first quantity threshold is greater than the second quantity threshold.

7. The method as described in claim 1, characterized in that, The calculation of the three-dimensional relative pose deviation between the inner edge of the fork tooth and the spatial position of the pallet fork hole includes: Based on the spatial position of the inner edge of the fork tooth and the spatial position of the pallet fork hole, calculate the lateral offset, vertical offset, and yaw angle. Wherein, the lateral offset is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the X-axis direction of the fork coordinate system; The vertical offset is the coordinate difference between the inner edge of the fork tooth and the pallet fork hole in the Y-axis direction of the fork coordinate system. The yaw angle is the angle calculated based on the extension direction of the fork teeth and the normal direction of the pallet plane.

8. The method as described in claim 1, characterized in that, The process of generating and outputting guidance information based on the three-dimensional relative pose deviation includes: Determine whether the three-dimensional relative pose deviation meets the preset alignment conditions; If the conditions are met, a message indicating successful alignment will be generated. If not satisfied, then based on the type and value of the three-dimensional relative pose deviation, an operation instruction is generated to guide the adjustment of the forks; The prompts or operation instructions are output in at least one human-computer interaction mode.

9. The method as described in claim 8, characterized in that, The step of generating operation instructions to guide the adjustment of the forks based on the type and value of the three-dimensional relative pose deviation includes: In the case of multiple deviations in the three-dimensional relative pose deviation, operation instructions are generated sequentially for the deviations that do not meet the conditions, according to a preset priority order. The priority order is as follows: the priority of the adjustment command for vertical offset is higher than that for lateral offset, and lower than that for yaw angle adjustment.

10. An alignment guide device, characterized in that, The device includes: The acquisition module is used to acquire three-dimensional depth information of the work area, which includes the fork teeth of the forks in the work equipment and the pallet fork holes on the shelf; The determination module is used to determine the spatial position of the inner edge of the fork tooth and the spatial position of the pallet fork hole based on the three-dimensional depth information. The calculation module is used to calculate the three-dimensional relative pose deviation between the inner edge of the fork tooth and the spatial position of the pallet fork hole; The output module is used to generate and output guidance information based on the three-dimensional relative pose deviation to guide the alignment adjustment of the fork teeth and the pallet fork hole.

11. A positioning and guiding device, characterized in that, The device includes: A depth camera, mounted at the end of the forks, is used to collect three-dimensional depth information of the work area; A processor, communicatively connected to the depth camera, is configured to execute the method as described in any one of claims 1 to 9; A human-computer interaction device is communicatively connected to the processor and is used to output guidance information.

12. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the alignment boot method according to any one of claims 1 to 9.

Images