An adaptive pallet insertion method and system for stacking forklifts

Abstract

This application discloses an adaptive pallet interlocking method and system for a stacker forklift, belonging to the technical field of forklift control technology. The adaptive pallet interlocking method includes: acquiring pallet images captured by a depth camera and inputting the pallet images into an image recognition model to obtain pallet pose information; calculating the pose deviation value between the pallet and the forks based on the pallet pose information; if the pose deviation value is less than a critical value and the photoelectric switch is not blocked, then controlling the stacker forklift to perform an interlocking operation on the pallet based on the pose deviation value and positioning information; wherein, the positioning information is obtained by fusing data from an odometer and an inertial measurement unit; during the data fusion process, the odometer uses Kalman filtering for state estimation, and the inertial measurement unit uses extended Kalman filtering for state estimation. This application can improve the pallet interlocking accuracy of a stacker forklift.

Description

Technical Field

[0001] This application relates to the field of forklift control technology, and in particular to an adaptive pallet insertion method and system for stacking forklifts. Background Technology

[0002] Stacking forklifts require inserting the forks into the pallets. In related technologies, stacking forklifts use QR code labels affixed to fixed positions on the racks for positioning assistance, and limit switches and photoelectric sensors are installed to locate the pallet positions. This requires a lot of preparation work and cannot achieve high-precision motion control.

[0003] Therefore, how to improve the pallet insertion accuracy of stacker forklifts is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide an adaptive pallet interlocking method, system, stacker forklift, and storage medium for stacker forklifts, which can improve the pallet interlocking accuracy of stacker forklifts.

[0005] To address the aforementioned technical problems, this application provides an adaptive pallet insertion method for a stacking forklift, applicable to a stacking forklift equipped with a depth camera, odometer, and inertial measurement unit. The fork tips of the stacking forklift are equipped with photoelectric switches. The adaptive pallet insertion method for the stacking forklift includes:

[0006] The tray image captured by the depth camera is obtained, and the tray image is input into an image recognition model to obtain the tray pose information;

[0007] Calculate the positional deviation between the pallet and the forks based on the pallet positional information;

[0008] If the pose deviation value is less than a critical value and the photoelectric switch is not blocked, the stacking forklift is controlled to perform an insertion operation on the pallet based on the pose deviation value and the positioning information; wherein, the positioning information is obtained by fusing data from the odometer and the inertial measurement unit; during the data fusion process, the odometer uses Kalman filtering for state estimation, and the inertial measurement unit uses extended Kalman filtering for state estimation.

[0009] Optionally, before inputting the tray image into the image recognition model, the method further includes:

[0010] Acquire multiple tray sample images with depth information;

[0011] The tray sample image is labeled, and the labeled tray sample image is used to train the image recognition model.

[0012] Optionally, after calculating the pose deviation value between the pallet and the forks based on the pallet pose information, the method further includes:

[0013] If the pose deviation value is greater than or equal to the critical value, the pose of the stacker forklift is adjusted until the updated pose deviation value is less than the critical value.

[0014] Optionally, the stacker forklift is a forklift with a single steering wheel structure, and the stacker forklift includes one main steering wheel and two driven wheels;

[0015] Accordingly, before controlling the stacker forklift to perform the insertion operation on the pallet based on the pose deviation value and positioning information, the method further includes:

[0016] Establish a motion control model for the stacker forklift so that interpolation operations can be performed based on the motion control model.

[0017] Optional, also includes:

[0018] The odometer is updated based on the Kalman filter algorithm to obtain the system predicted state variables and system covariance at time T.

[0019] The inertial measurement unit is updated based on the extended Kalman filter algorithm. During the update process of the inertial measurement unit, the system predicted state quantity and system covariance at time T are set to the system state quantity and covariance at time T-1 to obtain the state prediction result.

[0020] The location information is determined based on the most recently generated state prediction result.

[0021] Optional, also includes:

[0022] During the process of controlling the stacker forklift to perform the insertion operation on the pallet, it is determined whether the photoelectric switch is blocked;

[0023] If so, the stacker forklift is controlled to stop the insertion operation, and it is determined that the forks of the stacker forklift are not aligned with the pallet.

[0024] Optionally, the stacker forklift is also equipped with a lidar;

[0025] Correspondingly, it also includes:

[0026] The scene map is created using the aforementioned lidar;

[0027] During the process of controlling the stacking forklift to perform the insertion operation on the pallet, the positioning information is verified using the scene map.

[0028] This application also provides an adaptive pallet insertion system for a stacker forklift, applied to a stacker forklift equipped with a depth camera, odometer, and inertial measurement unit. The fork tips of the stacker forklift are equipped with photoelectric switches. The adaptive pallet insertion system for the stacker forklift includes:

[0029] The pose determination module is used to acquire the tray image captured by the depth camera and input the tray image into the image recognition model to obtain the tray pose information;

[0030] The deviation detection module is used to calculate the positional deviation value between the pallet and the forks based on the pallet positional information;

[0031] An interpolation control module is used to control the stacking forklift to perform an interpolation operation on the pallet based on the pose deviation value and positioning information if the pose deviation value is less than a critical value and the photoelectric switch is not blocked; wherein, the positioning information is information obtained by fusing data from the odometer and the inertial measurement unit; during the data fusion process, the odometer uses Kalman filtering for state estimation, and the inertial measurement unit uses extended Kalman filtering for state estimation.

[0032] This application also provides a storage medium storing a computer program thereon, which, when executed, implements the steps of the above-described stacker forklift adaptive pallet insertion method.

[0033] This application also provides a stacking forklift, including a depth camera, an odometer, an inertial measurement unit, a memory, and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the above-described stacking forklift adaptive pallet insertion method.

[0034] This application utilizes a depth camera to capture images of a pallet, and then determines the pallet's pose information based on the images. When the pose deviation is less than a critical value and the photoelectric switch is not obstructed, this application controls the stacking forklift to perform an insertion operation on the pallet based on the pose deviation and positioning information. The positioning information of the stacking forklift is obtained by fusing data from an odometer and an inertial measurement unit (IMU). The odometer and IMU provide distance, acceleration, and angular velocity information, respectively. By fusing data through Kalman filtering and extended Kalman filtering, more accurate and stable positioning information can be obtained, thereby reducing the impact of accumulated errors and noise. Therefore, this application can improve the pallet insertion accuracy of the stacking forklift. This application also provides an adaptive pallet insertion system for a stacking forklift, a storage medium, and a stacking forklift, all of which have the above-mentioned beneficial effects, which will not be elaborated further here. Attached Figure Description

[0035] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart illustrating an adaptive pallet interpolation method for a stacking forklift provided in this application embodiment;

[0037] Figure 2 This is a schematic diagram of the overall structure of a stacking forklift provided in an embodiment of this application;

[0038] Figure 3 A schematic diagram of the mounting position of a depth camera provided in an embodiment of this application;

[0039] Figure 4 A schematic diagram of the installation location of a host controller provided in an embodiment of this application;

[0040] Figure 5 This is a flowchart illustrating a pallet coordinate information output method provided in an embodiment of this application.

[0041] Figure 6 A schematic diagram of the pose of the tray center in the camera coordinate system provided for an embodiment of this application;

[0042] Figure 7 This is a schematic diagram of an interpolation constraint provided in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram of a stacking forklift kinematics model provided in an embodiment of this application;

[0044] Figure 9 This is a schematic diagram of motion analysis provided in an embodiment of this application;

[0045] Figure 10 A schematic diagram of mileage error provided in an embodiment of this application;

[0046] Figure 11 This is a schematic diagram of a mobile chassis mileage status update provided in an embodiment of this application;

[0047] Figure 12 This is a flowchart illustrating the operation of an automatic stacker truck insertion system provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Please see below. Figure 1 , Figure 1 This is a flowchart illustrating an adaptive pallet interpolation method for a stacking forklift, provided as an embodiment of this application.

[0050] Specific steps may include:

[0051] S101: Acquire the tray image captured by the depth camera, and input the tray image into the image recognition model to obtain the tray pose information;

[0052] This embodiment can be applied to a stacker forklift equipped with a depth camera, odometer, and inertial measurement unit, wherein the fork tips of the stacker forklift are equipped with photoelectric switches.

[0053] In this embodiment, the tray image captured by the depth camera can include both color and depth images. Inputting the tray image into an image recognition model yields the tray pose information. Prior to this step, the image recognition model may be trained to enable it to recognize the tray pose.

[0054] S102: Calculate the positional deviation value between the pallet and the forks based on the pallet positional information;

[0055] The depth camera is mounted on the stacker forklift and can pre-store the relative poses of the depth camera and the forks. In this embodiment, the relative poses remain unchanged. After obtaining the pallet pose information, the pose deviation between the pallet and the forks can be calculated based on the pallet pose information and the aforementioned relative poses. The pose deviation value can include positional deviation and angular deviation.

[0056] S103: If the pose deviation value is less than the critical value and the photoelectric switch is not blocked, then control the stacking forklift to perform an insertion operation on the pallet according to the pose deviation value and the positioning information;

[0057] Prior to this step, a check can be performed to determine if the pose deviation value is less than a critical value and if the photoelectric switch is not obstructed. If the pose deviation value is less than the critical value and the photoelectric switch is not obstructed, it indicates that the forks are in the correct position and there are no obstacles blocking them, allowing the insertion operation to proceed. After calculating the pose deviation value between the pallet and the forks based on the pallet pose information, if the pose deviation value is greater than or equal to the critical value, the pose of the stacker forklift is adjusted until the updated pose deviation value is less than the critical value.

[0058] Specifically, this embodiment can acquire the positioning information of the stacking forklift, and then control the stacking forklift to perform an insertion operation on the pallet based on the pose deviation value and the positioning information. The aforementioned positioning information is obtained by fusing data from an odometer and an inertial measurement unit (IMU), which enhances the accuracy and reliability of the positioning. An odometer typically provides relatively stable displacement information, while an IMU provides acceleration and angular velocity information; combining the two can compensate for their respective shortcomings and improve the accuracy of the positioning information.

[0059] The positioning information is obtained by fusing data from the odometer and the inertial measurement unit (IMU). During the data fusion process, the odometer uses Kalman filtering for state estimation, and the IMU uses extended Kalman filtering for state estimation. Kalman filtering and extended Kalman filtering can fully utilize the complementary characteristics of the odometer and the IMU, reduce their respective limitations, and provide a more accurate and robust system state estimate.

[0060] This embodiment utilizes a depth camera to capture images of the pallet, and then determines the pallet pose information based on these images. When the pose deviation value is less than a critical value and the photoelectric switch is not obstructed, this embodiment controls the stacking forklift to perform an insertion operation on the pallet based on the pose deviation value and positioning information. The positioning information of the stacking forklift is obtained by fusing data from an odometer and an inertial measurement unit (IMU). The odometer and IMU provide distance, acceleration, and angular velocity information, respectively. By fusing the data through Kalman filtering and extended Kalman filtering, more accurate and stable positioning information can be obtained, thereby reducing the impact of accumulated errors and noise. Therefore, this embodiment can improve the pallet insertion accuracy of the stacking forklift.

[0061] As for Figure 1Further description of the corresponding embodiment: Before inputting the tray image into the image recognition model, the image recognition model can be trained in the following way: acquiring multiple tray sample images with depth information; labeling the tray sample images; and using the labeled tray sample images to train the image recognition model. In the above embodiment, the boundaries and poses of the tray sample images can be labeled. After labeling, this labeled data can be used to train the image recognition model. Through iterative training, the image recognition model gradually learns to identify and locate trays from images, and ultimately can accurately perform these tasks in unknown images.

[0062] As for Figure 1 In a further description of the corresponding embodiment, the stacking forklift is a forklift with a single steering wheel structure, comprising one main steering wheel and two driven wheels. Before controlling the stacking forklift to perform the insertion operation on the pallet based on the pose deviation value and positioning information, a motion control model corresponding to the stacking forklift can be established so that the insertion operation can be performed based on the motion control model. Specifically, in this embodiment, the stacking forklift can be controlled to perform the insertion operation on the pallet based on the above-mentioned motion control model, according to the pose deviation value and positioning information.

[0063] As for Figure 1 In a further description of the corresponding embodiment, the positioning information can also be determined in the following way: The odometry is updated based on a Kalman filter algorithm to obtain the system predicted state quantity and system covariance at time T; the inertial measurement unit is updated based on an extended Kalman filter algorithm, and during the update process of the inertial measurement unit, the system predicted state quantity and system covariance at time T are set to the system state quantity and covariance at time T-1 to obtain the state prediction result; the positioning information is determined based on the most recently generated state prediction result.

[0064] At time T, this embodiment uses a Kalman filter to update the odometry state variables and covariance. Then, at time T, these predicted state variables and covariance are used as the initial conditions for the inertial measurement unit (IMU) state update in the extended Kalman filter algorithm. In this way, the extended Kalman filter algorithm can utilize the latest prediction information to further refine the state estimation, ultimately generating positioning information that integrates data from both the odometry and IMU. By using this method, each generated state prediction result can be used as the latest positioning information, improving the accuracy and reliability of the positioning.

[0065] As for Figure 1In a further description of the corresponding embodiment, during the process of controlling the stacking forklift to perform the insertion operation on the pallet, it can also be determined whether the photoelectric switch is blocked; if so, the stacking forklift is controlled to stop the insertion operation and it is determined that the forks of the stacking forklift are not aligned with the pallet; if not, the stacking forklift is controlled to continue performing the insertion operation on the pallet.

[0066] As for Figure 1 In a further description of the corresponding embodiment, the stacking forklift is also equipped with a lidar; correspondingly, this embodiment can also use the lidar to establish a scene map; during the process of controlling the stacking forklift to perform the insertion operation on the pallet, the positioning information is verified using the scene map.

[0067] The process described in the above embodiments is illustrated below through examples in practical applications.

[0068] In related technologies, the automatic insertion and insertion of palletizing forklifts requires numerous auxiliary positioning points and tools. Completing a full automated handling process necessitates extensive preparatory work and lacks adaptability. Failure of auxiliary positioning points can lead to the failure of automated palletizing tasks without the ability to correct it autonomously. Existing pallet recognition technologies vary, as do the training methods and models for pallets. However, they are not widely used in automated handling projects within the forklift industry for several reasons: 1. Kinematic modeling is difficult, requiring modifications to the forklift for steer-by-wire. 2. Forklifts are susceptible to both systematic and non-systematic errors, often resulting in control accuracy that falls short of expectations. High-precision mileage control is necessary to ensure the successful completion of automated handling tasks. Currently, fully automated palletizing systems exist, which rely on QR code labels affixed to fixed locations on the shelves for positioning and limit switches and photoelectric sensors for pallet location, requiring prior preparation. This solution proposes a vision-guided approach that eliminates the need for additional auxiliary sensors. It uses only a depth camera to collect pallet information and train a corresponding dataset, enabling pallet recognition and output of key coordinate information from the pallet center point to the fork center. Finally, the coordinate information is used to calculate the movement trajectory of the stacker forklift to the insertion point, achieving a more efficient and concise automated handling effect.

[0069] Please see Figure 2 , Figure 2 This is a schematic diagram of the overall structure of a stacking forklift provided in an embodiment of this application. Figure 2In this diagram, A1 represents the forks, A2 represents the binocular depth camera, A3 represents the LiDAR, A4 represents the display, A5 represents the host controller (i.e., the upper computer), A6 represents the actuators, and A7 represents the travel and steering electronic control systems. The fork mechanism controls the lifting and lowering of the forks via electronic commands. The binocular depth camera identifies features such as pallets in the environment and outputs the relative position of the pallet opening with respect to the fork center. The LiDAR is used for mapping and navigation. Users can issue work tasks and monitor real-time information of the interlocking system through the display. The host controller processes data from the depth camera and LiDAR, stores trained pallet models, resolves the relative position of the pallet center point to the fork center, and solves the inverse kinematics based on the pose relationship to obtain a series of control commands to move from the current position to the vicinity of the pallet insertion point and complete the interlocking action. The actuators include a steering motor, a travel motor, and a drive wheel integrating the steering drive. The travel and steering electronic control systems receive and parse travel and steering commands from the host control system. Data from the lidar and depth camera is connected to the host controller via a network cable. The host controller has a built-in wireless network card and an inertial measurement unit (IMU). The host controller communicates with the lower-level electronic control unit via a CAN (Controller Area Network) network.

[0070] Please see Figure 3 , Figure 3 This is a schematic diagram of the installation position of a depth camera provided in an embodiment of this application. Figure 3 In the diagram, B1 represents the bracket, B2 represents the depth camera, and B3 represents the fork stop. The depth camera is fixed to the fork stop via the bracket. Since the position of the depth camera is fixed, there is an absolute positional relationship between the depth camera and the bracket. This positional relationship exists between all components of the vehicle. Therefore, the position of the fork center relative to the camera can be obtained through coordinate transformation. The position of the pallet center relative to the camera, obtained subsequently, can also be converted into the position of the pallet center relative to the fork center.

[0071] Please see Figure 4 , Figure 4 This is a schematic diagram showing the installation location of a host controller according to an embodiment of this application. Figure 4 In this diagram, C1 represents the DC controller, C2 represents the switch, and C3 represents the host controller. The DC controller, switch, and host controller are all mounted on the rear upright of the forklift. The host controller uses an FPGA (Field-Programmable Gate Array) architecture with an ARM processor, providing powerful computing capabilities to process information transmitted from the camera and obtain the coordinates of the pallet's center point through algorithmic processing. The host controller connects to the underlying actuators via a CAN network to control the forklift's movement, steering, lifting, and other actions.

[0072] This embodiment provides an automated palletizing and insertion scheme for stacker forklifts. It requires no auxiliary positioning tools, only a depth camera to locate and identify pallets, outputting the relative position of the pallet's center point with respect to the fork center. Then, based on a kinematic model, the motion command for the forklift to move from the current point to the insertion point is calculated and sent to the underlying electronic control system to complete the automated handling task. To ensure the reliability of the task during movement, this embodiment proposes an odometer calibration method based on extended Kalman filtering and forklift kinematic modeling. This method provides high-precision motion control for the stacker forklift, ensuring a high success rate for automated handling tasks.

[0073] The automatic pallet insertion solution for stacking forklifts provided in this embodiment is as follows:

[0074] Collect color images and depth information of the pallet, use the image annotation tool LabelImg to annotate the images, classify and preprocess the training, validation and test datasets, train the image recognition model using YOLOv5s script, and finally use the trained image recognition model to infer new images to obtain the desired pallet recognition result, and output the position and orientation of the pallet center point relative to the fork center (i.e., pallet pose information).

[0075] Please see Figure 5 , Figure 5 The flowchart for outputting pallet coordinate information provided in this application embodiment includes the following steps: acquiring color images and depth information captured by a depth camera; preprocessing the image (noise filtering, color correction); importing the registered image into a trained .rknn model (i.e., an image recognition model); feature extraction (pallet edges and corners); coordinate calculation, outputting the coordinate information of the pallet center relative to the camera; coordinate transformation, obtaining the coordinates of the pallet center relative to the fork center.

[0076] Please see Figure 6 , Figure 6 This is a schematic diagram of the pose of the center of a pallet in the camera coordinate system provided in an embodiment of this application. XYZ is a first coordinate system established with the optical center point O of the depth camera as the origin. The coordinates of the center point a of the pallet in the first coordinate system can be calculated by the algorithm. The position of the fork center and the camera is a fixed mechanical connection. After coordinate transformation, the coordinates of the center of the pallet can be transformed to obtain the coordinate information of the center of the pallet in the second coordinate system with the fork center as the origin, thereby guiding the subsequent automatic interpolation operation.

[0077] To align the forks with the pallet opening, the coordinate information after coordinate transformation can ultimately extract two constraints: one representing the positional deviation (in mm) and the other representing the posture deviation (in °).

[0078] Please see Figure 7 , Figure 7 This is a schematic diagram of an interpolation constraint provided in an embodiment of this application. Figure 7 In this diagram, 'a' represents the center point of the pallet, 'b' represents the fork, 'l' represents the positional deviation, and 'α' represents the posture deviation. When both deviations are within the allowable range and the photoelectric switch signal of the fork tip is unobstructed, the forklift chassis does not need to adjust its posture and can perform the insertion operation.

[0079] To achieve automatic pallet recognition and insertion in any position and posture, the control precision of the forklift chassis is crucial. Steering angle accuracy within 0.1° and mileage accuracy within millimeters are necessary to ensure reliable insertion. Therefore, it is necessary to calibrate the mileage accuracy of the forklift chassis and correct inaccuracies caused by systematic and non-systematic errors during movement. This solution calibrates mileage accuracy by first repeating measurements to obtain the current mileage error and then recalibrating. Secondly, an extended Kalman filter algorithm is used to fuse inertial measurement unit and radar data to calculate the corrected pose of the forklift in the world coordinate system. The pose information of the identified pallet center point relative to the fork center point and the current coordinate information of the forklift are imported into a pre-built inverse kinematics model to obtain the inverse kinematic solution of the forklift's current pose relative to the pallet center point. Finally, the upper controller sends the parsed motion control commands to the lower-level motion control system via CAN communication, thereby controlling the forklift to adjust its pose, align with the pallet opening, and complete the insertion operation.

[0080] Stacker forklifts are a type of single-steering wheel structure; please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of the kinematics model of a stacker forklift provided in an embodiment of this application. In the figure, A1 represents the center of the main steering wheel, A2 and A3 represent the centers of the driven wheels, O represents the center of the driven wheel axle, P represents the rotation center, l represents the wheelbase, and d represents half of the wheel track. w represents the linear velocity of the motor, and ω represents the angular velocity. The value represents the steering angle, r represents the steering radius, and v represents the steering radius. forward Indicates speed in the direction of travel.

[0081] The speed in the forward direction is calculated as follows:

[0082] Equation (1);

[0083] The electronic control unit can obtain the linear velocity by analyzing the encoder data. This embodiment can also read the steering angle from the steering electronic control system. .

[0084] Equation (2);

[0085] Equation (3);

[0086] Please see Figure 9 , Figure 9 This is a schematic diagram of motion analysis provided in an embodiment of this application, where XOY represents the coordinate system. The linear velocity of the motor is represented by ω, the angular velocity by ω, and the coordinates of point N are... The coordinates of point M are If the pose of point O in the coordinate system is used This means that in a controller cycle The process of moving from point M to point N during the motion is as follows:

[0087] Equation (4);

[0088] In the above formula, , And x represents the coordinate along the x-axis. , y represents the coordinate along the y-axis. , 'a' represents the angular coordinate. This represents the angle of rotation of point N in the coordinate system of point O. This represents the rotation angle of point M in the coordinate system of point O; with the horizontal and vertical coordinates and the rotation angle, the pose of the target at different times can be determined.

[0089] Substituting formula 4 into formulas 2 and 3, and simplifying, we get:

[0090] Equation (5);

[0091] Therefore, only the initial position and attitude are needed, while the driving wheel speed and deflection angle can be read as needed. Then, the chassis position and attitude at any time can be derived from equation (5).

[0092] The motion control model is only the foundation of chassis motion control. Achieving high-precision motion control requires not only ensuring the overall attitude of the interpolation system meets interpolation constraints but also calibrating the odometer. Positioning accuracy is inevitably affected by two types of errors: systematic and unsystematic errors, which are unavoidable. Systematic errors arise from inaccurate model parameters, manufacturing errors in the mechanical structure, assembly issues, etc.; unsystematic errors are caused by wheel slippage or uneven ground conditions.

[0093] Please see Figure 10 , Figure 10 This is a schematic diagram of mileage error provided in an embodiment of this application. XOY represents the coordinate system. The diagram shows the desired trajectory A (solid line) and the actual trajectory B (dashed line), wherein... and Let's consider the target position and the actual position after the chassis has completed its movement, respectively. Then, what is the positional error? It can be caused by lateral error With longitudinal error express The attitude error is Therefore, it is necessary to calibrate the position and attitude of the mobile chassis using effective methods. To this end, this solution uses sensor fusion to calibrate the motion control accuracy of the chassis.

[0094] Sensors are often affected by their own refresh rate and environmental factors, causing measurement results to drift, and errors accumulate over time. Therefore, it is necessary to select a suitable filter for processing. The core idea of ​​Kalman filtering (KF) is to combine measurement updates with prediction updates to obtain the best state estimate. Extended Kalman filtering (EKF) differs from Kalman filtering in that it can handle nonlinear systems. Both Kalman filtering and extended Kalman filtering require establishing the system's state equation and observation equation, as shown in equation (6).

[0095] Equation (6);

[0096] in Let t be the state of the system at time t. Let A be the system state at the previous moment, A be the state transition matrix, and B be the control input matrix. The system input... Mapped onto the system state vector, The noise in the prediction process generally follows a Gaussian distribution with a mean of 0, i.e. . The sensor's measured value, Represents the transformation matrix, which will Mapped to the space where the measured value is located, The noise in the measurement process also follows a Gaussian distribution with a mean of 0, i.e. .

[0097] The state estimation in a Kalman filter (KF) consists of two steps: state prediction and state update. State prediction is as follows:

[0098] Equation (7);

[0099] In the formula , The covariance matrix corresponding to the system state variables. Let be the covariance matrix of the system process. The state update consists of the following three steps. , For system state variables, T represents transpose.

[0100] 1) Update the Kalman filter gain : ;

[0101] 2) Update system status: ;

[0102] 3) Update covariance: ;

[0103] Let C represent the covariance matrix corresponding to the system state variables. Let C represent the observation matrix and R represent the measurement noise covariance (which follows a Gaussian distribution).

[0104] The moving chassis moves on a two-dimensional plane, select As a system state variable, and This indicates the coordinates of the chassis on the plane. This represents the angle relative to the initial pose. and It is the velocity along the X and Y directions on the plane, and w is the angular velocity of the chassis itself.

[0105] Odometer readings IMU measurements This is the value after preprocessing. The system state variable is . This represents the angular acceleration of rotation. The odometer's measurement value has a linear relationship with the system state variables; a Kalman filter algorithm is used for state estimation of the odometer. However, the relationship between the inertial measurement unit's measurement value and the system state variables is non-linear; an extended Kalman filter algorithm is used for state estimation of the inertial measurement unit.

[0106] A Kalman filter algorithm is used to estimate the state of the odometer. The measured value is... State variables are defined as follows: The prediction equation for the odometer is as follows:

[0107] Equation (8);

[0108] In the formula:

[0109] Equation (9);

[0110] Equation (10);

[0111] This represents the system noise during odometer state estimation. This represents the covariance matrix of the system process when the odometer performs state estimation.

[0112] This represents a 3×3 identity matrix, with 1s on the diagonal and 0s elsewhere. Represents a 3×3 zero matrix; This is a 6x1 identity matrix. The system state is updated as follows:

[0113] 1) Update the gain of the odometer : ;

[0114] 2) Update the odometer system status: ;

[0115] 3) Update covariance: ;

[0116] The extended Kalman filter algorithm is used to estimate the state of the inertial measurement unit. The state variables are defined as follows: The measured values ​​were selected after preprocessing. , and It is the acceleration along the X and Y directions on the plane. Let be the angular acceleration of the moving chassis rotating about the Z-axis. The prediction equation of the IMU is basically the same as that of the odometer, as shown in equation (11):

[0117] Equation (11);

[0118] This represents the system noise when using an IMU for state estimation. This represents the covariance matrix of the system process when using an IMU for state estimation.

[0119] The mapping from state space to measurement space is as follows:

[0120] Equation (12);

[0121] It is the sampling period of the IMU, and the Jacobian matrix. The state update of the inertial measurement unit is as follows:

[0122] 1) Update the extended Kalman filter gain : ;

[0123] 2) Update system status: ;

[0124] 3) Update covariance: ;

[0125] This represents the Jacobian matrix.

[0126] Kalman filter-based sensor data fusion mainly involves the transfer of sensor data. After updating the observation equation of the previous sensor, the predicted state variables of the system can be obtained. and the system's variance matrix Both are used as the system state variables of the previous moment in the next sensor update process. Covariance The state is updated continuously until the last sensor is updated, at which point the final system state output is obtained. and system covariance matrix Both will be used in the next iteration. From the previous derivation, it can be seen that the state outputs of both the odometer and the inertial measurement unit are... Therefore, the odometer system state output and covariance matrix are used as the system state variables and covariance of the inertial measurement unit at the previous moment to update the state. The update process is as follows: Figure 11 As shown, the predicted output of the mobile chassis can ultimately be obtained as the final odometer information.

[0127] Figure 11 This is a schematic diagram of a mobile chassis mileage status update provided in an embodiment of this application. At times t, t+1, and t+2, state prediction, odometer (ODM) measurement update, and inertial measurement unit (IMU) measurement update can be performed sequentially. In the above process, the published system matrix (i.e., system state variable) is X, the covariance matrix is ​​P, and the input and output system state variables and covariance are as follows: ( ), ( ), ( ), ( ), ( ).

[0128] After using the pallet recognition algorithm to obtain the position and orientation of the pallet relative to the fork center, the derived kinematic model, and the calibrated odometer enable high-precision motion control. At this point, the preliminary work for autonomous pallet recognition and interpolation has been completed. Please refer to [link to relevant documentation]. Figure 12 , Figure 12The flowchart of an automatic pallet insertion system for a stacker truck provided in this application includes the following steps: Locating a pallet and determining its approximate coordinates; approaching the pallet and activating the pallet recognition program (built-in trained model); outputting the distance from the center point of the pallet to the center of the forks. Based on the coordinate information and kinematic model, the motion commands required to complete the insertion action are analyzed and executed. It is determined whether the alignment constraint condition (pose deviation less than a set value) is met; if so, automatic insertion is considered complete; if not, the forklift's pose is fine-tuned based on the mileage calculation result and the detected pose difference, and the alignment constraint condition is determined again. The above embodiment provides an adaptive pallet insertion scheme for a stacker forklift, mainly used for automatically finding and recognizing pallets and completing the insertion action; it also provides a vision-guided automatic pallet insertion scheme, which uses a depth camera to find and recognize the pallet pose and output the coordinates of key information points, while using LiDAR and an inertial measurement unit to update its own positioning. Finally, by modeling the kinematics of the stacker forklift, the motion commands for the forklift to move to the accurate pallet insertion point are solved, thereby controlling the stacker forklift to complete the automatic pallet insertion action.

[0129] This embodiment proposes an automatic interlocking scheme relying on vision guidance without the need for additional sensors. The coordinates of the pallet center obtained from the camera are imported into the subsequent kinematic model, enabling rapid calculation of the control commands for the forklift to move from its current position to the target interlocking position. This approach is not only simple to install but also more robust, successfully completing the interlocking action even if the pallet position deviates from the pre-set position. This embodiment also proposes an odometer calculation method based on extended Kalman filtering, effectively solving the odometer deviation problem caused by systematic and non-systematic errors in the mobile chassis. The calibrated odometer meets the requirements of high-precision motion control, effectively achieving coordinated operation with the forks. Automatic interlocking task failures due to insufficient motion control precision of the mobile chassis are avoided, and repeated calibrations are unnecessary to guarantee interlocking success, enhancing the system's reliability and efficiency.

[0130] This application provides an adaptive pallet insertion system for a stacker forklift. This system can be applied to stacker forklifts equipped with depth cameras, odometers, and inertial measurement units. The fork tips of the stacker forklift are equipped with photoelectric switches. The adaptive pallet insertion system for the stacker forklift includes:

[0131] The pose determination module is used to acquire the tray image captured by the depth camera and input the tray image into the image recognition model to obtain the tray pose information;

[0132] The deviation detection module is used to calculate the positional deviation value between the pallet and the forks based on the pallet positional information;

[0133] An interpolation control module is used to control the stacking forklift to perform an interpolation operation on the pallet based on the pose deviation value and positioning information if the pose deviation value is less than a critical value and the photoelectric switch is not blocked; wherein, the positioning information is information obtained by fusing data from the odometer and the inertial measurement unit; during the data fusion process, the odometer uses Kalman filtering for state estimation, and the inertial measurement unit uses extended Kalman filtering for state estimation.

[0134] This embodiment utilizes a depth camera to capture images of the pallet, and then determines the pallet pose information based on these images. When the pose deviation value is less than a critical value and the photoelectric switch is not obstructed, this embodiment controls the stacking forklift to perform an insertion operation on the pallet based on the pose deviation value and positioning information. The positioning information of the stacking forklift is obtained by fusing data from an odometer and an inertial measurement unit (IMU). The odometer and IMU provide distance, acceleration, and angular velocity information, respectively. By fusing the data through Kalman filtering and extended Kalman filtering, more accurate and stable positioning information can be obtained, thereby reducing the impact of accumulated errors and noise. Therefore, this embodiment can improve the pallet insertion accuracy of the stacking forklift.

[0135] Furthermore, it also includes:

[0136] The model training module is used to acquire multiple tray sample images with depth information before inputting the tray image into the image recognition model; it is also used to annotate the tray sample images and use the annotated tray sample images to train the image recognition model.

[0137] Furthermore, it also includes:

[0138] The pose adjustment module is used to adjust the pose of the stacker forklift if the pose deviation value is greater than or equal to the critical value after calculating the pose deviation value between the pallet and the fork based on the pallet pose information, until the updated pose deviation value is less than the critical value.

[0139] Furthermore, the stacking forklift is a forklift with a single steering wheel structure, and the stacking forklift includes one main steering wheel and two driven wheels;

[0140] Correspondingly, it also includes:

[0141] The modeling module is used to establish a motion control model corresponding to the stacking forklift before controlling the stacking forklift to perform an interlocking operation on the pallet based on the pose deviation value and positioning information, so as to perform the interlocking operation based on the motion control model.

[0142] Furthermore, it also includes:

[0143] The data fusion module is used to update the state of the odometer based on the Kalman filter algorithm to obtain the system predicted state quantity and system covariance at time T; it is also used to update the state of the inertial measurement unit based on the extended Kalman filter algorithm, and during the update process of the inertial measurement unit, the system predicted state quantity and system covariance at time T are set to the system state quantity and covariance at time T-1 to obtain the state prediction result; it is also used to determine the positioning information based on the most recently generated state prediction result.

[0144] Furthermore, it also includes:

[0145] The obstruction detection module is used to determine whether the photoelectric switch is obstructed during the process of controlling the stacking forklift to perform the insertion operation on the pallet; if so, it controls the stacking forklift to stop the insertion operation and determines that the forks of the stacking forklift are not aligned with the pallet.

[0146] Furthermore, the stacker forklift is also equipped with a lidar.

[0147] Correspondingly, it also includes:

[0148] The positioning verification module is used to establish a scene map using the lidar; it is also used to verify the positioning information using the scene map during the process of controlling the stacking forklift to perform an insertion operation on the pallet.

[0149] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and they will not be repeated here.

[0150] This application also provides a storage medium on which a computer program is stored, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0151] This application also provides a stacking forklift, which may include a depth camera, an odometer, an inertial measurement unit, a memory, and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the stacking forklift may also include various network interfaces, power supplies, and other components.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0153] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for adaptive pallet insertion of a stacking forklift, characterized in that, A stacking forklift equipped with a depth camera, odometer, and inertial measurement unit, wherein the fork tips of the stacking forklift are equipped with photoelectric switches, and the adaptive pallet insertion method of the stacking forklift includes: The tray image captured by the depth camera is obtained, and the tray image is input into an image recognition model to obtain the tray pose information; Calculate the positional deviation between the pallet and the forks based on the pallet positional information; If the pose deviation value is less than a critical value and the photoelectric switch is not blocked, the stacker forklift is controlled to perform an insertion operation on the pallet based on the pose deviation value and positioning information. After calculating the pose deviation value between the pallet and the forks based on the pallet pose information, the method further includes: if the pose deviation value is greater than or equal to the critical value, the pose of the stacker forklift is adjusted until the updated pose deviation value is less than the critical value. The positioning information is obtained by fusing data from the odometer and the inertial measurement unit. During the data fusion process, the odometer uses Kalman filtering for state estimation, and the inertial measurement unit uses extended Kalman filtering for state estimation. The state outputs of both the odometer and the inertial measurement unit are... , and This indicates the coordinates of the chassis on the plane. This represents the angle relative to the initial pose. and It is the velocity along the X and Y axes in the plane. The angular velocity of the chassis itself; the odometer system state output and covariance matrix are used as the system state variables and covariance of the inertial measurement unit at the previous moment for state update; The odometer is updated based on the Kalman filter algorithm to obtain the system predicted state variables and system covariance at time T. The inertial measurement unit is updated based on the extended Kalman filter algorithm. During the update process of the inertial measurement unit, the system predicted state quantity and system covariance at time T are set to the system state quantity and covariance at time T-1 to obtain the state prediction result. The location information is determined based on the most recently generated state prediction result.

2. The adaptive pallet insertion method for stacking forklifts according to claim 1, characterized in that, Before inputting the tray image into the image recognition model, the method further includes: Acquire multiple tray sample images with depth information; The tray sample image is labeled, and the labeled tray sample image is used to train the image recognition model.

3. The adaptive pallet insertion method for stacking forklifts according to claim 1, characterized in that, The stacker forklift is a forklift with a single steering wheel structure, which includes one main steering wheel and two driven wheels; Accordingly, before controlling the stacker forklift to perform the insertion operation on the pallet based on the pose deviation value and positioning information, the method further includes: Establish a motion control model for the stacker forklift so that interpolation operations can be performed based on the motion control model.

4. The adaptive pallet insertion method for stacking forklifts according to claim 1, characterized in that, Also includes: During the process of controlling the stacker forklift to perform the insertion operation on the pallet, it is determined whether the photoelectric switch is blocked; If so, the stacker forklift is controlled to stop the insertion operation, and it is determined that the forks of the stacker forklift are not aligned with the pallet.

5. The adaptive pallet insertion method for stacking forklifts according to claim 1, characterized in that, The stacker forklift is also equipped with a lidar. Correspondingly, it also includes: The scene map is created using the aforementioned lidar; During the process of controlling the stacking forklift to perform the insertion operation on the pallet, the positioning information is verified using the scene map.

6. A stacker forklift adaptive pallet insertion system, characterized in that, An adaptive pallet-connecting system for a stacker forklift equipped with a depth camera, odometer, and inertial measurement unit, wherein photoelectric switches are installed at the tips of the forks, includes: The pose determination module is used to acquire the tray image captured by the depth camera and input the tray image into the image recognition model to obtain the tray pose information; The deviation detection module is used to calculate the positional deviation value between the pallet and the forks based on the pallet positional information; The interpolation control module is used to control the stacker forklift to perform an interpolation operation on the pallet based on the pose deviation value and positioning information if the pose deviation value is less than a critical value and the photoelectric switch is not blocked; after calculating the pose deviation value between the pallet and the fork based on the pallet pose information, it further includes: if the pose deviation value is greater than or equal to the critical value, adjusting the pose of the stacker forklift until the updated pose deviation value is less than the critical value. The positioning information is obtained by fusing data from the odometer and the inertial measurement unit (IMU). During the data fusion process, the odometer uses Kalman filtering for state estimation, and the IMU uses extended Kalman filtering for state estimation. The state outputs of both the odometer and the IMU are... , and This indicates the coordinates of the chassis on the plane. This represents the angle relative to the initial pose. and It is the velocity along the X and Y axes in the plane. The angular velocity of the chassis itself; the odometer system state output and covariance matrix are used as the system state variables and covariance of the inertial measurement unit at the previous moment for state update; The data fusion module is used to update the state of the odometer based on the Kalman filter algorithm to obtain the system predicted state quantity and system covariance at time T; it is also used to update the state of the inertial measurement unit based on the extended Kalman filter algorithm, and during the update process of the inertial measurement unit, the system predicted state quantity and system covariance at time T are set to the system state quantity and covariance at time T-1 to obtain the state prediction result; it is also used to determine the positioning information based on the most recently generated state prediction result.

7. A stacking forklift, characterized in that, It includes a depth camera, an odometer, an inertial measurement unit, a memory, and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the stacker forklift adaptive pallet insertion method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the stacker forklift adaptive pallet insertion method as described in any one of claims 1 to 5.

Images