Forklift pallet butt joint method, device and equipment and storage medium

Abstract

The invention provides a forklift pallet butt joint method, device and equipment and a storage medium, and the method comprises the steps that camera equipment is installed on a forklift and calibrated, and internal and external parameters of a camera and relative pose information of the camera and the forklift are obtained; and after a tray butt joint instruction is received, the forklift is controlled to move to a designated position, and an external environment image is shot through a camera. The position of a target tray in the image is determined through a visual identification technology, depth marking is carried out on the image to generate false point cloud data, and then pose information of the tray is obtained through calculation. In addition, the pose information of the forklift is accurately calculated by combining the pose information of the camera and the pose information of the tray, so that the forklift is guided to automatically complete a tray docking task. According to the method, through integrated camera equipment and an advanced calculation processing technology, automatic butt joint of the forklift and the trays is achieved, and the efficiency and accuracy of logistics operation are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a forklift pallet docking method, device, equipment and storage medium. Background Art

[0002] In modern logistics and warehousing, forklift-pallet docking is a core daily task, and its efficiency and precision directly impact the operational efficiency of the entire logistics chain. With the continuous advancement of automation and intelligent technologies, improving the automation and accuracy of forklift-pallet docking while reducing manual intervention has become a key research and practical issue. Traditional forklift operation relies on the operator's experience and skills, which is not only inefficient but also poses safety risks. Especially in complex or changing working environments, the uncertainty of human operation can lead to operational errors, increased material loss, and operational delays. Summary of the Invention

[0003] The main purpose of the present invention is to solve the technical problem that the operation of existing forklifts depends on the experience and skills of the operator, resulting in low efficiency.

[0004] A first aspect of the present invention provides a forklift pallet docking method, the forklift pallet docking method comprising: Installing a camera device on a target forklift and calibrating the camera device and the target forklift to obtain internal and external parameters of the camera device, as well as relative pose information between the camera device and the target forklift; When a forklift-pallet docking instruction is received, controlling the target forklift to move to a target position indicated by the forklift-pallet docking instruction; photographing the external environment with the camera device to obtain a plurality of external environment images, and performing visual recognition on the plurality of external environment images to determine the position of the target pallet in the plurality of external environment images; Performing depth annotation on the plurality of external environment images to obtain depth images corresponding to the plurality of external environment images, and generating pseudo point cloud data of the depth images according to internal parameters and external parameters of the camera device; Inputting the plurality of external environment images into a preset posture detection model to obtain camera posture information of the camera device when taking the plurality of external environment images; determining pallet pose information of the target pallet based on the pseudo point cloud data, the target pallet position in the plurality of external environment images, and the camera pose information; Calculating the forklift pose information of the target forklift according to the camera pose information and the relative pose information; The target forklift is controlled to perform forklift-pallet docking according to the forklift posture information of the target forklift and the pallet posture information of the target pallet.

[0005] Optionally, in a first implementation of the first aspect of the present invention, photographing the external environment with the camera device to obtain multiple external environment images, performing visual recognition on the multiple external environment images, and determining the target pallet position in the multiple external environment images includes: During the process of the target forklift moving to the target position, the camera device captures the external environment to obtain a plurality of external environment images; Performing image denoising and brightness adjustment processing on the plurality of external environment images to obtain a corresponding plurality of pre-processed images; Performing a visual recognition on the plurality of pre-processed images to identify a target tray and specific markers around the target tray in the plurality of pre-processed images; Determining a rough position of the target pallet according to specific markers around the target pallet, and performing image segmentation on the multiple pre-processed images to determine a pallet boundary of the target pallet; The target tray position of the target tray in the plurality of external environment images is calculated according to the tray boundary and the rough position.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the depth annotation of the multiple external environment images to obtain depth images corresponding to the multiple external environment images, and generating pseudo point cloud data of the depth images based on the internal parameters and external parameters of the camera device includes: Performing depth annotation on the plurality of external environment images to obtain initial depth images corresponding to the plurality of external environment images; Performing geometric correction on the initial depth image according to internal parameters of the camera device, and adjusting the geometrically corrected initial depth image according to external parameters of the camera device to obtain a depth image; The depth information in the depth image is converted into a camera coordinate system according to the internal parameters and external parameters of the camera device to obtain pseudo point cloud data of the depth image.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the pose detection model includes an optical flow estimation network, a long short-term memory network, and a fully connected layer; Inputting the plurality of external environment images into a preset posture detection model to obtain camera posture information of the camera device when shooting the plurality of external environment images includes: Generating adjacent image pairs from the plurality of external environment images according to a pairwise adjacency principle, and inputting the adjacent image pairs into a preset posture detection model; extracting optical flow information between the adjacent image pairs through the optical flow estimation network, and extracting motion features of the multiple external environment images based on the optical flow information; Serializing the motion features to obtain an input sequence, and inputting the input sequence into the long short-term memory network to obtain a temporal feature; The time series features are mapped using a fully connected layer to obtain camera pose information corresponding to each continuous external environment image.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the serializing the motion features to obtain an input sequence, and inputting the input sequence into the long short-term memory network to obtain the temporal features includes: Serializing the motion features to obtain an input sequence, inputting the serialized input sequence into a long short-term memory network, and integrating the input sequence in a time dimension through the long short-term memory network to obtain a feature sequence with a time context; Selectively processing the feature sequence through the gating unit in the long short-term memory network and updating the hidden state in the long short-term memory network to obtain the output feature of each time step; The output features of each time step are subjected to time aggregation processing to obtain time series features.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, determining the pallet pose information of the target pallet based on the pseudo point cloud data, the target pallet position in the multiple external environment images, and the camera pose information includes: Generate an area of ​​interest based on the target pallet position information in the plurality of external environment images, and perform point cloud screening on the pseudo point cloud data based on the area of ​​interest to obtain corresponding pseudo point cloud data of interest; performing clustering processing on the pseudo point cloud data of interest to determine a pallet pseudo point cloud corresponding to the target pallet in the pseudo point cloud data of interest; Determine the three-dimensional position and posture of the target pallet in the camera coordinate system according to the pallet pseudo point cloud; The three-dimensional position and posture of the target pallet in the camera coordinate system are converted to the world coordinate system through geometric transformation calculation to obtain the absolute posture information of the target pallet in the world coordinate system; The absolute pose information is calibrated according to the camera pose information to obtain the pallet pose information of the target pallet.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, calculating the forklift pose information of the target forklift based on the camera pose information and the relative pose information includes: Extracting the position and direction of the camera device at different time points according to the camera pose information to obtain a motion trajectory of the camera device; Calculating the relative motion trajectory of the target forklift according to the motion trajectory and the relative position information between the camera device and the target forklift; Smoothing the relative motion trajectory of the target forklift to obtain a smooth motion trajectory; A posture sequence of the target forklift is generated according to the smooth motion trajectory, and forklift posture information of the target forklift is determined according to the posture sequence of the target forklift.

[0011] A second aspect of the present invention provides a forklift pallet docking device, the forklift pallet docking device comprising: a calibration module, configured to install a camera device on a target forklift and calibrate the camera device and the target forklift to obtain internal and external parameters of the camera device, as well as relative pose information between the camera device and the target forklift; a movement control module, configured to control the target forklift to move to a target position indicated by the forklift pallet docking instruction upon receiving the forklift pallet docking instruction; a visual processing module, configured to photograph the external environment using the camera device to obtain a plurality of external environment images, and perform visual recognition on the plurality of external environment images to determine the position of the target pallet in the plurality of external environment images; a pseudo point cloud generation module, configured to perform depth annotation on the plurality of external environment images, obtain depth images corresponding to the plurality of external environment images, and generate pseudo point cloud data of the depth images according to internal and external parameters of the camera device; a camera pose detection module, configured to input the plurality of external environment images into a preset pose detection model to obtain camera pose information of the camera device when taking the plurality of external environment images; a pallet posture detection module, configured to determine the pallet posture information of the target pallet based on the pseudo point cloud data, the target pallet position in the plurality of external environment images, and the camera posture information; a forklift posture detection module, configured to calculate the forklift posture information of the target forklift based on the camera posture information and the relative posture information; The docking control module is used to control the target forklift to perform forklift-pallet docking according to the forklift posture information of the target forklift and the pallet posture information of the target pallet.

[0012] A third aspect of the present invention provides a forklift pallet docking device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor calls the instructions in the memory to cause the forklift pallet docking device to execute the steps of the forklift pallet docking method described above.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored therein, which, when executed on a computer, causes the computer to execute the steps of the forklift pallet docking method described above.

[0014] The above-mentioned forklift pallet docking method, device, equipment and storage medium obtain the internal and external parameters of the camera and its relative position information with the forklift by installing a camera device on the forklift and calibrating it. After receiving the pallet docking instruction, the forklift is controlled to move to the specified position, and the camera is used to capture an image of the external environment. The position of the target pallet in the image is determined by visual recognition technology, and the image is deeply annotated to generate pseudo point cloud data, and then the pallet's position information is calculated. In addition, the camera pose information and the pallet pose information are combined to accurately calculate the forklift's position information, thereby guiding the forklift to automatically complete the pallet docking task. This method realizes the automatic docking of the forklift to the pallet through integrated camera equipment and advanced computing and processing technology, thereby improving the efficiency and accuracy of logistics operations.

[0015] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of an embodiment of a forklift pallet docking method according to an embodiment of the present invention; Figure 2 A schematic diagram of an embodiment of a forklift pallet docking device according to an embodiment of the present invention; Figure 3 Schematic diagram of an embodiment of a forklift pallet docking device in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] The terms "including," "having," and any variations thereof, as used in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or device.

[0020] To facilitate understanding of this embodiment, a forklift pallet docking method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, this method includes the following steps: 101. Install a camera device on a target forklift, and calibrate the camera device and the target forklift to obtain internal and external parameters of the camera device, as well as relative pose information between the camera device and the target forklift; In one embodiment of the present invention, implementing a forklift-pallet docking method requires first installing a camera on the target forklift. This step forms the foundation of the entire automated docking system. Installing the camera typically involves selecting an appropriate mounting position and angle to ensure the camera covers key viewing angles during the forklift docking operation. For example, the camera can be mounted on the top or front of the forklift to clearly capture images of the pallet and its surroundings. After installation, the next key step is to calibrate the camera and the target forklift. This process aims to obtain the camera's intrinsic and extrinsic parameters, as well as the relative pose information between the camera and the target forklift. The camera's intrinsic parameters, including focal length, principal point coordinates, and distortion coefficients, are crucial for subsequent image processing and pose estimation, as they determine how the camera maps the three-dimensional world onto the two-dimensional image plane. Camera calibration is typically performed by capturing images of a calibration plate of known size and pattern. By analyzing images of the calibration plate in different poses, computer vision algorithms are used to calculate the camera's intrinsic parameters. The camera's extrinsic parameters and the relative pose information between the camera and the forklift describe the camera's position and orientation in the forklift coordinate system. Obtaining this information typically requires a specific calibration process, such as using feature point matching and 3D reconstruction techniques to determine the camera's precise position and pose relative to the forklift. This process might involve attaching several markers of known locations to the forklift, capturing these markers with the camera, and finally analyzing the positions of the markers in the image to calculate the camera's extrinsic parameters and pose information.

[0021] 102. When a forklift-pallet docking instruction is received, control the target forklift to move to the target position indicated by the forklift-pallet docking instruction; In one embodiment of the present invention, when a forklift receives a pallet docking command, the entire process involves complex control logic and precise motion control to ensure that the target forklift accurately moves to the designated pallet docking location. This process relies primarily on an effective communication mechanism through which the forklift receives the docking command from a central control system or operator. Upon receiving the docking command, the system immediately initiates a series of predetermined operational procedures, the most critical of which is parsing the target location information contained in the command. This information is typically provided in the form of coordinates within the forklift's environment, indicating the exact location the forklift needs to reach.

[0022] Next, the system needs to assess the current position of the target forklift, including its current location, orientation, and relative distance and direction to the target location. This assessment relies on the forklift's onboard positioning and navigation systems, such as GPS or vision-based positioning systems, which provide precise information about the forklift's current location. After acquiring this basic information, the control system begins calculating the optimal path from the current position to the target location. This calculation process takes into account various factors, such as obstacles on the path, the maximum allowable speed, and the shortest possible path, to determine a safe and efficient route.

[0023] After calculating the optimal path, the forklift's control system begins executing the movement instructions, fine-tuning the forklift's drive and steering mechanisms to move along the predetermined path toward the target location. During this process, the forklift's sensor system continuously monitors the surrounding environment and the forklift's real-time status, ensuring that the forklift can safely avoid obstacles while adjusting its speed and direction to suit the actual driving conditions. For example, if an unexpected obstacle is encountered during movement, the forklift can respond immediately, adjusting its path to bypass the obstacle or, if necessary, stopping to await further instructions.

[0024] Throughout the entire movement process, the forklift's control system and algorithms play a crucial role. They not only need to process large amounts of sensor data in real time but also make quick and accurate decisions based on this data. Every forklift movement is the result of real-time feedback and adjustments to its control system and algorithms, ensuring the forklift reaches its target location smoothly and accurately. Once the forklift reaches its target location, the system executes the next step, pallet docking, marking the successful execution of the forklift's docking command.

[0025] 103. Shooting the external environment with a camera device to obtain multiple external environment images, performing visual recognition on the multiple external environment images, and determining the target pallet position in the multiple external environment images; In one embodiment of the present invention, photographing the external environment by the camera device to obtain a plurality of external environment images, and performing visual recognition on the plurality of external environment images to determine the target pallet position in the plurality of external environment images includes: photographing the external environment by the camera device during the process of the target forklift moving to the target position to obtain a plurality of external environment images; performing image denoising and brightness adjustment on the plurality of external environment images to obtain a corresponding plurality of pre-processed images; performing visual recognition on the plurality of pre-processed images to identify the target pallet in the plurality of pre-processed images and specific markers around the target pallet; determining the rough position of the target pallet based on the specific markers around the target pallet, and performing image segmentation on the plurality of pre-processed images to determine the pallet boundary of the target pallet; and calculating the target pallet position of the target pallet in the plurality of external environment images based on the pallet boundary and the rough position.

[0026] Specifically, in the forklift-pallet docking method, the process of using a camera to capture the external environment and determine the target pallet's location through visual recognition technology is a complex system that integrates multiple computer vision technologies. First, as the forklift moves toward the target location, the camera continuously captures images of the environment. These images contain rich environmental information, but are also accompanied by noise and lighting variations. To improve recognition accuracy, these raw images must first be preprocessed, including denoising and brightness adjustment. Denoising typically uses methods such as Gaussian blurring and median filtering to remove random noise from the image. For example, applying a Gaussian filter can smooth each pixel based on the weighted average of its neighboring pixels, thereby reducing random noise in the image. Brightness adjustment calculates the image's histogram to analyze the overall brightness distribution, and then uses histogram equalization or other brightness adjustment algorithms to improve the image's contrast and brightness levels. This step ensures clearer identification and segmentation of the target and background in the image. After preprocessing, the image enters the visual recognition stage, where deep learning models such as convolutional neural networks (CNNs) are used to extract features and detect objects. Through the features learned through training, the network is able to identify the target pallet in the image and the specific landmarks around it. Next, based on the results of model recognition, the system further refines the determination of the pallet position. After determining the rough position of the pallet with the assistance of landmarks, the system will cut the image and retain only the image area containing the pallet to reduce the computational burden of subsequent processing. Image cutting can be simplified to selecting the minimum rectangular area surrounding the target pallet, and then applying further image processing techniques, such as edge detection, to this area to accurately determine the boundary of the pallet. Finally, based on the determined pallet boundary, combined with the internal and external parameters of the camera, the exact position of the pallet in three-dimensional space is calculated through geometric transformation. This step may involve perspective transformation, which converts two-dimensional image coordinates into three-dimensional space coordinates. Specifically, if the intrinsic parameter matrix K of the camera and the pose P of the camera relative to a fixed coordinate system are known, the points on the image can be mapped back to three-dimensional space through inverse perspective transformation. The calculation formula is , in, is a point on the image, is the corresponding point in three-dimensional space.

[0027] 104. Perform depth annotation on the multiple external environment images to obtain depth images corresponding to the multiple external environment images, and generate pseudo point cloud data of the depth images according to the internal parameters and external parameters of the camera device; In one embodiment of the present invention, the depth annotation of the multiple external environment images to obtain depth images corresponding to the multiple external environment images, and generating pseudo point cloud data of the depth images according to the internal parameters and external parameters of the camera device include: depth annotation of the multiple external environment images to obtain initial depth images corresponding to the multiple external environment images; geometrically correcting the initial depth image according to the internal parameters of the camera device, and adjusting the geometrically corrected initial depth image according to the external parameters of the camera device to obtain a depth image; converting the depth information in the depth image to the camera coordinate system according to the internal parameters and external parameters of the camera device to obtain pseudo point cloud data of the depth image.

[0028] Specifically, first, the depth of the captured image of the external environment is annotated. This process usually requires the use of a depth sensor or computer vision technology to estimate the depth information of the image and generate an initial depth image corresponding to each image. These initial depth images provide distance information from the camera lens for each pixel, but this information may be distorted without correction. In order to obtain more accurate depth information, geometric correction is required based on the internal parameters of the camera. The internal parameters of the camera include focal length, optical center, etc. These parameters are crucial to understanding how the camera projects three-dimensional space onto a two-dimensional image plane. The purpose of geometric correction is to correct image distortion caused by the characteristics of the camera lens, including radial distortion and tangential distortion. For example, the following formula can be used to correct each pixel in the depth image: ; in, is the position of the original pixel, d is the distortion coefficient, is the inverse of the camera intrinsic parameter matrix, is the corrected pixel position. After completing the geometric correction, the depth image needs to be adjusted according to the external parameters of the camera. The external parameters of the camera describe the position and orientation of the camera in the global coordinate system, which is crucial for mapping the points in the image to the global coordinate system. By using external parameters, the corrected depth image can be converted from the camera coordinate system to the world coordinate system to obtain more consistent and accurate depth information. Finally, based on the depth information in the depth image and the internal and external parameters of the camera, pseudo point cloud data is generated. Pseudo point cloud data is a set of three-dimensional coordinates of each pixel in the image, which provides three-dimensional geometric information of the scene. The generation of point cloud can be achieved by the following formula: ; Where X is the 3D point in the world coordinate system, and Z is the depth value obtained from the depth image. is the inverse of the camera intrinsic parameter matrix, is the point in the image coordinate system, and C is the position of the camera in the world coordinate system.

[0029] 105. Input the plurality of external environment images into a preset posture detection model to obtain camera posture information of the camera device when taking the plurality of external environment images; In one embodiment of the present invention, the pose detection model includes an optical flow estimation network, a long short-term memory network and a fully connected layer; the inputting of the multiple external environment images into the preset pose detection model to obtain the camera pose information of the camera device when shooting the multiple external environment images includes: generating adjacent image pairs according to the two-by-two adjacent principle from the multiple external environment images, and inputting the adjacent image pairs into the preset pose detection model; extracting the optical flow information between the adjacent image pairs through the optical flow estimation network, and extracting the motion features of the multiple external environment images based on the optical flow information; serializing the motion features to obtain an input sequence, and inputting the input sequence into the long short-term memory network to obtain a time series feature; performing mapping processing on the time series feature by the fully connected layer to obtain the camera pose information corresponding to each continuous external environment image.

[0030] Specifically, in implementing the forklift pallet docking method, the core function of the pose detection model is to accurately extract the pose information of the camera device from a series of external environment images. This process begins by organizing multiple continuously captured external environment images into a series of image pairs based on the principle of two-by-two proximity, which serve as the basic input for subsequent processing. The purpose of this step is to provide the optical flow estimation network with a series of image data with a time series relationship, so that the network can capture the motion changes between images. The image pairs are input to the optical flow estimation network, which is specially designed to analyze the pixel movement between the image pairs to estimate the motion vector of each pixel, that is, the optical flow information. In this step, the optical flow estimation network processes the input image pairs through a series of convolutional layers within the deep learning model, and outputs a motion vector field, which describes the displacement direction and distance of each pixel between the first image and the second image. The calculation of optical flow information can be completed by solving the problem of minimizing the energy function, where the energy function measures the difference between the image reconstruction based on the optical flow estimation and the actual second image. Formulaically, it can be expressed as: ; Among them, E(f) represents the energy function, p represents the pixel in the image, and are the first and second images in the image pair respectively. E(f) represents the energy function, which is a commonly used concept in optical flow estimation. The energy function is used to measure the difference or error between the image estimated based on the optical flow field f and the actual image. In the optical flow estimation problem, the goal is to find an optical flow field f such that the first image After the displacement transformation indicated by f, the second image The difference between them is minimal.

[0031] Specifically, the energy function E(f) is an objective function in an optimization problem, and the optimal optical flow field is usually solved by minimizing this function. is to calculate the difference between two images, where represents the square error, p represents each pixel in the image, is the brightness value of the first image at pixel p, is the pixel position of the second image after optical flow f transformation By minimizing the sum of the errors of all pixels in the entire image, we can obtain the optical flow field f that describes the motion of pixels between the two images. This optical flow field can be further used to estimate the motion of the camera or object.

[0032] The motion features extracted by the optical flow estimation network are then serialized and converted into an input sequence, providing data that reflects time series changes to the Long Short-Term Memory (LSTM) network. The LSTM network utilizes its specially designed gating structure to effectively process this time series data. It uses forget gates, input gates, and output gates to control the flow of information, retaining the historical information most useful for predicting the current camera pose and forgetting irrelevant information. After processing all input sequences, the LSTM outputs a series of time series features that capture the camera's motion patterns over time.

[0033] Finally, the temporal features output by the LSTM are fed into a fully connected layer for mapping and conversion into continuous camera pose information. As part of the neural network, the fully connected layer converts the high-dimensional feature vectors of the LSTM layer into specific pose parameters, such as the position and orientation of the camera in three-dimensional space, through a series of weighted summation operations and nonlinear activation functions. This mapping process can be viewed as an optimization problem, where the goal is to find a set of network parameters that makes the pose information output by the network as close as possible to the actual pose information obtained from external sensors or labeled data. For example, if the pose parameters are represented as Euler angles and position vectors, the output of the fully connected layer can be directly mapped to these parameters, thereby achieving a direct mapping from image to camera pose.

[0034] Furthermore, the motion features are serialized to obtain an input sequence, and the input sequence is input into the long short-term memory network to obtain a time series feature, including: serializing the motion features to obtain an input sequence, and inputting the serialized input sequence into the long short-term memory network, performing time dimension integration processing on the input sequence through the long short-term memory network to obtain a feature sequence with a time context; selectively processing the feature sequence through the gating unit in the long short-term memory network, and updating the hidden state in the long short-term memory network to obtain output features of each time step; and performing time aggregation processing on the output features of each time step to obtain a time series feature.

[0035] Specifically, the motion features extracted by the optical flow estimation network need to be converted into a structured sequence format. This process is called serialization. The main purpose of serialization is to reorganize the original motion features into a chronological data sequence that can be effectively processed by the LSTM network.

[0036] After the motion features are serialized, the next step is to feed this serialized input sequence into an LSTM network. An LSTM network is a special type of recurrent neural network (RNN) that uses a gating mechanism to control long-term dependencies and short-term memory of information, effectively solving the vanishing gradient problem encountered by traditional RNNs when processing long sequences of data. Specifically, LSTM uses three types of gating units: input gate, forget gate, and output gate to dynamically regulate the flow of information, enabling efficient learning of time series data.

[0037] When an LSTM processes an input sequence, the motion features at each time step are first evaluated by the input gate to determine what new information should be incorporated into the cell state. Simultaneously, the forget gate determines which old information in the current cell state should be discarded, preventing the accumulation of irrelevant information from interfering with model performance. After processing by these two gates, the cell state is updated, and the output gate ultimately determines which information from the current cell state will constitute the output for that time step. This output feature not only incorporates information from the current moment but also incorporates information from previous time steps, thus providing rich temporal context. After LSTM processing, the output features for each time step are further aggregated to form the final time series feature. This time series feature not only captures the motion information of each frame but also incorporates the dynamic changes of the entire time series, providing critical information for estimating the camera's pose. By inputting these time series features into a fully connected layer for mapping, the camera pose information corresponding to the continuous external environment image is ultimately obtained. This coherent data processing process demonstrates the powerful ability of deep learning technology to solve practical problems and provides a foundation for accurate pose estimation for automated forklift pallet docking operations.

[0038] 106. Determine the pallet pose information of the target pallet based on the pseudo point cloud data, the target pallet position in the plurality of external environment images, and the camera pose information; In one embodiment of the present invention, determining the pallet pose information of the target pallet based on the pseudo point cloud data, the target pallet position in the multiple external environment images and the camera pose information includes: generating a region of interest based on the target pallet position information in the multiple external environment images, and performing point cloud screening on the pseudo point cloud data based on the region of interest to obtain corresponding pseudo point cloud data of interest; clustering the pseudo point cloud data of interest to determine the pallet pseudo point cloud corresponding to the target pallet in the pseudo point cloud data of interest; determining the three-dimensional position and posture of the target pallet in the camera coordinate system based on the pallet pseudo point cloud; converting the three-dimensional position and posture of the target pallet in the camera coordinate system to the world coordinate system through geometric transformation calculation to obtain the absolute pose information of the target pallet in the world coordinate system; calibrating the absolute pose information based on the camera pose information to obtain the pallet pose information of the target pallet.

[0039] Specifically, a region of interest (ROI) is first defined based on the location of the target pallet in the external environment image. This narrows the data processing scope to focus only on image regions that may contain the pallet, thereby improving the efficiency and accuracy of subsequent steps. Next, the collected pseudo point cloud data is filtered based on the defined ROI, retaining only the point cloud data corresponding to the ROI. This data represents the three-dimensional spatial location of the target pallet. This filtering process is achieved by checking whether the coordinates of each point in the point cloud data fall within the boundaries of the ROI, thereby forming a set of filtered point clouds, known as pseudo point clouds of interest. These filtered pseudo point clouds of interest are then clustered. Clustering aims to group points in the point cloud data that belong to the same object. This is particularly important when processing pseudo point clouds, as it helps identify point clouds generated by the target pallet. Clustering algorithms such as K-means or DBSCAN can be used in this step. By calculating the distances between points in the point cloud data, they are divided into multiple groups. Each group represents an independent object in the image. The largest group is generally identified as the pseudo point cloud corresponding to the target pallet. Once the pseudo point cloud of the pallet is obtained, the next step is to determine the target pallet's 3D position and pose in the camera coordinate system based on this point cloud. This typically involves geometric analysis of the point cloud, such as calculating the point cloud's centroid to determine the position and principal component analysis (PCA) to determine the point cloud's principal directions, thereby estimating the pallet's pose. Furthermore, the target pallet is transformed from the camera coordinate system to the world coordinate system through geometric transformation calculations to establish its absolute pose information. This transformation requires the use of the camera's pose information, including the camera's rotation matrix and translation vector, which can be obtained from the camera's extrinsic parameters. The final step is to calibrate the pallet's absolute pose information based on the camera's pose information, ensuring that the calculated pallet pose information is as close to the actual pose as possible. This step may involve further optimization, such as using an optimization algorithm such as the least squares method to adjust the pose parameters to minimize the difference between the predicted pose and the actual pose.

[0040] 107. Calculate the forklift posture information of the target forklift according to the camera posture information and the relative posture information; In one embodiment of the present invention, the calculating of the forklift pose information of the target forklift based on the camera pose information and the relative pose information includes: extracting the position and direction of the camera device at different time points based on the camera pose information to obtain the motion trajectory of the camera device; calculating the relative motion trajectory of the target forklift based on the motion trajectory and the relative pose information between the camera device and the target forklift; smoothing the relative motion trajectory of the target forklift to obtain a smooth motion trajectory; generating a pose sequence of the target forklift based on the smooth motion trajectory, and determining the forklift pose information of the target forklift based on the pose sequence of the target forklift.

[0041] Specifically, in practice, this can be achieved by recording the camera's three-dimensional coordinates (usually expressed in X, Y, and Z axes) and orientation (expressed using Euler angles or quaternions) at each time point, thereby forming a time series dataset that reflects the camera's motion trajectory. Next, the relative pose information between the camera and the target forklift is combined with the camera's motion trajectory to calculate the target forklift's relative motion trajectory. This step requires first determining the relative position and orientation between the camera and forklift, then inferring the forklift's motion from the camera's motion trajectory. This calculation can be achieved using geometric relationships or relative kinematics principles, and may involve complex mathematical transformations, such as using homogeneous coordinate transformation matrices to describe and calculate the relative motion between the two. To make the calculated forklift's motion trajectory smoother, data smoothing is typically required. This process can be achieved using various data smoothing techniques, such as sliding average and Gaussian filtering. The purpose of smoothing is to reduce noise during data acquisition, making the calculated motion trajectory more accurate to the actual situation. This step is critical, as the actual data collected is often affected by various factors, including equipment accuracy and environmental factors, resulting in data fluctuations. Smoothing effectively improves the accuracy and reliability of pose calculations. Finally, a pose sequence of the target forklift can be generated based on the smoothed trajectory. This process involves extracting the forklift's position and orientation at a specific moment from the smoothed trajectory, forming a time sequence where each element represents the forklift's pose at a specific point in time. This process may rely on the mathematical model and transformation matrix defined in the previous steps to ensure accurate extraction of pose information from the relative motion trajectory.

[0042] 108. Control the target forklift to perform forklift-pallet docking according to the forklift posture information of the target forklift and the pallet posture information of the target pallet; In one embodiment of the present invention, first, the posture information (including position and direction) of the forklift is obtained through the data processing method based on camera monitoring discussed previously. Similarly, the acquisition of the pallet posture information also needs to rely on a visual system or other sensor system that can identify the specific position and orientation of the pallet and convert this information into digital data. Once the posture information of the forklift and the pallet is obtained, it is necessary to process and analyze this information in order to develop an effective control strategy to guide the movement of the forklift. For the processing of posture information, it is first necessary to determine the relative position and direction difference between the forklift and the pallet. This step usually involves coordinate transformation and angle calculation, which can be achieved through geometry and linear algebra methods. Assume that the current posture of the forklift is represented by the position vector and orientation angle The position vector of the pallet is represented by and orientation angle Represented, then the relative position vector between the forklift and the pallet can be calculated by the following formula, ; The relative orientation angle difference can be obtained by With the relative position and orientation information between the forklift and the pallet, the next task is to design a control strategy to accurately guide the forklift to the pallet and perform docking. This process requires consideration of the forklift's motion capabilities, including its maximum speed, steering capabilities, and acceleration and deceleration performance. By analyzing the forklift's motion model, a control algorithm can be designed that calculates the forklift's optimal movement path and speed based on the relative position information between the forklift and the pallet. During the forklift's movement, the forklift's current position must be continuously monitored and updated. Simultaneously, the forklift's speed and direction are adjusted based on the real-time relative position information between the forklift and the pallet to ensure that the forklift accurately moves along the predetermined path to the pallet. During this process, feedback control algorithms, such as PID control, may be employed to adjust the forklift's driving state in real time. By adjusting control parameters, the forklift's actual movement trajectory can be kept as close to the predetermined optimal path as possible, thereby improving docking accuracy and efficiency. Finally, when the forklift approaches the pallet and adjusts to the correct docking position and orientation, it performs the final docking operation. This typically involves controlling the lift and retraction of the forklift's fork arm to ensure it fits precisely under the pallet. This step may also require precise control of the forklift's minute displacements and posture adjustments to ensure precise docking between the fork arm and the pallet. By combining various control technologies, such as position control and torque control, smooth and accurate docking between the forklift and the pallet can be achieved.

[0043] In this embodiment, by installing a camera device on a forklift and calibrating it, the internal and external parameters of the camera and its relative position information with the forklift are obtained. After receiving the pallet docking instruction, the forklift is controlled to move to the specified position, and the camera is used to capture an image of the external environment. The position of the target pallet in the image is determined by visual recognition technology, and the image is depth-annotated to generate pseudo-point cloud data, and then the pallet's position information is calculated. In addition, the camera pose information and the pallet pose information are combined to accurately calculate the forklift's position information, thereby guiding the forklift to automatically complete the pallet docking task. This method realizes the automatic docking of the forklift to the pallet through integrated camera equipment and advanced computing and processing technology, thereby improving the efficiency and accuracy of logistics operations.

[0044] The forklift pallet docking method according to the embodiment of the present invention is described above. The forklift pallet docking device according to the embodiment of the present invention is described below. Figure 2 In one embodiment of the present invention, a forklift pallet docking device includes: A calibration module 201 is configured to install a camera device on a target forklift and calibrate the camera device and the target forklift to obtain internal and external parameters of the camera device, as well as relative pose information between the camera device and the target forklift; The movement control module 202 is configured to control the target forklift to move to the target position indicated by the forklift pallet docking instruction when receiving the forklift pallet docking instruction; A visual processing module 203 is configured to capture the external environment using the camera device to obtain a plurality of external environment images, and to perform visual recognition on the plurality of external environment images to determine the position of the target pallet in the plurality of external environment images; a pseudo point cloud generation module 204 for performing depth annotation on the plurality of external environment images to obtain depth images corresponding to the plurality of external environment images, and generating pseudo point cloud data of the depth images according to internal and external parameters of the camera device; A camera pose detection module 205 is configured to input the plurality of external environment images into a preset pose detection model to obtain camera pose information of the camera device when taking the plurality of external environment images; a pallet posture detection module 206 for determining the pallet posture information of the target pallet based on the pseudo point cloud data, the target pallet position in the plurality of external environment images, and the camera posture information; a forklift posture detection module 207 for calculating the forklift posture information of the target forklift based on the camera posture information and the relative posture information; The docking control module 208 is configured to control the target forklift to perform forklift-pallet docking according to the forklift posture information of the target forklift and the pallet posture information of the target pallet.

[0045] In an embodiment of the present invention, the forklift pallet docking device runs the above-mentioned forklift pallet docking method, and the forklift pallet docking device obtains the internal and external parameters of the camera and its relative posture information with the forklift by installing a camera device on the forklift and calibrating it. After receiving the pallet docking instruction, the forklift is controlled to move to the specified position, and the camera is used to capture an image of the external environment. The position of the target pallet in the image is determined by visual recognition technology, and the image is deeply annotated to generate pseudo point cloud data, and then the posture information of the pallet is calculated. In addition, the posture information of the forklift is accurately calculated by combining the camera posture information and the pallet posture information, thereby guiding the forklift to automatically complete the pallet docking task. This method realizes the automatic docking of the forklift to the pallet through integrated camera equipment and advanced computing and processing technology, thereby improving the efficiency and accuracy of logistics operations.

[0046] above Figure 2 The forklift pallet docking device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The forklift pallet docking device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0047] Figure 3 Figure 3 is a schematic diagram of the structure of a forklift-pallet docking device provided in an embodiment of the present invention. The forklift-pallet docking device 300 may vary significantly depending on configuration or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors), memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) storing application programs 333 or data 332. The memory 320 and storage media 330 may be either transient or persistent storage. The program stored in the storage medium 330 may include one or more modules (not shown), each of which may include a series of instructions for operating on the forklift-pallet docking device 300. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, executing the series of instructions stored in the storage medium 330 on the forklift-pallet docking device 300 to implement the steps of the forklift-pallet docking method described above.

[0048] The forklift pallet docking device 300 may further include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 3 The structure of the forklift-pallet docking device shown does not constitute a limitation on the forklift-pallet docking device provided by the present invention, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, cause the computer to execute the steps of the forklift pallet docking method.

[0050] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0051] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0052] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A forklift pallet docking method, characterized in that: The forklift pallet docking method comprises: Installing a camera device on a target forklift, and calibrating the camera device and the target forklift to obtain internal parameters and external parameters of the camera device, and relative position information between the camera device and the target forklift; When a forklift-pallet docking instruction is received, controlling the target forklift to move to a target position indicated by the forklift-pallet docking instruction; Using the camera device to photograph the external environment to obtain a plurality of external environment images, and performing visual recognition on the plurality of external environment images to determine the position of the target pallet in the plurality of external environment images; Performing depth annotation on the multiple external environment images to obtain depth images corresponding to the multiple external environment images, and generating pseudo point cloud data of the depth images according to internal parameters and external parameters of the camera device; Inputting the plurality of external environment images into a preset posture detection model to obtain camera posture information of the camera device when taking the plurality of external environment images; Determine the pallet position information of the target pallet according to the pseudo point cloud data, the target pallet position in the plurality of external environment images and the camera position information; Calculating the forklift posture information of the target forklift according to the camera posture information and the relative posture information; The target forklift is controlled to perform forklift-pallet docking according to the forklift posture information of the target forklift and the pallet posture information of the target pallet.

2. The forklift pallet docking method according to claim 1, characterized in that: The step of photographing the external environment by the camera device to obtain a plurality of external environment images, and performing visual recognition on the plurality of external environment images to determine the target pallet position in the plurality of external environment images includes: During the process of the target forklift moving to the target position, the camera device is used to photograph the external environment to obtain a plurality of external environment images; Performing image denoising and brightness adjustment processing on the plurality of external environment images to obtain a corresponding plurality of pre-processed images; Performing a visual recognition on the plurality of pre-processed images to identify a target tray in the plurality of pre-processed images and specific markers around the target tray; Determine the rough position of the target pallet according to specific markers around the target pallet, and perform image segmentation on the multiple pre-processed images to determine the pallet boundary of the target pallet; The target tray position of the target tray in the plurality of external environment images is calculated according to the tray boundary and the rough position.

3. The forklift pallet docking method according to claim 2, characterized in that: The depth annotation of the plurality of external environment images to obtain depth images corresponding to the plurality of external environment images, and generating pseudo point cloud data of the depth images according to the internal parameters and external parameters of the camera device comprises: Performing depth annotation on the multiple external environment images to obtain initial depth images corresponding to the multiple external environment images; According to the internal parameters of the camera device, the initial depth image is geometrically corrected, and the initial depth image after the geometric correction is adjusted according to the external parameters of the camera device to obtain a depth image; The depth information in the depth image is converted into a camera coordinate system according to the internal parameters and external parameters of the camera device to obtain pseudo point cloud data of the depth image.

4. The forklift pallet docking method according to claim 3, characterized in that: The pose detection model includes an optical flow estimation network, a long short-term memory network and a fully connected layer; The step of inputting the plurality of external environment images into a preset posture detection model to obtain the camera posture information of the camera device when shooting the plurality of external environment images includes: Generate adjacent image pairs from the plurality of external environment images according to a two-by-two adjacency principle, and input the adjacent image pairs into a preset posture detection model; Extracting optical flow information between the adjacent image pairs through the optical flow estimation network, and extracting motion features of the multiple external environment images based on the optical flow information; Serializing the motion features to obtain an input sequence, and inputting the input sequence into the long short-term memory network to obtain a time series feature; The time series features are mapped in a fully connected layer to obtain camera pose information corresponding to each continuous external environment image.

5. The forklift pallet docking method according to claim 4, characterized in that: The step of serializing the motion features to obtain an input sequence, and inputting the input sequence into the long short-term memory network to obtain the time series features includes: The motion features are serialized to obtain an input sequence, and the serialized input sequence is input into a long short-term memory network, and the input sequence is integrated in the time dimension by the long short-term memory network to obtain a feature sequence with a time context; The feature sequence is selectively processed by the gating unit in the long short-term memory network, and the hidden state in the long short-term memory network is updated to obtain the output feature of each time step; The output features of each time step are subjected to time aggregation processing to obtain time series features.

6. The forklift pallet docking method according to claim 5, characterized in that: Determining the pallet position information of the target pallet according to the pseudo point cloud data, the target pallet position in the plurality of external environment images, and the camera position information includes: Generate an area of ​​interest according to the target pallet position information in the plurality of external environment images, and perform point cloud screening on the pseudo point cloud data according to the area of ​​interest to obtain corresponding pseudo point cloud data of interest; Performing clustering processing on the pseudo point cloud data of interest to determine a pallet pseudo point cloud corresponding to the target pallet in the pseudo point cloud data of interest; Determine the three-dimensional position and posture of the target pallet in the camera coordinate system according to the pallet pseudo point cloud; The three-dimensional position and posture of the target pallet in the camera coordinate system are converted into the world coordinate system by geometric transformation calculation to obtain the absolute position and posture information of the target pallet in the world coordinate system; The absolute posture information is calibrated according to the camera posture information to obtain the pallet posture information of the target pallet.

7. The forklift pallet docking method according to claim 1, characterized in that: The calculating the forklift posture information of the target forklift according to the camera posture information and the relative posture information includes: According to the camera posture information, extract the position and direction of the camera device at different time points to obtain the motion trajectory of the camera device; Calculating the relative motion trajectory of the target forklift according to the motion trajectory and the relative position information between the camera device and the target forklift; Smoothing the relative motion trajectory of the target forklift to obtain a smooth motion trajectory; A posture sequence of the target forklift is generated according to the smooth motion trajectory, and forklift posture information of the target forklift is determined according to the posture sequence of the target forklift.

8. A forklift pallet docking device, characterized in that: The forklift pallet docking device comprises: A calibration module, used to install a camera device on a target forklift, and calibrate the camera device and the target forklift to obtain internal parameters and external parameters of the camera device, and relative position information between the camera device and the target forklift; A movement control module, for controlling the target forklift to move to a target position indicated by the forklift pallet docking instruction when receiving the forklift pallet docking instruction; A visual processing module, used to photograph the external environment through the camera device to obtain a plurality of external environment images, and to perform visual recognition on the plurality of external environment images to determine the position of the target pallet in the plurality of external environment images; a pseudo point cloud generation module, configured to perform depth annotation on the plurality of external environment images, obtain depth images corresponding to the plurality of external environment images, and generate pseudo point cloud data of the depth images according to internal and external parameters of the camera device; A camera posture detection module, used to input the multiple external environment images into a preset posture detection model to obtain camera posture information of the camera device when shooting the multiple external environment images; A pallet posture detection module, used to determine the pallet posture information of the target pallet according to the pseudo point cloud data, the target pallet position in the multiple external environment images and the camera posture information; A forklift posture detection module, used to calculate the forklift posture information of the target forklift according to the camera posture information and the relative posture information; The docking control module is used to control the target forklift to perform forklift-pallet docking according to the forklift posture information of the target forklift and the pallet posture information of the target pallet.

9. A forklift pallet docking device, characterized in that: The forklift pallet docking device comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the forklift pallet docking device to perform the steps of the forklift pallet docking method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the forklift pallet docking method as described in any one of claims 1-7 are implemented.

Images