An unmanned forklift path planning method and system based on dynamic prediction

Abstract

The application discloses an unmanned forklift path planning method and system based on dynamic prediction, which comprises the following steps: the unmanned forklift is equipped with a millimeter wave radar, a laser radar and an infrared camera, millimeter wave radar images, laser point cloud images and infrared images are acquired, and the millimeter wave radar images, the laser point cloud images and the infrared images are evenly divided into multiple visual field regions, and the millimeter wave radar image visual field regions are divided into dynamic visual field regions and static visual field regions; the visual field regions matched with the static visual field regions in the millimeter wave radar image are found in the laser point cloud image and the infrared image, and the position of the unmanned forklift is calculated according to the position of the laser radar and the position of the camera; a motion compensation planning algorithm is designed to obtain an optimal path. The scheme integrates a millimeter wave radar, a laser radar and an infrared camera multi-modal sensor, constructs a complementary perception system, designs a motion compensation planning algorithm, accurately adapts to the scene requirements of a metal factory, and further improves the positioning accuracy and the path planning reliability.

Description

Technical Field

[0001] This invention belongs to the field of unmanned forklift positioning and path planning technology, and particularly relates to an unmanned forklift path planning method and system based on dynamic prediction. Background Technology

[0002] In the process of intelligent upgrading of metal factories, unmanned forklifts, as core equipment for material handling, directly impact production efficiency and operational safety with their positioning accuracy and path planning reliability. Metal factory scenarios exhibit significant uniqueness and complexity: the presence of numerous highly reflective obstacles such as metal components and steel stacks within the factory area can severely interfere with sensor signals; harsh environmental factors such as high temperatures, dust, and welding sparks further exacerbate the difficulty of environmental perception; simultaneously, scenarios involving crane lifting, worker movement, and the dynamic movement of other transfer equipment make the operating environment highly dynamic. Current mainstream unmanned forklift path planning and positioning solutions are mostly designed based on general warehousing or ordinary workshop scenarios and have not been adapted to the specific interference problems of metal factories.

[0003] In existing technologies, single-lidar positioning schemes are susceptible to strong reflections from metal components, leading to problems such as increased point cloud noise and feature matching failure, resulting in positioning drift. Schemes relying on visual sensors are prone to image blurring and target recognition failure in scenarios with dust obstruction or sudden changes in lighting. Traditional path planning algorithms are mostly based on static environment assumptions and employ passive obstacle avoidance modes, which are prone to collision risks due to untimely path replanning responses in the dynamic and frequently disturbed environment of metal factories. Furthermore, metal factories are characterized by dense material stacking, narrow passageways, and dynamically changing layouts; traditional planning algorithms, with their simplistic evaluation functions, struggle to balance path length, passage efficiency, and obstacle avoidance safety. Therefore, a novel path planning method for unmanned forklifts is urgently needed. Summary of the Invention

[0004] This invention provides a method and system for unmanned forklift path planning based on dynamic prediction, in order to solve the technical problems of low positioning accuracy, lack of dynamic detection, and single evaluation function of unmanned forklifts.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, the present invention provides a method for unmanned forklift path planning based on dynamic prediction, which includes: S1: The unmanned forklift is equipped with millimeter-wave radar, lidar and infrared camera to acquire millimeter-wave radar image, lidar point cloud image and infrared image, and divide the millimeter-wave radar image, lidar point cloud image and infrared image into multiple field of view areas evenly; S2: Based on the distance values ​​of pixels in each field of view region of the millimeter-wave radar image, obtain the field of view region to be divided, and then based on the radial velocity values ​​of each pixel in the field of view region to be divided, divide the multiple field of view regions to be divided into dynamic field of view regions and static field of view regions. S3: Find the field of view area that matches the dynamic and static field of view areas in the millimeter-wave radar image in the laser point cloud image and infrared image. Design a marker feature extraction algorithm for the laser point cloud image to calculate the laser radar position. Design a hotspot feature extraction algorithm for the infrared image to calculate the camera position. Calculate the unmanned forklift position from the laser radar position and the camera position. S4: Based on the position of the unmanned forklift, motion compensation is designed. The planning algorithm calculates the motion factor from the Manhattan distance between the unmanned forklift's location and the target location, then calculates the evaluation function from the motion factor to obtain the optimal path. S5: Outputs the location and optimal path of the unmanned forklift.

[0006] Preferably, step S1 includes: The unmanned forklift is equipped with millimeter-wave radar, lidar and infrared camera, and the millimeter-wave radar, lidar and infrared camera have the same field of view; The environmental information collected by millimeter-wave radar includes the distance and radial velocity of the unmanned forklift to the target point; each laser point collected by lidar includes the distance and reflection intensity of the unmanned forklift to the target point. Infrared cameras capture infrared images, and the environmental information in these images includes grayscale values ​​and temperature values. The environmental information collected by millimeter-wave radar and lidar is preprocessed, including: using a Gaussian filtering algorithm to remove noise points from the environmental information collected by millimeter-wave radar, and projecting the denoised environmental information to obtain a millimeter-wave radar map; using a mean filtering algorithm to remove noise points from the environmental information collected by lidar, and projecting the denoised environmental information to obtain a lidar point cloud map. Divide the millimeter-wave radar image evenly into A field of view region; the laser point cloud map is uniformly divided into... Each field of view region; the infrared image is evenly divided into... One field of view area; The millimeter-wave radar, lidar, and infrared camera have the same field of view. Each field of view in the millimeter-wave radar image corresponds one-to-one with each field of view in the lidar point cloud image and each field of view in the infrared image, and the field of view is consistent. The field of view in the millimeter-wave radar image is matched and numbered with each field of view in the lidar point cloud image and each field of view in the infrared image.

[0007] Preferably, step S2 includes: (1) In the millimeter-wave radar image, each pixel contains the distance and radial velocity from the unmanned forklift to the target point; calculate the mean distance of each field of view in the millimeter-wave radar image. The average distance of the i-th visual region The formula is as follows: ; Where j represents the index of the pixel within the field of view. This represents the number of pixels within the field of view, where i represents the index of the field of view. This represents the distance to the j-th pixel within the field of view; (2) Initialize the distance threshold and If the average distance of the field of view is greater than the distance threshold And less than or equal to the distance threshold If so, the field of view area is marked as a mid-range area; this mid-range area is the field of view area to be divided. (3) Extract the radial velocity of each pixel within the field of view to be divided, and preset the effective range of radial velocity [ , ], The radial velocity values ​​of pixels within each region are filtered to remove abnormal pixels that are outside the valid range. The effective pixels after filtering are statistically analyzed, and the number of effective radial velocity points and velocity distribution characteristics of each field of view to be divided are recorded. The velocity distribution characteristics include the maximum radial velocity and the average radial velocity. (4) Calculate the region motion coefficient using the radial velocity values ​​of each pixel in the region to be divided, the number of effective radial velocity points, and the velocity distribution characteristics. The region motion coefficient of the i-th region to be divided is... The specific formula is as follows: ; in, Indicates the number of effective velocity points. This represents the average velocity within the i-th visual field. This represents the maximum velocity within the i-th visual region, and || denotes the absolute value calculation. This represents the radial velocity of the j-th pixel in the i-th region of the field of view to be segmented; Initialize dynamic threshold If the region motion coefficient of the i-th field of view to be divided is less than or equal to the dynamic threshold, then the field of view to be divided is a static field of view; if the region motion coefficient of the i-th field of view to be divided is greater than the dynamic threshold, then the field of view to be divided is a dynamic field of view; and all field of view to be divided are divided into dynamic and static field of view.

[0008] Preferably, in step S3, a marker feature extraction algorithm is designed for the laser point cloud map to calculate the location of the lidar. The specific steps are as follows: (1) Initialize the feature radius and feature coefficient, calculate the number of pixels within the feature radius of each pixel in the field of view area that matches the static field of view area in the laser point cloud map and the millimeter-wave radar map. If the number of pixels within the feature radius of a certain pixel is greater than the feature coefficient, then randomly select a certain number of pixels to calculate the feature value of the pixel. If the number of pixels within the feature radius of a certain pixel is less than or equal to the feature coefficient, then select all pixels within the feature radius to calculate the feature value of the pixel. The first matching region in the laser point cloud map with the static field of view in the millimeter-wave radar map. Feature value of the j-th pixel in the field of view The specific calculation formula is as follows: ; Where u represents the pixel index within the feature radius. This represents the number of pixels used to calculate the feature value of the j-th pixel. Indicates the first The distance between the j-th pixel in a visual region and the u-th pixel within the feature radius; (2) If the feature value of a pixel is greater than or equal to the feature threshold, the pixel is an edge point; if the feature value of a pixel is less than the feature threshold, the pixel is a plane point to be determined. (3) For the undetermined plane point, calculate the variance of the pixel point with the pixels directly above, below, to the left and to the right, initialize the variance threshold. If the variance of the undetermined plane point is less than or equal to the variance threshold, the undetermined plane point is a plane point. If the variance of the undetermined plane point is greater than the variance threshold, the undetermined plane point is not a plane point. The matching results were optimized by using the iterative nearest point algorithm for edge points and planar points respectively. The number of iterations was set to 100, and the convergence threshold was set to 0.05m. Finally, the position of the lidar in the factory coordinate system was obtained.

[0009] Preferably, in step S3, a hotspot feature extraction algorithm is designed for the infrared image to calculate the camera position. The specific steps are as follows: A hotspot feature extraction algorithm is designed for the field of view region that matches the static field of view region in the infrared image and the millimeter-wave radar image. SHIFT feature points are extracted based on the temperature value of each pixel in the field of view region of the infrared image, and the SHIFT feature points are used as hotspot features. The iterative nearest-point algorithm is used to obtain the camera position of the infrared camera in the factory coordinate system based on hotspot features; The position of the unmanned forklift is calculated from the position of the LiDAR and the position of the camera, with a weighting coefficient of 0.5 for both the LiDAR and camera positions.

[0010] Preferably, step S4 includes: (1) Obtain the location of the unmanned forklift and the target point, and calculate the estimated cost based on the location of the unmanned forklift and the target point. The specific calculation formula is as follows: ; in, This represents the estimated cost of traveling from the position of the unmanned forklift at time t to the target position. The Manhattan x-coordinate represents the position of the unmanned forklift at time t. The Manhattan y-coordinate represents the position of the unmanned forklift at time t. The Manhattan x-axis coordinates representing the target location. The Manhattan y-axis coordinates representing the target location; (2) The motion factor is calculated based on the Manhattan distance from the unmanned forklift's location to the target location. The specific calculation formula is shown in the following formula: ; Where k represents the number of turns from the starting position of the automated forklift to its position at time t. The weights for near and far distances are represented by the following formula: ; in, The Manhattan x-axis coordinates representing the starting position. The Manhattan y-axis coordinate representing the starting position; (3) The evaluation function q is calculated from the motion factor, and the specific formula is as follows: ; in, This indicates the position of the unmanned forklift at time t. This represents the total cost of the unmanned forklift traveling from its current position to the target position at time t. This represents the actual cost from the starting position of the automated forklift to its position at time t. For motor compensation The planning algorithm iteratively solves for the location to obtain the optimal path.

[0011] On the other hand, the present invention also provides a dynamic prediction-based unmanned forklift path planning system, which includes: The data preprocessing module, equipped with millimeter-wave radar, lidar and infrared camera, preprocesses the environmental information collected by millimeter-wave radar and lidar to obtain millimeter-wave radar map and lidar point cloud map, and evenly divides the millimeter-wave radar map, lidar point cloud map and infrared map collected by infrared camera into multiple field of view areas. The dynamic and static region segmentation module obtains the field of view to be segmented based on the distance values ​​of pixels in each field of view region of the millimeter-wave radar image, and then divides multiple field of view regions into dynamic field of view regions and static field of view regions based on the radial velocity values ​​of each pixel in the field of view regions to be segmented. The unmanned forklift positioning module finds the field of view area that matches the dynamic and static field of view areas in the millimeter-wave radar image in the laser point cloud image and infrared image. It designs a marker feature extraction algorithm for the laser point cloud image to calculate the laser radar position and a hotspot feature extraction algorithm for the infrared image to calculate the camera position. The unmanned forklift position is calculated from the laser radar position and the camera position. The path planning module, based on the position of the unmanned forklift, designs motion compensation. The planning algorithm calculates the motion factor from the Manhattan distance between the unmanned forklift's location and the target location, calculates the evaluation function from the motion factor, obtains the optimal path, and outputs the unmanned forklift's location and the optimal path.

[0012] The beneficial effects of the technical solution provided by this invention include at least the following: 1. Multi-sensor fusion sensing adapts to the highly interference-prone environment of metal factories, enabling accurate prediction of dynamic areas. This invention addresses the characteristics of metal factories—high reflectivity, high dust levels, and high dynamism—by integrating millimeter-wave radar, lidar, and infrared cameras into an unmanned forklift to construct a complementary sensing system. Combined with a targeted environmental information preprocessing workflow, millimeter-wave radar maps, lidar point cloud maps, and infrared maps are generated, effectively overcoming the limitations of single-sensor applications in metal factories. Millimeter-wave radar possesses dust and strong reflection resistance characteristics, accurately capturing target velocity information; lidar provides high-precision spatial geometric features; and infrared cameras can penetrate some dust interference, identifying high-temperature components and personnel targets through temperature differences. Based on the distance values ​​from the millimeter-wave radar map, preliminary area division is completed. Combined with pixel velocity values, motion coefficients are calculated to accurately define dynamic and static field-of-view areas. Compared to traditional sensing solutions, this significantly improves the reliability of sensing and the accuracy of dynamic target recognition in the complex environment of metal factories, providing accurate environmental prior data for subsequent positioning and path planning, and avoiding sensing failures caused by environmental interference from the outset.

[0013] 2. Combining cross-sensor region matching and distinctive feature extraction for precise positioning in metal factories. This invention innovatively designs a cross-sensor region matching mechanism, accurately matching the dynamic and static field-of-view areas defined by millimeter-wave radar images in laser point cloud maps and infrared images, ensuring spatial consistency of multi-source data. Addressing the strong reflection interference problem of laser point clouds in metal factories, a dedicated marker feature extraction algorithm is designed to accurately calculate the lidar position by filtering point cloud noise. For dust and light interference in the factory area, a hotspot feature extraction algorithm is designed, utilizing high-temperature features captured by infrared cameras as positioning auxiliary markers to accurately calculate the camera position. Finally, the lidar and camera position data are fused to obtain the precise positioning result of the unmanned forklift. This solution overcomes the positioning drift bottleneck of traditional positioning technologies in metal factories, eliminates the need for additional dedicated positioning markers, reduces factory modification and maintenance costs, and balances positioning accuracy and environmental adaptability. It provides a reliable position benchmark for path planning and effectively solves the core problem of inaccurate positioning of unmanned forklifts under complex interference in metal factories.

[0014] 3. Exercise-compensated type This invention adapts to dynamic factory environments, enabling efficient and safe path planning. Based on precise positioning results from metal factories, it designs motion compensation mechanisms. The planning algorithm, combining the Manhattan distance from the unmanned forklift's location to the target location to calculate motion factors, constructs a multi-objective evaluation function, accurately adapting to the needs of metal factories characterized by narrow passages, dense stacking, and frequent dynamic interference. The Manhattan distance precisely quantifies the actual movement cost of the unmanned forklift within the factory's grid-like passageways. The introduction of motion factors allows the evaluation function to fully consider the unmanned forklift's inertia and the dynamic changes in the factory environment, achieving coordinated optimization of path length, passage efficiency, and obstacle avoidance safety. Compared to traditional passive obstacle avoidance algorithms, this solution, based on accurate early dynamic area prediction, can proactively avoid dynamically interfering areas during path search, reducing the frequency of path replanning and ensuring the continuity of material transfer. The motion compensation mechanism effectively improves the path tracking accuracy of the unmanned forklift in narrow passages, avoiding collision risks caused by positioning deviations or inertia. The optimal path obtained through this algorithm can accurately adapt to the metal factory's operational scenarios, significantly improving the unmanned forklift's transfer efficiency and operational safety, and promoting the efficient operation of the metal factory's automated logistics system. Attached Figure Description

[0015] Figure 1 A schematic diagram of the overall execution flow of an unmanned forklift path planning method based on dynamic prediction provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the dynamic and static field of view division of an unmanned forklift path planning method based on dynamic prediction, provided in an embodiment of the present invention. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0017] Example 1 This embodiment provides a method for unmanned forklift path planning based on dynamic prediction, which can be implemented by electronic devices, such as... Figure 1 As shown. Specifically, the method in this embodiment includes the following steps: The unmanned forklift is equipped with millimeter-wave radar, lidar and infrared camera, and the millimeter-wave radar, lidar and infrared camera have the same field of view. It should be further noted that in this solution, the position calculated by lidar and infrared camera is the position of unmanned forklift by default. The millimeter-wave radar uses 4D millimeter-wave radar. The environmental information collected by the millimeter-wave radar includes the distance and radial velocity from the unmanned forklift to the target point, with a horizontal resolution of 1° and a vertical resolution of 2°. The lidar uses a 3D lidar with 16 beams and a horizontal resolution of 0.2°. Each laser point collected includes the distance from the unmanned forklift to the target point and the reflection intensity. Infrared cameras collect invisible infrared light from the environment, and infrared images are captured. The environmental information in the infrared images includes grayscale values ​​and temperature values. The environmental information collected by millimeter-wave radar and lidar is preprocessed, including: using a Gaussian filtering algorithm to remove noise points from the environmental information collected by millimeter-wave radar, and projecting the denoised environmental information to obtain a millimeter-wave radar map; using a mean filtering algorithm to remove noise points from the environmental information collected by lidar, and projecting the denoised environmental information to obtain a lidar point cloud map; through the preprocessed sensor information, the interference of noise points on subsequent positioning can be removed, improving positioning accuracy and laying a good foundation for subsequent path planning, thereby improving the accuracy of path planning; The resolution of the millimeter-wave radar image is The millimeter-wave radar image is evenly divided into The resolution of the laser point cloud map is [resolution] for each field of view region. The laser point cloud map is uniformly divided into... The infrared image obtained by the infrared camera has a resolution of [missing information] within a field of view. The infrared image is evenly divided into Regarding the field of view, it should be further explained that dividing the image into regions can effectively improve the processing efficiency of the entire frame and reduce computational redundancy. Since millimeter-wave radar, lidar, and infrared cameras share the same field of view, each field of view in the millimeter-wave radar image corresponds one-to-one with each field of view in the lidar point cloud image and the infrared image, and their perspectives are identical. Therefore, each field of view in the millimeter-wave radar image is matched and numbered with each field of view in the lidar point cloud image and the infrared image. The numbering follows a top-to-bottom and left-to-right principle.

[0018] After obtaining millimeter-wave radar images, laser point cloud images, and infrared images, the characteristics of millimeter-wave radar can be used to divide the field of view into dynamic and static areas. In actual metal factory scenarios, unmanned forklifts will always encounter dynamic objects when moving. These dynamic objects will interfere with the positioning of the unmanned forklifts. Therefore, it is necessary to remove the interference of dynamic objects in a timely manner to improve the positioning accuracy of unmanned forklifts. In addition, these dynamic interferences not only reduce positioning accuracy but also waste the computing resources of the unmanned forklift control system. Therefore, it is even more necessary to process dynamic objects.

[0019] Regions are divided based on the distance values ​​of pixels in each field of view of the millimeter-wave radar image to obtain the field of view to be divided. Then, based on the radial velocity values ​​of each pixel in the field of view to be divided, multiple field of view regions to be divided are further divided into dynamic field of view regions and static field of view regions. The specific steps are as follows: (1) In the millimeter-wave radar image, each pixel contains the distance and radial velocity from the unmanned forklift to the target point; calculate the mean distance of each field of view in the millimeter-wave radar image. The average distance of the i-th visual region The formula is as follows: ; Where j represents the index of the pixel within the field of view. This represents the number of pixels within the field of view, where i represents the index of the field of view. This represents the distance to the j-th pixel within the field of view; (2) Initialize the distance threshold and If the mean distance of the visual field is less than or equal to the distance threshold If the mean distance of the visual field is greater than a distance threshold, then the visual field region is marked as a near-field region. And less than or equal to the distance threshold If the average distance to this region is greater than or equal to the distance threshold, then this region is marked as a medium-distance region. Then the area of ​​view is marked as a distant area, for example. The value can be 0.5. The value can be 10, or it can be modified to match the specific environment of the factory. For the three types of distance areas mentioned above, the near distance area and the far distance area may affect path planning and dynamic and static area judgment due to being too close or too far. Therefore, only the medium distance area is divided into dynamic and static visual areas, that is, the medium distance area is the visual area to be divided. (3) Extract the radial velocity of each pixel within the field of view to be divided, and preset the effective range of radial velocity [ , ],default The radial velocity values ​​of pixels within each region are filtered to remove abnormal pixels that are outside the valid range. The effective pixels after filtering are statistically analyzed, and the number of effective radial velocity points and velocity distribution characteristics of each field of view to be divided are recorded. The velocity distribution characteristics include the maximum radial velocity and the average radial velocity. (4) Calculate the region motion coefficient using the radial velocity values ​​of each pixel in the region to be divided, the number of effective radial velocity points, and the velocity distribution characteristics. The region motion coefficient of the i-th region to be divided is... The specific formula is as follows: ; in, Indicates the number of effective velocity points. This represents the average velocity within the i-th visual field. This represents the maximum velocity within the i-th visual region, and || denotes the absolute value calculation. This represents the radial velocity of the j-th pixel in the i-th region of the field of view to be segmented; Initialize dynamic threshold If the motion coefficient of the i-th visual field region to be segmented is less than or equal to the dynamic threshold, then the visual field region to be segmented is a static visual field region. If the motion coefficient of the i-th visual field region to be segmented is greater than the dynamic threshold, then the visual field region to be segmented is a dynamic visual field region. For example, when the dynamic threshold is 0.5 and the motion coefficient is 0.8, the visual field region is a dynamic visual field region because the motion coefficient is greater than the dynamic threshold. Figure 2 As shown.

[0020] After obtaining the dynamic and static field of view regions, it is necessary to find the field of view regions that match the dynamic and static field of view regions in the millimeter-wave radar image in the laser point cloud image, and to find the field of view regions that match the dynamic and static field of view regions in the millimeter-wave radar image in the infrared image.

[0021] An algorithm for extracting landmark features from laser point cloud images and calculating the location of the lidar is designed. The specific steps are as follows: (1) Initialize the feature radius and feature coefficient, calculate the number of pixels within the feature radius of each pixel in the field of view area that matches the static field of view area in the laser point cloud map and the millimeter-wave radar map. If the number of pixels within the feature radius of a certain pixel is greater than the feature coefficient, then randomly select a certain number of pixels to calculate the feature value of the pixel. If the number of pixels within the feature radius of a certain pixel is less than or equal to the feature coefficient, then select all pixels within the feature radius to calculate the feature value of the pixel. The feature radius is a pre-set integer. The number of pixels within the circle with the integer as the radius is calculated. The number of pixels within the feature radius is as shown above. The first matching region in the laser point cloud map with the static field of view in the millimeter-wave radar map. Feature value of the j-th pixel in the field of view The specific calculation formula is as follows: ; Where u represents the pixel index within the feature radius. This represents the number of pixels used to calculate the feature value of the j-th pixel. Indicates the first The distance between the j-th pixel in a visual region and the u-th pixel within the feature radius; (2) If the feature value of a pixel is greater than or equal to the feature threshold, the pixel is an edge point; if the feature value of a pixel is less than the feature coefficient, the pixel is a plane point to be determined. For example, the feature threshold can be set to 0.75. (3) For a point to be determined as a plane, calculate the variance of the pixel and the pixels directly above, below, to the left and to the right, and initialize the variance threshold. If the variance of the point to be determined as a plane is less than or equal to the variance threshold, the point to be determined as a plane is a plane. If the variance of the point to be determined as a plane is greater than the variance threshold, the point to be determined as a plane is not a plane. It should be further noted that if the pixel does not have pixels directly above, below, to the left and to the right, then it is also possible to calculate the variance of the pixels directly above, below, to the left or to the right. It is not necessary for all four points to exist. The matching results were optimized using the iterative nearest point algorithm for edge points and planar points respectively. The number of iterations was set to 100, and the convergence threshold was set to 0.05m. Finally, the position of the lidar in the factory coordinate system was obtained. It should be further noted that the Manhattan distance also needs to be calculated using the factory coordinate system. The starting point of the factory coordinate system is manually marked, usually an edge point on the outermost edge of the factory area.

[0022] A hotspot feature extraction algorithm is designed for infrared images to calculate camera positions. The specific steps are as follows: A hotspot feature extraction algorithm is designed for the field of view region that matches the static field of view region in the infrared image and the millimeter-wave radar image. SHIFT feature points are extracted based on the temperature value of each pixel in the field of view region of the infrared image, and the SHIFT feature points are used as hotspot features. It should be further noted that, in order to further improve the positioning accuracy, the extraction of hotspot features can also be performed as follows: (1) Calculate the coordinates of the hotspot center ,in The x-axis coordinates representing the center of the hotspot The y-axis coordinate represents the center of the hotspot; (2) Initialize the temperature threshold, and use the k-means clustering algorithm to cluster the hot spots and surrounding pixels to obtain the number of clustered pixels. ; (3) Calculate the temperature variance in hotspot clusters ; (4) Construct hotspot features F, where, ; Then, the iterative nearest-point algorithm is used to obtain the camera position of the infrared camera in the factory coordinate system based on the hotspot features. The specific operation is as follows: randomly match the hotspot features between adjacent images, calculate the absolute value of the difference between the hotspot center coordinates, the number of clustered pixels and the temperature variance in the hotspot cluster within the hotspot features, and then sum the above absolute values ​​to obtain a random value. Continue to randomly match until the smallest random value is found, or the random value is less than the random threshold, then end the operation. Then use the SVD algorithm to solve for the camera position. The position of the unmanned forklift is calculated from the position of the LiDAR and the position of the camera, with a weighting coefficient of 0.5 for both the LiDAR and camera positions.

[0023] Based on the position of the unmanned forklift, a motion compensation design is implemented. The planning algorithm calculates the motion factor from the Manhattan distance between the unmanned forklift's current position and the target position, and then calculates the evaluation function from the motion factor. The specific steps are as follows: (1) Obtain the location of the unmanned forklift and the target point, and calculate the estimated cost based on the location of the unmanned forklift and the target point. The specific calculation formula is as follows: ; in, This represents the estimated cost of traveling from the position of the unmanned forklift at time t to the target position. The Manhattan x-coordinate represents the position of the unmanned forklift at time t. The Manhattan y-coordinate represents the position of the unmanned forklift at time t. The Manhattan x-axis coordinates representing the target location. The Manhattan y-axis coordinates representing the target location; (2) The motion factor is calculated based on the Manhattan distance from the unmanned forklift's location to the target location. The specific calculation formula is shown in the following formula: ; in, k represents the number of turns from the starting position of the automated forklift to its position at time t. The weights for near and far distances are represented by the following formula: ; in, The Manhattan x-axis coordinates representing the starting position. The Manhattan y-axis coordinate representing the starting position; (3) The evaluation function q is calculated from the motion factor, and the specific formula is as follows: ; in, This indicates the position of the unmanned forklift at time t. This represents the total cost of the unmanned forklift traveling from its current position to the target position at time t. This represents the actual cost from the starting position of the automated forklift to its position at time t. For motor compensation The planning algorithm iteratively solves for the location to obtain the optimal path; Output the location and optimal path of the unmanned forklift.

[0024] Example 2 This embodiment provides an unmanned forklift path planning system based on dynamic prediction, such as... Figure 2 As shown, the unmanned forklift path planning system based on dynamic prediction includes the following modules: The data preprocessing module, equipped with millimeter-wave radar, lidar and infrared camera, preprocesses the environmental information collected by millimeter-wave radar and lidar to obtain millimeter-wave radar map and lidar point cloud map, and evenly divides the millimeter-wave radar map, lidar point cloud map and infrared map collected by infrared camera into multiple field of view areas. The dynamic and static region segmentation module obtains the field of view to be segmented based on the distance values ​​of pixels in each field of view region of the millimeter-wave radar image, and then divides multiple field of view regions into dynamic field of view regions and static field of view regions based on the radial velocity values ​​of each pixel in the field of view regions to be segmented. The unmanned forklift positioning module finds the field of view area that matches the dynamic and static field of view areas in the millimeter-wave radar image in the laser point cloud image and infrared image. It designs a marker feature extraction algorithm for the laser point cloud image to calculate the laser radar position and a hotspot feature extraction algorithm for the infrared image to calculate the camera position. The unmanned forklift position is calculated from the laser radar position and the camera position. The path planning module, based on the position of the unmanned forklift, designs motion compensation. The planning algorithm calculates the motion factor from the Manhattan distance between the unmanned forklift's position and the target position, calculates the evaluation function from the motion factor, obtains the optimal path, and outputs the unmanned forklift's position and the optimal path. As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0025] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0026] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0027] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for unmanned forklift path planning based on dynamic prediction, characterized in that, Includes the following steps: S1: The unmanned forklift is equipped with millimeter-wave radar, lidar and infrared camera to acquire millimeter-wave radar image, lidar point cloud image and infrared image, and divide the millimeter-wave radar image, lidar point cloud image and infrared image into multiple field of view areas evenly; S2: Based on the distance values ​​of pixels in each field of view region of the millimeter-wave radar image, obtain the field of view region to be divided. Then, based on the radial velocity values ​​of each pixel in the field of view region to be divided, divide the multiple field of view regions into dynamic field of view regions and static field of view regions, including: (1) In the millimeter-wave radar image, each pixel contains the distance and radial velocity from the unmanned forklift to the target point; calculate the mean distance of each field of view in the millimeter-wave radar image. The mean distance of the i-th visual region The formula is as follows: ； Where j represents the index of the pixel within the field of view. This represents the number of pixels within the field of view, where i represents the index of the field of view. This represents the distance to the j-th pixel within the field of view; (2) Initialize the distance threshold and If the average distance of the field of view is greater than the distance threshold And less than or equal to the distance threshold If so, the field of view area is marked as a mid-range area; this mid-range area is the field of view area to be divided. (3) Extract the radial velocity of each pixel within the field of view to be divided, and preset the effective range of radial velocity [ , ], The radial velocity values ​​of pixels within each region are filtered to remove abnormal pixels that are outside the valid range. The effective pixels after filtering are statistically analyzed, and the number of effective radial velocity points and velocity distribution characteristics of each field of view to be divided are recorded. The velocity distribution characteristics include the maximum radial velocity and the average radial velocity. (4) Calculate the region motion coefficient using the radial velocity values ​​of each pixel in the region to be divided, the number of effective radial velocity points, and the velocity distribution characteristics. The region motion coefficient of the i-th region to be divided is... The specific formula is as follows: ； in, Indicates the number of effective velocity points. This represents the average velocity within the i-th visual field. This represents the maximum velocity within the i-th visual region, and || denotes the absolute value calculation. This represents the radial velocity of the j-th pixel in the i-th region of the field of view to be segmented; Initialize dynamic threshold If the region motion coefficient of the i-th field of view to be divided is less than or equal to the dynamic threshold, then the field of view to be divided is a static field of view; if the region motion coefficient of the i-th field of view to be divided is greater than the dynamic threshold, then the field of view to be divided is a dynamic field of view; and all field of view to be divided are divided into dynamic and static field of view. S3: Find the field of view area that matches the dynamic and static field of view areas in the millimeter-wave radar image in the laser point cloud image and infrared image. Design a marker feature extraction algorithm for the laser point cloud image to calculate the laser radar position. Design a hotspot feature extraction algorithm for the infrared image to calculate the camera position. Calculate the unmanned forklift position from the laser radar position and the camera position. S4: Based on the position of the unmanned forklift, motion compensation is designed. The planning algorithm calculates the motion factor from the Manhattan distance between the unmanned forklift's location and the target location, then calculates the evaluation function from the motion factor to obtain the optimal path. S5: Output the location and optimal path of the unmanned forklift.

2. The unmanned forklift path planning method based on dynamic prediction according to claim 1, characterized in that, Step S1 includes: The unmanned forklift is equipped with millimeter-wave radar, lidar and infrared camera, and the millimeter-wave radar, lidar and infrared camera have the same field of view; The environmental information collected by millimeter-wave radar includes the distance and radial velocity of the unmanned forklift to the target point; each laser point collected by lidar includes the distance and reflection intensity of the unmanned forklift to the target point. Infrared cameras capture infrared images, and the environmental information in the infrared images includes grayscale values ​​and temperature values. The environmental information collected by millimeter-wave radar and lidar is preprocessed, including: using a Gaussian filtering algorithm to remove noise points from the environmental information collected by millimeter-wave radar, and projecting the denoised environmental information to obtain a millimeter-wave radar map; using a mean filtering algorithm to remove noise points from the environmental information collected by lidar, and projecting the denoised environmental information to obtain a lidar point cloud map. Divide the millimeter-wave radar image evenly into A field of view region; the laser point cloud map is uniformly divided into... Each field of view region; the infrared image is evenly divided into... One field of view area; The millimeter-wave radar, lidar, and infrared camera have the same field of view. Each field of view in the millimeter-wave radar image corresponds one-to-one with each field of view in the lidar point cloud image and each field of view in the infrared image, and the field of view is consistent. The field of view in the millimeter-wave radar image is matched and numbered with each field of view in the lidar point cloud image and each field of view in the infrared image.

3. The unmanned forklift path planning method based on dynamic prediction according to claim 1, characterized in that, In step S3, a marker feature extraction algorithm is designed for the laser point cloud map to calculate the location of the lidar. The specific steps are as follows: (1) Initialize the feature radius and feature coefficient, calculate the number of pixels within the feature radius of each pixel in the field of view area that matches the static field of view area in the laser point cloud map and the millimeter-wave radar map. If the number of pixels within the feature radius of a certain pixel is greater than the feature coefficient, then randomly select a certain number of pixels to calculate the feature value of the pixel. If the number of pixels within the feature radius of a certain pixel is less than or equal to the feature coefficient, then select all pixels within the feature radius to calculate the feature value of the pixel. Calculate the first matching region between the laser point cloud map and the static field of view in the millimeter-wave radar map. Feature value of the j-th pixel in the field of view The specific calculation formula is as follows: ； Where u represents the pixel index within the feature radius. This represents the number of pixels used to calculate the feature value of the j-th pixel. Indicates the first The distance between the j-th pixel in a visual region and the u-th pixel within the feature radius; (2) If the feature value of a pixel is greater than or equal to the feature threshold, the pixel is an edge point; if the feature value of a pixel is less than the feature threshold, the pixel is a plane point to be determined. (3) For the undetermined plane point, calculate the variance of the pixel point with the pixels directly above, below, to the left and to the right, initialize the variance threshold. If the variance of the undetermined plane point is less than or equal to the variance threshold, the undetermined plane point is a plane point. If the variance of the undetermined plane point is greater than the variance threshold, the undetermined plane point is not a plane point. The matching results were optimized by using the iterative nearest point algorithm for edge points and planar points respectively. The number of iterations was set to 100, and the convergence threshold was set to 0.05m. Finally, the position of the lidar in the factory coordinate system was obtained.

4. The unmanned forklift path planning method based on dynamic prediction according to claim 1, characterized in that, In step S3, a hotspot feature extraction algorithm is designed for the infrared image, and the camera position is calculated. The specific steps are as follows: A hotspot feature extraction algorithm is designed for the field of view region that matches the static field of view region in the infrared image and the millimeter-wave radar image. SIFT feature points are extracted based on the temperature value of each pixel in the field of view region of the infrared image, and the SIFT feature points are used as hotspot features. The iterative nearest-point algorithm is used to obtain the camera position of the infrared camera in the factory coordinate system based on hotspot features; The position of the unmanned forklift is calculated from the position of the LiDAR and the position of the camera, with a weighting coefficient of 0.5 for both the LiDAR and camera positions.

5. The unmanned forklift path planning method based on dynamic prediction according to claim 1, characterized in that, Step S4 includes: (1) Obtain the location of the unmanned forklift and the target point, and calculate the estimated cost based on the location of the unmanned forklift and the target point. The specific calculation formula is as follows: ； in, This represents the estimated cost of traveling from the position of the unmanned forklift at time t to the target position. The Manhattan x-coordinate represents the position of the unmanned forklift at time t. The Manhattan y-coordinate represents the position of the unmanned forklift at time t. The Manhattan x-axis coordinates representing the target location. The Manhattan y-coordinate represents the target location; (2) The motion factor is calculated based on the Manhattan distance from the unmanned forklift's location to the target location. The specific calculation formula is shown in the following formula: ； Where k represents the number of turns from the starting position of the automated forklift to its position at time t. The weights for near and far distances are represented by the following formula: ； in, The Manhattan x-axis coordinates representing the starting position. The Manhattan y-axis coordinate representing the starting position; (3) The evaluation function q is calculated from the motion factor, and the specific formula is as follows: ； in, This indicates the position of the unmanned forklift at time t. This represents the total cost of the unmanned forklift traveling from its current position to the target position at time t. This represents the actual cost from the starting position of the automated forklift to its position at time t. For motor compensation The planning algorithm iteratively solves for the location to obtain the optimal path.

6. A path planning system for unmanned forklifts based on dynamic prediction, characterized in that, include: The data preprocessing module, equipped with millimeter-wave radar, lidar and infrared camera, preprocesses the environmental information collected by millimeter-wave radar and lidar to obtain millimeter-wave radar map and lidar point cloud map, and evenly divides the millimeter-wave radar map, lidar point cloud map and infrared map collected by infrared camera into multiple field of view areas. The dynamic and static region segmentation module obtains the field of view to be segmented based on the distance values ​​of pixels in each field of view region of the millimeter-wave radar image, and then divides the multiple field of view regions to be segmented into dynamic field of view regions and static field of view regions based on the radial velocity values ​​of each pixel in the field of view regions to be segmented. The unmanned forklift positioning module finds the field of view area that matches the dynamic and static field of view areas in the millimeter-wave radar image in the laser point cloud image and infrared image. It designs a marker feature extraction algorithm for the laser point cloud image to calculate the laser radar position and a hotspot feature extraction algorithm for the infrared image to calculate the camera position. The unmanned forklift position is calculated from the laser radar position and the camera position. The path planning module, based on the position of the unmanned forklift, designs motion compensation. The planning algorithm calculates the motion factor from the Manhattan distance between the unmanned forklift's position and the target position, calculates the evaluation function from the motion factor, obtains the optimal path, and outputs the unmanned forklift's position and the optimal path. Implement the unmanned forklift path planning method based on dynamic prediction as described in any one of claims 1-5.

Images