A visual picking and placing method for an unmanned omnidirectional forklift

Abstract

The present application relates to the field of mobile robot / AGV control technology, in particular to a visual picking and placing method of an unmanned omnidirectional forklift, comprising a picking method and an omnidirectional forklift, wherein the omnidirectional forklift comprises a visual sensor, an omnidirectional chassis and a fork arm action mechanism, the omnidirectional unmanned forklift adopts advanced omnidirectional movement technology, can realize flexible turning and accurate positioning in a small space, thereby overcoming the deficiency of traditional single rudder chassis unmanned forklift in flexibility. At the same time, combined with a visual recognition algorithm, the omnidirectional unmanned forklift can realize real-time sensing of the position, shape and state information of goods in the working environment, and realize accurate picking and placing operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile robot / AGV control, in particular to a visual picking and placing method for an unmanned omnidirectional forklift. BACKGROUND

[0002] In the warehousing and logistics industry, unmanned forklifts, as a cutting-edge automated equipment, are gradually becoming a powerful assistant for various industries, and their application range is increasingly wide. Compared with traditional manual forklifts, unmanned forklifts have shown significant advantages in terms of operation accuracy, consistency and efficiency. They can accurately perform various tasks such as cargo handling, storage and sorting, thereby greatly improving the overall efficiency of warehousing and logistics operations.

[0003] However, although unmanned forklifts have shown great potential in automated operations, they still face some challenges in actual application. In particular, in complex scenarios such as narrow aisle space, inaccurate cargo placement, and overloading or overloading of goods on the shelves, traditional single-rudder chassis unmanned forklifts are prone to be unable to successfully complete the task due to insufficient flexibility.

[0004] Specifically, when traditional single-rudder chassis unmanned forklifts turn in narrow spaces, they often require a large turning radius, which limits their flexible application in narrow aisles. In addition, when the placement of goods is not accurate, single-rudder chassis unmanned forklifts may not be able to accurately adjust the position of the vehicle body, resulting in the inability to accurately place the goods at the designated location. More complex is when there are overloading or overloading of goods on the shelves, single-rudder chassis unmanned forklifts may be unable to work due to limitations in vehicle height or stability. SUMMARY

[0005] (I) Invention purpose

[0006] Therefore, the purpose of the present application is to provide a visual picking and placing method for an unmanned omnidirectional forklift. The omnidirectional unmanned forklift uses advanced omnidirectional mobile technology, which can realize flexible turning and accurate positioning in narrow spaces, thereby overcoming the shortcomings of traditional single-rudder chassis unmanned forklifts in flexibility.

[0007] (II) Technical solutions

[0008] A visual picking and placing method of an omnidirectional unmanned forklift, comprising a picking method and an omnidirectional unmanned forklift, the omnidirectional unmanned forklift comprising a visual sensor, an omnidirectional chassis and a fork arm action mechanism, the omnidirectional chassis comprising a plurality of rudders, each rudder having a steering and walking function, and the steering angle and speed of each rudder being controlled by an algorithm to realize lateral movement, forward / backward movement and self-rotation, the fork arm action mechanism comprising a hydraulic system, a fork arm and a distance sensor, and different operation logics are executed by controlling the lifting and forward / backward movement of the fork arm, the visual sensor can be fixed at any position on the fork arm and the fork tip and move with them, or be fixed on the vehicle body of the forklift, and is used to identify the pose information of the goods, the picking method comprising the following steps:

[0009] 1) Pretreatment stage: a plurality of pictures of the goods are collected, and a model is established by using a reinforcement learning algorithm to identify goods in different poses;

[0010] 2) Motion instruction execution stage: the omnidirectional unmanned forklift is started, and a motion instruction for moving to a preset storage location is executed;

[0011] 3) Approach storage location point judgment stage: whether the forklift approaches the preset storage location point is judged during the movement of the forklift; if yes, the next step is entered; if no, the motion instruction is continuously executed to move to the storage location; the judgment is based on the distance or time information between the forklift and the preset storage location point;

[0012] 4) Visual identification and target pose acquisition stage: when the forklift approaches the preset storage location point, the visual sensor is started, the pose of the target goods or goods shelf is identified, and the identification result is fed back to the control system;

[0013] 5) Path planning stage: a collision-free path is dynamically generated according to the pose information of the goods or goods shelf fed back by the visual sensor, and the current position and attitude of the forklift; the path planning strategy makes decisions according to the degree of deviation of the goods and the feasible space conditions, including generating a curved path or a straight path for lateral movement after deceleration and parking, while considering the kinematic constraints and space occupation optimization of the forklift;

[0014] 6) Moving to the storage location stage: the omnidirectional unmanned forklift is controlled to move to the preset storage location along the planned path;

[0015] 7) Fork arm action execution stage: the lifting and forward / backward movement of the fork arm are controlled according to the pose information of the target goods to execute the picking or placing task; during the execution of the task, if the current pose cannot meet the operation requirements detected by the visual system, the pose of the forklift is adjusted according to the pose information provided by the visual system, and the path is re-planned to ensure the safe and accurate completion of the task;

[0016] 8) End stage: when the picking or placing task is completed, the whole process is ended.

[0017] Preferably, the approaching warehouse position judgment in step 3 is based on the relative speed or acceleration information between the forklift and the preset warehouse position.

[0018] Preferably, the path planning strategy in step 5 also includes considering energy consumption optimization and path smoothness of the forklift.

[0019] Preferably, the forklift arm action execution phase in step 7 also includes adopting different execution strategies for different types of warehouse positions, such as shelves, ground pile warehouse positions, and stacked rack warehouse positions.

[0020] Preferably, for stacked rack warehouse positions, before executing the put-away task, the method further includes the step of identifying the features of the racks through the vision system to confirm whether the upper and lower racks are correctly aligned.

[0021] Preferably, the forklift arm action execution phase in step 7 also includes real-time monitoring and feedback adjustment of the forklift arm movement to ensure the accuracy and stability of the operation.

[0022] From the above technical solutions, the present application has the following beneficial effects:

[0023] 1. The omnidirectional unmanned forklift of the present application adopts advanced omnidirectional mobile technology, which can realize flexible turning and precise positioning in a narrow space, thereby overcoming the shortcomings of traditional single-rudder chassis unmanned forklifts in flexibility. At the same time, combined with the vision recognition algorithm, the omnidirectional unmanned forklift can real-time perceive the position, shape and state information of the goods in the working environment, and realize accurate picking and putting operations. This innovative solution aims to improve the adaptability and operation efficiency of the unmanned forklift in complex working environments.

[0024] 2. The omnidirectional forklift vision picking / putting strategy of the present application is also optimized for different task types, using vision sensors to real-time judge whether the picking / putting conditions are met, thereby ensuring the safety and reliability of the picking / putting actions. Combined with advanced omnidirectional mobile technology, the unmanned forklift can realize flexible turning in a narrow space while accurately perceiving the goods information in the working environment, realizing accurate picking and putting operations. This solution not only improves the adaptability of the unmanned forklift in complex working environments, but also further guarantees the safety and efficiency of the operation process, providing strong support for the intelligent development of the warehousing and logistics industry. BRIEF DESCRIPTION OF DRAWINGS

[0025] Fig. 1 The omnidirectional forklift according to the present application moves to the warehouse position flowchart;

[0026] Fig. 2 The omnidirectional forklift according to the present application moves to the warehouse position flowchart;

[0027] In the diagram: 1. Omnidirectional forklift; 2. Omnidirectional chassis; 3. Fork arm operating mechanism; 4. Steering wheel; 5. Vision sensor. Detailed Implementation

[0028] The following description is exemplary in nature and is not intended to limit the scope, application, or use of this disclosure. It should be understood that in all these figures, the same or similar reference numerals indicate the same or similar parts and features. The figures are merely schematic representations of the concept and principles of embodiments of this disclosure and do not necessarily show the specific dimensions and scale of the various embodiments of this disclosure. Certain details or structures of embodiments of this disclosure may be exaggerated in particular portions of certain figures.

[0029] Please see Figs. 1-2 One embodiment provided by the present invention:

[0030] A visual picking and placing method for unmanned omnidirectional forklifts includes a picking method and an omnidirectional forklift 1. The omnidirectional forklift 1 includes a vision sensor 5, an omnidirectional chassis 2, and a fork arm actuation mechanism 3. The omnidirectional chassis 2 includes multiple sets of steering wheels 4, each of which has both steering and walking functions. The steering angle and rotation speed are controlled by an algorithm to achieve lateral movement, forward / backward movement, and rotation. The fork arm actuation mechanism 3 includes a hydraulic system, fork arms, and a distance sensor. Different operating logics are executed by controlling the lifting and lowering and forward / backward movement of the fork arms. The vision sensor 5 can be fixed at any position on the fork arm and on the fork tip and move accordingly, or it can be fixed to the forklift body to identify the positional information of the goods. The picking method includes the following steps:

[0031] 1) Preprocessing stage: Collect multiple sets of images of the goods and use reinforcement learning algorithms to build a model to identify the goods in different poses;

[0032] 2) Execution of motion command phase: Start the unmanned omnidirectional forklift and execute the motion command to move towards the preset storage location;

[0033] 3) Approaching Storage Location Judgment Stage: During the forklift movement, it is determined whether it is approaching the preset storage location; if it is, it proceeds to the next step; if it is not, it continues to execute the movement command and moves towards the storage location; this judgment is based on the distance or time information between the forklift and the preset storage location.

[0034] 4) Visual recognition and target pose acquisition stage: When the forklift approaches the preset storage point, the vision sensor 5 is activated to identify the pose of the target goods or shelves and the recognition result is fed back to the control system.

[0035] 5) Path planning stage: Based on the position and pose information of goods or shelves fed back by vision sensor 5, as well as the current position and attitude of the forklift, a collision-free path is dynamically generated; the path planning strategy makes decisions based on the degree of deviation of goods and feasible space conditions, including generating curved paths or planning straight paths for lateral movement after deceleration and stopping, while taking into account the kinematic constraints and space occupation optimization of the forklift.

[0036] 6) Moving to the storage location stage: Control the unmanned omnidirectional forklift to move to the preset storage location along the planned path;

[0037] 7) Forklift Action Execution Phase: Based on the position and orientation information of the target goods, control the lifting and lowering and forward and backward movement of the forklift to perform the task of picking up or placing goods; during the execution of the task, if the vision system detects that the current position and orientation cannot meet the operation requirements, the position and orientation of the forklift will be adjusted according to the position and orientation information provided by the vision system, and the path will be replanned to ensure the safe and accurate completion of the task.

[0038] 8) End Phase: Once the pickup or delivery task is completed, the entire process ends.

[0039] Furthermore, the proximity judgment in step 3 is based on the relative speed or acceleration information between the forklift and the preset storage location, which can more accurately determine whether the forklift is close to the target position, thereby activating the vision recognition system in a timely manner, reducing unnecessary waiting time, and improving work efficiency.

[0040] Furthermore, the path planning strategy in step 5 also includes considering the energy consumption optimization and path smoothness of the forklift, which can ensure that the forklift moves in a more energy-efficient manner when performing tasks, reducing operating costs.

[0041] Furthermore, the forklift action execution phase in step 7 also includes adopting different execution strategies for different types of storage locations (such as racks, floor storage locations, and stacked material rack storage locations). By designing specific execution strategies for each type of storage location, it can be ensured that the forklift can complete the task efficiently and accurately in different environments.

[0042] Furthermore, for stacked rack storage locations, before executing the unloading task, a step is included in which the characteristics of the racks are identified through a vision system to confirm whether the upper and lower racks are correctly aligned, thereby preventing safety hazards caused by improper stacking. Identifying rack characteristics and confirming alignment through a vision system before unloading goods at stacked rack storage locations ensures that goods are stacked neatly and stably, avoiding safety issues such as goods collapsing or forklift damage caused by improper stacking.

[0043] Furthermore, the forklift action execution stage in step 7 also includes real-time monitoring and feedback adjustment of the forklift movement to ensure the accuracy and stability of the operation, and to ensure that the position and posture of the forklift are accurate during the picking and placing of goods.

[0044] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for unmanned omnidirectional forklift vision-based picking and placing of goods, characterized in that: The system includes a picking method and an omnidirectional forklift (1). The omnidirectional forklift (1) includes a vision sensor (5), an omnidirectional chassis (2), and a fork arm action mechanism (3). The omnidirectional chassis (2) includes multiple steering wheels (4), each of which has both steering and walking functions. The steering angle and speed are controlled by an algorithm to achieve lateral movement, forward / backward movement, and self-rotation. The fork arm action mechanism (3) includes a hydraulic system, fork arms, and a distance sensor. Different operating logics are executed by controlling the lifting and lowering and forward and backward movement of the fork arms. The vision sensor (5) is fixed at any position on the fork arm and on the fork tip and moves accordingly, or is fixed on the forklift body to identify the position and posture information of the goods. The picking method includes the following steps: 1) Preprocessing stage: Collect multiple sets of images of the goods and use reinforcement learning algorithms to build a model to identify the goods in different poses; 2) Execution of motion command phase: Start the unmanned omnidirectional forklift and execute the motion command to move towards the preset storage location; 3) Approaching Storage Location Judgment Stage: During the forklift movement, it is determined whether it is approaching the preset storage location; if it is, it proceeds to the next step; if it is not, it continues to execute the movement command and moves towards the storage location; this judgment is based on the distance or time information between the forklift and the preset storage location. 4) Visual recognition and target pose acquisition stage: When the forklift approaches the preset storage point, the visual sensor (5) is activated to identify the pose of the target goods or shelves and the recognition result is fed back to the control system; 5) Path planning stage: Based on the position and pose information of goods or shelves fed back by the vision sensor (5), as well as the current position and posture of the forklift, a collision-free path is dynamically generated; the path planning strategy is decided based on the degree of deviation of goods and feasible space conditions, including generating a curved path or planning a straight path for lateral movement after deceleration and stopping, while considering the kinematic constraints and space occupation optimization of the forklift. 6) Moving to the storage location stage: Control the unmanned omnidirectional forklift to move to the preset storage location along the planned path; 7) Forklift Action Execution Phase: Based on the position and orientation information of the target goods, control the lifting and lowering and forward and backward movement of the forklift to perform the task of picking up or placing goods; during the execution of the task, if the vision system detects that the current position and orientation cannot meet the operation requirements, the position and orientation of the forklift will be adjusted according to the position and orientation information provided by the vision system, and the path will be replanned to ensure the safe and accurate completion of the task. 8) End Phase: Once the pickup or delivery task is completed, the entire process ends.

2. A method for unmanned omnidirectional forklift vision-based picking and placing of goods according to claim 1: characterized in that: The proximity determination in step 3) is based on the relative speed or acceleration information between the forklift and the preset storage location.

3. A method for unmanned omnidirectional forklift vision-based picking and placing of goods according to claim 1: characterized in that: The path planning strategy in step 5) also includes consideration of forklift energy consumption optimization and path smoothness.

4. A method for unmanned omnidirectional forklift vision-based picking and placing of goods according to claim 1: characterized in that: For stacked rack storage locations, before executing the delivery task, there is also a step of identifying the characteristics of the racks through a vision system to confirm whether the upper and lower racks are correctly aligned.

5. A method for unmanned omnidirectional forklift vision-based picking and placing of goods according to claim 1: characterized in that: The fork arm movement execution stage in step 7) also includes real-time monitoring and feedback adjustment of the fork arm movement to ensure the accuracy and stability of the operation.

Images