Stacking machine perception interaction system and method for operation interaction

Abstract

The invention relates to a stacking machine perception interaction system and method for operation interaction. The system communicates with a wharf central control system and comprises a stacking machine body, an electric control cabinet installed on the stacking machine body, a sensing module comprising a plurality of sub-modules and sensors, a decision module and an interaction module. The sensing module is used for acquiring self information and operation environment information of the stacking machine during operation; the decision module is used for controlling each sensor of the sensing module and calling each sub-module according to the operation environment information, and sending a request instruction to the interaction module; and the interaction module is used for interacting with a wharf central control system, acquiring task information, issuing the task information to the decision module, and reporting container number information, operation container truck number information and container position information to the wharf central control system after a container receiving and sending task is completed. Compared with the prior art, the system has the advantages that the operation safety is guaranteed, the intellectualization of operation interaction is realized, the operation of a driver is reduced, and the operation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of smart port technology, and in particular to a forklift perception interaction system and method for operation interaction. Background Art

[0002] Empty container stackers (referred to as stackers) are key equipment for container transportation. They are widely used for stacking and transshipping empty containers in ports, docks, railway and highway transfer stations, and yards. They have the characteristics of high stacking layers, fast stacking and handling speed, high operating efficiency, flexibility, and space saving. With the rapid construction of smart ports and unmanned docks around the world, the intelligence of stackers has become an important part of terminal automation for container yard loading and unloading scenarios.

[0003] At present, in the operational interaction between forklifts and container trucks, the forklift drivers mainly rely on judging the relevant information in the current task and manually confirming the task data, resulting in low work efficiency. The forklifts are large in size and the drivers cannot fully consider the surrounding environment. The increasing number of AIV carts in ports has brought new challenges to the driving safety of forklifts. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a forklift perception interaction system and method for operation interaction, which can accurately and comprehensively obtain the environmental information of the forklift during operation, perform automatic perception interaction, and improve operation efficiency.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, there is provided a forklift perception interaction system for operation interaction, the system communicates with a terminal central control system, the system comprises a forklift body, an electric control cabinet installed on the forklift body, a perception module including a plurality of submodules and sensors, a decision module and an interaction module, the electric control cabinet comprises at least an industrial computer and a power module, wherein: the industrial computer is used to obtain the forklift's own sensor signals and run the perception module, the decision module and the interaction module, the power module is used to supply power to the industrial computer and each sensor; the perception module is used to obtain the forklift's own information and operating environment information during operation; the decision module is used to control each sensor of the perception module and call each submodule of the perception module according to the operating environment information, and send a request instruction to the interaction module; the interaction module is used to interact with the terminal central control system, obtain task information and send it to the decision module, and after completing the task of receiving and sending boxes, report the container number information, the operation truck number information and the box position information to the terminal central control system.

[0007] As a preferred technical solution, the forklift body includes a traveling mechanism, a gantry, a sling, a lifting cylinder and a swing cylinder. The gantry is respectively connected to the traveling mechanism and the sling, the lifting cylinder is installed on the gantry, and the swing cylinder is respectively connected to the traveling mechanism and the gantry.

[0008] As a preferred technical solution, the multiple sensors in the perception module include: a laser ranging sensor arranged at the headlight on one side near the bottom end of the gantry, a first pull-wire sensor arranged at the lifting cylinder, a second pull-wire sensor arranged at the swing cylinder, a first visual sensor arranged at the headlight on the other side near the bottom end of the gantry, a second visual sensor arranged on one side of the forklift cab, a positioning sensor arranged on the top of the forklift cab, a first laser radar arranged on one side of the tail of the forklift, and a second laser radar arranged on the other side of the tail of the forklift, and the industrial computer is connected to each sensor respectively.

[0009] As a preferred technical solution, the stacker's own sensor signal includes a rotation lock signal, an unlock signal and an alarm signal of the stacker body.

[0010] As a preferred technical solution, the multiple submodules in the perception module include: a data acquisition submodule, a positioning submodule, a path planning submodule, a visual recognition submodule, an on-board sensor signal processing submodule, an obstacle detection submodule, a box position calculation submodule and an abnormal alarm submodule; the data acquisition submodule is used to acquire and send the perception data of each sensor, and the remaining submodules are used to receive the perception data and process them separately.

[0011] As a preferred technical solution, the decision module includes a task decision submodule, a perception control submodule, a deviable reporting submodule and a task result reporting submodule; the task decision submodule is used to receive and store task information, the perception control submodule is used to control the various sensors of the perception module and call the various submodules of the perception module, the deviable reporting submodule is used to send a working vehicle deviable signal to the interaction module, and the task result reporting submodule is used to report the task result to the interaction module.

[0012] As a preferred technical solution, the interaction module includes a heartbeat submodule, an abnormality reporting submodule, a task data receiving submodule, an equipment status feedback submodule, a request for drivability module and a task result feedback submodule; the heartbeat submodule is used to communicate with the terminal central control system, the abnormality reporting submodule is used to report the forklift's own abnormal status and detected obstacle results to the terminal central control system, the task data receiving submodule is used to obtain task information from the terminal central control system and send it to the decision module, the equipment status feedback submodule is used to upload the current working status and spreader status of the forklift, the request for drivability module is used to request the operation container truck to leave the terminal central control system, and at the same time report the perceived container box number information and operation container truck number information; the task result feedback submodule is used to report the completion time, container box number, operation container truck number and container box position information of each operation task.

[0013] According to a second aspect of the present invention, there is provided a forklift perception interaction method for operation interaction, the method is implemented based on the system, and the method comprises the following steps: obtaining the forklift's own information and operating environment information during operation; judging the current container area according to the self information, and then obtaining a task list for the current container area; performing path planning according to the task list and the operating environment information; obtaining a first image, and judging whether the current object in front of the forklift is a container truck according to the first image, if so, entering the container receiving operation process, otherwise entering the container sending operation process; after completing the container receiving operation or the container sending operation, reporting the corresponding container number information, the operating container truck number information, and the container position information.

[0014] As an optimal technical solution, the container collection operation process specifically includes: according to the spreader height information, obtaining the second image and identifying the corresponding container number information, and the spreader height information is calculated according to the own information; according to the container number information, matching the task list to obtain the task information of the current operation; when the spreader state is the rotary lock, obtaining the third image and identifying the corresponding vehicle number information; according to the position offset of the forklift, sending a signal requesting the container truck to leave, and the position offset is calculated according to the own information; when the spreader state is unlocked, calculating the current container position information; reporting the current task result to complete this container collection operation.

[0015] As an optimal technical solution, the box dispatching operation process specifically includes: when the spreader status is twist-locked, calculating the current container position information; when the spreader status is unlocked, obtaining the fourth image and identifying the corresponding vehicle number information; according to the current position offset and spreader height information, obtaining the fifth image and identifying the corresponding container number information; matching the task list according to the container number information to obtain the task information of the current operation; sending a signal requesting the container truck to leave; reporting the current task result to complete this box dispatching operation.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. The present invention utilizes the interactive module to communicate with the terminal central control system to realize the transmission and identification of the task list. The perception module and the decision module communicate with the interactive module respectively to realize the mutual transmission of signals and information, and accurately control whether to enter the container receiving operation process or the container sending operation process. After entering the corresponding operation process, the mutual cooperation between the perception module, the decision module and the interactive module is used again to autonomously obtain the relevant task information during the operation, and the driver is replaced to confirm the autonomous task data, thereby improving the intelligence of the forklift and the efficiency of operation interaction, and facilitating port data management;

[0018] 2. The perception module in the present invention includes multiple sensors and sub-modules. The combined operation of the positioning sub-module and the path planning sub-module can provide the driver with the optimal path, and accurately identify and calculate the container area information, spreader height information, container number information, and vehicle number information. The interactive module can report the task results to the terminal central control system in a timely manner. By sensing the surrounding environment information, it can provide real-time environmental information for the forklift driver, eliminate the driver's driving blind spots, and ensure the safety of terminal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the hardware structure of the forklift perception interaction system provided by an embodiment of the present invention;

[0020] Figure 2 A structural block diagram of the software part of the forklift perception interaction system provided by an embodiment of the present invention;

[0021] Figure 3 A flowchart of the perception interaction of the in-and-out box operation of a forklift provided in an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a coordinate system for a forklift provided in an embodiment of the present invention;

[0023] Among them: 1. Forklift body; 2. Traveling mechanism; 3. Door frame; 4. Spreader; 5. Lifting cylinder; 6. Swinging cylinder; 7. Electric control cabinet; 8. Laser ranging sensor; 9. First pull-wire sensor; 10. Second pull-wire sensor; 11. First visual sensor; 12. Second visual sensor; 13. Positioning sensor; 14. First laser radar; 15. Second laser radar; 100. Perception module; 200. Decision module; 300. Interaction module; 101. Data acquisition submodule; 102. Positioning submodule; 103. Path planning submodule; 1 04, visual recognition submodule; 105, vehicle-mounted sensor signal processing submodule; 106, obstacle detection submodule; 107, box position calculation submodule; 108, abnormal alarm submodule; 201, task decision submodule; 202, perception control submodule; 203, derivable reporting submodule; 204, task result reporting submodule; 301, heartbeat submodule; 302, abnormal reporting submodule; 303, task data receiving submodule; 304, equipment status feedback submodule; 305, request derivable ion submodule; 306, task result feedback submodule. DETAILED DESCRIPTION

[0024] In view of the problems that the prior art still has difficulty in fully acquiring the operating environment information of the forklift, poor perception and interaction capabilities, and low operating efficiency, the present invention provides a forklift perception and interaction system for operation interaction, the system communicates with the terminal central control system, comprises a forklift body, an electric control cabinet installed on the forklift body, a perception module including multiple submodules and sensors, a decision module and an interaction module, the electric control cabinet comprises at least an industrial computer and a power module, wherein: the industrial computer is used to acquire the sensor signals of the forklift itself and run the perception module, the decision module and the interaction module, the power module is used to supply power to the industrial computer and each sensor; the perception module is used to acquire the forklift's own information and operating environment information during operation; the decision module is used to control each sensor of the perception module and call each submodule of the perception module according to the operating environment information, and send a request instruction to the interaction module; the interaction module is used to interact with the terminal central control system, acquire task information and send it to the decision module, and after completing the task of receiving and sending boxes, the container number information, the operation truck number information, and the box position information are reported to the terminal central control system.

[0025] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0026] Example

[0027] Figure 1 and Figure 2The hardware part and the software part of a forklift perception interaction system for operation interaction provided by this embodiment are respectively shown.

[0028] like Figure 1 As shown, the stacker body 1 includes a travel mechanism 2, a gantry 3, a spreader 4, a lifting cylinder 5 and a swing cylinder 6. The gantry 3 is connected to the travel mechanism 2 and the spreader 4 respectively, the lifting cylinder 5 is installed on the gantry 3, and the swing cylinder 6 is connected to the travel mechanism 2 and the gantry 3 respectively. The electric control cabinet 7 is arranged on the stacker body 1, and the electric control cabinet 7 includes an industrial computer, a switch and a power module, wherein: the industrial computer is a processor of the perception module 100, the decision module 200 and the interaction module 300, and is used to obtain the stacker's own sensor signals and operate the perception module 100, the decision module 200 and the interaction module 300. The stacker's own sensor signals are the rotation lock signal, the unlocking signal and the alarm signal of the spreader 4; the power module is used to power the industrial computer and each sensor in the perception module 100.

[0029] The perception module 100 includes multiple submodules and multiple sensors. Figure 1 The side where the middle mast 3 is located is the front, and the multiple sensors in the perception module 100 include: a laser ranging sensor 8 arranged at the left front wheel headlight of the forklift, a first pull-wire sensor 9 arranged at the lifting cylinder 5, a second pull-wire sensor 10 arranged at the swing cylinder 6, a first visual sensor 11 arranged at the right front wheel headlight of the forklift, a second visual sensor 12 arranged on the right side of the forklift cab, a positioning sensor 13 arranged on the top of the forklift cab, a first laser radar 14 arranged on the left side of the tail of the forklift, and a second laser radar 15 arranged on the right side of the tail of the forklift. The industrial computer is connected to each sensor of the perception module 100 respectively. Specifically, the laser ranging sensor 8 is used to obtain the distance information between the forklift and the object in front; the first pull-wire sensor 9 is used to obtain the height information of the spreader 4; the second pull-wire sensor 10 is used to obtain the inclination information of the spreader 4; the first visual sensor 11 is used to identify the vehicle number of the container truck and identify the working vehicle number information; the second visual sensor 12 is used to identify the container number and obtain the container number information; the positioning sensor 13 is used to obtain the position and posture information of the forklift body; the first laser radar 14 and the second laser radar 15 are used to obtain the surrounding environment information of the forklift, and the surrounding environment information of the forklift includes pedestrians, containers, guardrails, container trucks, and terminal buildings.

[0030] like Figure 2As shown, the perception module 100 is used to perceive the self information and working environment information of the forklift during operation. Among them, the self information includes the position and posture information of the forklift, the height and tilt information of the spreader 4, and the working environment information includes the operating container number, the distance information between the forklift and the object in front, and the current operating container truck number information. The container truck includes the container truck and the unmanned flatbed truck (AIV). The various submodules in the perception module 100 specifically include: a data acquisition submodule 101, a positioning submodule 102, a path planning submodule 103, a visual recognition submodule 104, an on-board sensor signal processing submodule 105, an obstacle detection submodule 106, a box position calculation submodule 107 and an abnormal alarm submodule 108. The functions of each submodule are as follows:

[0031] The data acquisition submodule 101 is used to obtain the data of each sensor and provide the corresponding sensor data to other submodules; the positioning submodule 102 is used to calculate the position of the forklift in the yard through the position information and obtain the information of the container area where it is located; the path planning submodule 103 generates the path of the forklift from the current point to the task target point according to the task information and position information; the visual recognition submodule 104 is used to identify the container box number and the truck number information; the on-board sensor signal processing submodule 105 is used to process the rotation lock signal, the unlocking signal and the alarm signal; the obstacle detection submodule 106 performs obstacle detection on the surrounding environment information of the forklift through the data obtained by the first laser radar 14 and the second laser radar 15; the box position calculation submodule 107 uses the position and posture information of the forklift and the inclination information of the spreader 4 to calculate the box position information of the operating container; the abnormal alarm submodule 108 is used to feed back the alarm signal of the on-board sensor signal processing submodule 105 and the obstacle information detected by the obstacle detection submodule 106 to the driver.

[0032] The decision module 200 is used to control each sensor in the perception module 100 and call each submodule in the perception module 100 according to the environmental information perceived by the perception module 100, and send a request instruction to the interaction module 300. The decision module 200 includes a task decision submodule 201, a perception control submodule 202, a deviable reporting submodule 203 and a task result reporting submodule 204. The task decision submodule 201 is used to obtain and store tasks from the interaction module 300 and control the perception interaction process during the operation; the perception control submodule 202 is used to control each sensor in the perception module 100 and call each submodule in the perception module 100; the deviable reporting submodule 203 sends a signal to the interaction module 300 that the operating vehicle can leave according to the perception information; the task result reporting submodule 204 reports the task result to the interaction module 300 according to the perception results of each submodule in the perception module 100.

[0033] The interaction module 300 is used to interact with the terminal central control system, obtain task information and send it to the decision module 200, and report the container number information, operation truck number information, and box position information to the terminal central control system after completing the task of receiving and sending containers. The interaction module 300 includes a heartbeat submodule 301, an abnormality reporting submodule 302, a task data receiving submodule 303, an equipment status feedback submodule 304, a request for drivable ion module 305, and a task result feedback submodule 306. The heartbeat submodule 301 is used to maintain communication with the terminal central control system; the abnormal reporting submodule 302 is used to report the abnormal state of the forklift itself and the obstacle results detected by the obstacle detection submodule 106 in the perception module 100 to the terminal central control system; the task data receiving submodule 303 is used to obtain tasks from the terminal central control system and send tasks to the decision module 200; the equipment status feedback submodule 304 is used to upload the current working status and spreader status of the forklift, the working status includes waiting for tasks, working and completing the work, and the spreader status includes locking and unlocking; the request for drivable submodule 305 is used to request the operation container truck to leave from the terminal central control system, and at the same time report the perceived container box number information and operation container truck number information; the task result feedback submodule 306 is used to report the completion time, container box number, operation container truck number and container box position information of each operation task.

[0034] The workflow of the forklift perception interaction system is as follows:

[0035] When the forklift is running, the positioning submodule is used to calculate the current position information and container area information of the forklift. The interactive module obtains the task list from the terminal central control system according to the container area information of the forklift, calls the path planning submodule to plan the path, and provides the driver with the optimal path. When it is detected that the forklift is 3-4 meters away from the object in front, the second visual sensor 12 is controlled to take a photo and the visual recognition submodule is called to determine whether the object in front is a container truck. If it is a container truck, the container collection operation process is entered, otherwise the container delivery operation process is entered.

[0036] When the driver is collecting containers, the decision module controls the second visual sensor 12 to take pictures and calls the visual recognition submodule to identify the container number information according to the spreader height information. The decision module determines the currently executed task information according to the container number information. After the spreader 4 changes from unlocked to twist locked, the decision module controls the first visual sensor 11 to take pictures and calls the visual recognition submodule to identify the container truck number information. According to the forklift's retreat distance and the spreader height information, a request for the container truck to leave is sent to the request drivable submodule. After the spreader 4 changes from twist locked to unlocked, the container slot calculation submodule is called to calculate the current container slot information. Finally, the interactive module uploads the completion time of the task, container number, operation container truck number and container slot information to the terminal central control system through the task result feedback submodule.

[0037] When the driver is performing the container dispatching operation, when the spreader 4 changes from unlocked to twist locked, the container position calculation submodule is called to calculate the current container position information. When the spreader 4 changes from twist locked to unlocked, the decision module controls the first visual sensor 11 to take pictures and calls the visual recognition submodule to identify the container truck number information. The decision module controls the second visual sensor 12 to take pictures and calls the visual recognition submodule to identify the container number information based on the forklift retreat distance and the spreader height information. The decision module determines the currently executed task information based on the container number information, calls the request drivable submodule to issue a request for the container truck to leave, and finally the interactive module uploads the completion time of the operation task, container number, operation container truck number and container position information to the terminal central control system through the task result feedback submodule.

[0038] Furthermore, based on the aforementioned system, this embodiment also provides a forklift perception interaction method for operation interaction, such as Figure 3 As shown, the method comprises the following steps:

[0039] A1. Determine the container area where the forklift is located and obtain the current container area task list. Specifically:

[0040] The positioning submodule 102 calculates the current container area where the forklift is located based on the forklift position information (one of its own information) provided by the data acquisition submodule 101. The task decision submodule 201 sends the calculated container area to the task data receiving submodule 303. The task data receiving submodule 303 obtains the task list of the current container area from the terminal central control system and sends it to the task decision submodule 201.

[0041] A2. Calling the route planning submodule 103 to perform route planning and provide the driver with the optimal route. Specifically:

[0042] The perception control submodule 202 calls the path planning submodule 103 to perform path planning based on the position information in the task information of the task list, obtains the optimal path from the current position of the forklift to the specified task target point, and displays the optimal path to the driver on the cab screen.

[0043] A3. When it is detected that the forklift is 3-4 meters away from the object in front, the second visual sensor 12 is controlled to take a photo and the visual recognition submodule 104 is called to determine whether the object in front is a container truck. If it is a container truck, the receiving operation process B is entered, otherwise the sending operation process C is entered. Specifically:

[0044] When the data of the laser ranging sensor 8 is 3-4 meters, the perception control submodule 202 controls the second visual sensor 12 to take pictures (i.e., obtain the first image), and the visual recognition submodule 104 uses the YOLOv4 algorithm to detect the target and determine whether the object in front is a container truck. If it is a container truck, the task decision submodule 201 sets the perception process to the box receiving operation process, otherwise, it is set to the box sending operation process.

[0045] B. Box collection process

[0046] B1. When the spreader height is 4-4.5 meters, the second visual sensor 12 is controlled to take pictures and the visual recognition submodule 104 is called to recognize the box number. Specifically:

[0047] The data of the first pull-wire sensor 9 multiplied by 4 is the sling height. When the sling height is 4-4.5 meters, the perception control submodule 202 controls the second visual sensor 12 to take a photo (i.e., obtain a second image), and the visual recognition submodule 104 uses the OCR text recognition algorithm to recognize the box number of the photo.

[0048] B2. Match the task list according to the box number information and obtain the task information of the current job, specifically:

[0049] The task decision submodule 201 searches for a task containing the box number information in the task list according to the identified box number, and sets it as the current task.

[0050] B3. When the forklift spreader 4 is locked, the first visual sensor 11 is controlled to take a photo and the visual recognition submodule 104 is called to recognize the vehicle number. Specifically:

[0051] When the lifting device 4 is a rotary lock, the perception control submodule 202 controls the first visual sensor 11 to take a photo (ie, obtain a third image), and the visual recognition submodule 104 uses an OCR text recognition algorithm to recognize the vehicle number on the photo.

[0052] B4. When the forklift moves back 2-3 meters, the drivable ion module 305 is called to issue a request for the container truck to leave. Specifically:

[0053] The positioning submodule 102 calculates the position offset of the forklift. When the forklift moves back 2-3 meters, the task decision submodule 201 calls the requesting drivable submodule 305 to send a signal requesting the container truck to leave to the terminal central control system.

[0054] B5. When the forklift spreader 4 is unlocked, the container location calculation submodule 107 is called to calculate the current container location information, which includes the container location, column and layer;

[0055] The forklift continues to operate. When the spreader 4 is unlocked, the forklift has placed the container on the specified position. At this time, the position calculation submodule 107 is called to perform position calculation. First, the coordinate value of the center of the container in the spreader coordinate system is converted into the coordinate value in the world coordinate system. The coordinate value is calculated with the position information of the container area in the yard to obtain the current container position.

[0056] B6. Upload the task result feedback to the terminal central control system, specifically:

[0057] The task result feedback submodule 306 uploads the completion time of the operation task, the operation container number, the truck number and the container position to the terminal central control system, and the container collection operation is completed.

[0058] C. Shipping Process

[0059] C1. When the forklift spreader 4 is locked, the container position calculation submodule 107 is called to calculate the current container position information, specifically:

[0060] When the spreader 4 is a twist lock, the container position calculation submodule 107 is called to perform container position calculation. First, the coordinate value of the container center in the spreader coordinate system is converted into the coordinate value in the world coordinate system, and the coordinate value is calculated with the container position information of the yard container area to obtain the current container position.

[0061] C2. When the forklift spreader 4 is unlocked, the first visual sensor 11 is controlled to take a photo and the visual recognition submodule 104 is called to recognize the vehicle number. Specifically:

[0062] The forklift continues to operate, and when the spreader 4 is unlocked, the perception control submodule 202 controls the first visual sensor 11 to take a photo (ie, obtain a fourth image), and the visual recognition submodule 104 uses an OCR text recognition algorithm to recognize the vehicle number on the photo.

[0063] C3. When the forklift moves back 3-4 meters and the spreader is at a height of 4-4.5 meters, the second visual sensor 12 is controlled to take a picture and the visual recognition submodule 104 is called to recognize the container number. Specifically:

[0064] When the data of the laser ranging sensor 8 is 3-4 meters and the sling height is 4-4.5 meters, the perception control submodule 202 controls the second visual sensor 12 to take pictures (ie, obtain the fifth image), and the visual recognition submodule 104 uses the OCR text recognition algorithm to recognize the box number of the photo.

[0065] C4. Match the task list according to the box number information and obtain the task information of the current job, specifically:

[0066] The task decision submodule 201 searches for a task containing the box number information in the task list according to the identified box number, and sets it as the current task.

[0067] C5, calling the drivable ion module 305 to issue a request for the container truck to leave;

[0068] The module 305 for requesting the truck to leave is called to send a signal for requesting the truck to leave to the terminal central control system.

[0069] C6. Upload the task result feedback to the terminal central control system, specifically:

[0070] The task result feedback submodule 306 uploads the completion time of the operation task, the operation container number, the truck number and the container position to the terminal central control system, and the container dispatch operation is completed.

[0071] In steps B5 and C1, the coordinate value of the center of the container in the spreader coordinate system is converted into the coordinate value in the world coordinate system. The specific process includes:

[0072] like Figure 4 As shown, it contains the world coordinate system X n Y n Z n , body coordinate system X b Y b Z b , spreader coordinate system X d Y d Z d The data obtained by the positioning sensor 13 is the longitude and latitude data in the WGS84 coordinate system. The positioning module 102 first converts the longitude and latitude into plane rectangular coordinates in the UTM coordinate system, and establishes the world coordinate system X with the Z axis pointing to the sky. n Y n Z n , the z coordinate of the origin of the body coordinate system is consistent with the z coordinate of the origin of the world coordinate system. When the spreader grabs the container, the coordinate value of the center of the container in the spreader coordinate system is P d , the coordinate value of the container center in the world coordinate system is P n , then the conversion relationship expression from the coordinates in the spreader coordinate system to the coordinates in the world coordinate system is as follows:

[0073]

[0074] in, is the homogeneous transformation matrix of the body coordinate system relative to the world coordinate system, It is the homogeneous transformation matrix of the sling coordinate system relative to the vehicle body coordinate system.

[0075] When the forklift is driving, due to the flat road surface, Z nThe direction remains unchanged, only X n , Y n The direction changes, and the position of the vehicle body coordinate system relative to the world coordinate system is calculated as:

[0076]

[0077] The posture of the vehicle body coordinate system relative to the world coordinate system is:

[0078]

[0079] In the above formula, the three coordinate values ​​are the roll, pitch and heading angles of the vehicle body.

[0080] The center of the sling and the body coordinate system Y b direction, therefore, the position of the sling coordinate system relative to the vehicle body coordinate system is:

[0081]

[0082] Among them, a is the distance between the center of the rear wheel axle of the forklift and the center of the spreader at Y b Horizontal distance in the direction, z bd is the height of the sling, and the posture of the sling coordinate system relative to the vehicle body coordinate system is:

[0083]

[0084] In the formula, roll bd It is the front and rear tilt angle of the mast.

[0085] According to the above formula, the xyz value of the center of the container in the world coordinate system can be obtained.

[0086] In summary, the forklift perception interaction system for operation interaction provided by the present invention includes a forklift body, an electric control cabinet, a perception module, a decision module and an interaction module. Through the perception module, the decision module and the interaction module, the intelligence of the forklift operation interaction can be realized, and the surrounding environment can be perceived during operation, thereby realizing the automation of operation interaction, effectively improving the operation efficiency of the forklift, ensuring the driving safety of the driver, and ultimately improving the intelligence of the forklift operation.

[0087] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A forklift sensing and interaction system for operation interaction, the system communicates with the terminal central control system, characterized in that: The system includes a stacker body, an electric control cabinet installed on the stacker body, a perception module including multiple submodules and sensors, a decision module and an interaction module, and the electric control cabinet includes at least an industrial computer and a power module, wherein: The industrial computer is used to obtain the sensor signals of the forklift itself and run the perception module, the decision module and the interaction module, and the power module is used to supply power to the industrial computer and each sensor; The sensing module is used to obtain the forklift's own information and operating environment information during operation; The decision module is used to control the various sensors of the perception module and call the various sub-modules of the perception module according to the working environment information, and send a request instruction to the interaction module; The interaction module is used to interact with the terminal central control system, obtain task information and send it to the decision module, and report the container number information, operation truck number information, and box location information to the terminal central control system after completing the box receiving and sending task.

2. The forklift perception interaction system for operation interaction according to claim 1, characterized in that: The forklift body includes a traveling mechanism, a gantry, a sling, a lifting cylinder and a swinging cylinder. The gantry is respectively connected to the traveling mechanism and the sling, the lifting cylinder is installed on the gantry, and the swinging cylinder is respectively connected to the traveling mechanism and the gantry.

3. The forklift perception interaction system for operation interaction according to claim 2 is characterized in that: The multiple sensors in the perception module include: a laser ranging sensor arranged at the headlight on one side near the bottom end of the gantry, a first pull-wire sensor arranged at the lifting cylinder, a second pull-wire sensor arranged at the swing cylinder, a first visual sensor arranged at the headlight on the other side near the bottom end of the gantry, a second visual sensor arranged on one side of the forklift cab, a positioning sensor arranged on the top of the forklift cab, a first laser radar arranged on one side of the tail of the forklift, and a second laser radar arranged on the other side of the tail of the forklift, and the industrial computer is connected to each sensor respectively.

4. The forklift perception interaction system for operation interaction according to claim 1, characterized in that: The stacker's own sensor signals include a rotation lock signal, an unlock signal and an alarm signal of the stacker body.

5. The forklift perception interaction system for operation interaction according to claim 1, characterized in that: The multiple submodules in the perception module include: a data acquisition submodule, a positioning submodule, a path planning submodule, a visual recognition submodule, an on-board sensor signal processing submodule, an obstacle detection submodule, a box position calculation submodule and an abnormal alarm submodule; the data acquisition submodule is used to acquire and send the perception data of each sensor, and the remaining submodules are used to receive the perception data and process them separately.

6. The forklift perception interaction system for operation interaction according to claim 1, characterized in that: The decision module includes a task decision submodule, a perception control submodule, a deviable reporting submodule and a task result reporting submodule; the task decision submodule is used to receive and store task information, the perception control submodule is used to control the various sensors of the perception module and call the various submodules of the perception module, the deviable reporting submodule is used to send a deviable signal of the operating vehicle to the interaction module, and the task result reporting submodule is used to report the task result to the interaction module.

7. The forklift perception interaction system for operation interaction according to claim 1, characterized in that: The interaction module includes a heartbeat submodule, an abnormality reporting submodule, a task data receiving submodule, an equipment status feedback submodule, a request for drivability module and a task result feedback submodule; the heartbeat submodule is used to communicate with the terminal central control system, the abnormality reporting submodule is used to report the abnormal state of the forklift itself and the detected obstacle results to the terminal central control system, the task data receiving submodule is used to obtain task information from the terminal central control system and send it to the decision module, the equipment status feedback submodule is used to upload the current working state and spreader state of the forklift, the request for drivability module is used to request the operation container truck to leave from the terminal central control system, and at the same time report the perceived container box number information and operation container truck number information; the task result feedback submodule is used to report the completion time, container box number, operation container truck number and container box position information of each operation task.

8. A forklift perception interaction method for operation interaction, characterized in that: The method is implemented based on the system according to any one of claims 1 to 7, and the method comprises the following steps: Obtain the forklift's own information and operating environment information during operation; Determine the current box area according to the self information, and then obtain the task list of the current box area; Performing path planning according to the task list and the working environment information; Acquire a first image, and determine whether the object in front of the forklift is a container truck according to the first image. If yes, enter the box receiving operation process; otherwise, enter the box sending operation process; After completing the container receiving or sending operation, report the corresponding container number information, operation truck number information, and box location information.

9. The forklift perception interaction method for operation interaction according to claim 8, characterized in that: The box collection process specifically includes: According to the spreader height information, acquiring the second image and identifying the corresponding container number information, wherein the spreader height information is calculated according to the self information; Match the task list according to the box number information to obtain the task information of the current job; When the state of the spreader is the rotation lock, the third image is acquired and the corresponding vehicle number information is identified; Sending a signal requesting the container truck to leave according to the position offset of the forklift, wherein the position offset is calculated according to the self information; When the spreader status is unlocked, calculate the current container location information; Report the current task results and complete this box collection operation.

10. The forklift perception interaction method for operation interaction according to claim 9, characterized in that: The box delivery process specifically includes: When the spreader is in the twistlock state, calculate the current container location information; When the spreader is in unlocked state, acquiring a fourth image and identifying corresponding vehicle number information; According to the current position offset and spreader height information, a fifth image is acquired and corresponding container number information is identified; Match the task list according to the box number information to obtain the task information of the current job; Send a signal requesting the container truck to leave; Report the current task results and complete this delivery operation.

Images