A method for automatically loading AGV

Abstract

The present invention discloses an AGV automatic loading method, which includes: an AGVS dispatching system dispatches an AGV equipped with a scanning device to circle a truck compartment, and sends instructions to a three-dimensional scene reconstruction module, and the three-dimensional scene reconstruction module reconstructs the three-dimensional scene of the compartment; after collecting the three-dimensional scene graph, the three-dimensional scene reconstruction module transmits the point cloud data to the target object type detection module; the target object type detection module identifies the type of goods in the compartment, determines whether it can be stacked, calculates the stacking information, and packages the stacking information and sends it to the task management system WMS; after the task management system WMS receives the stacking information, it assigns the cargo handling or loading and unloading task to the stacking type AGV. Not only does it reduce the workload of manual loading and unloading, but it also solves the problem of confusion in the types of cargo stacking, and at the same time, it can obtain cargo information for unified management, and realizes automated handling, thereby improving handling efficiency.

Description

Technical Field

[0001] The invention belongs to the field of AGV automation and relates to an AGV automatic loading method. Background Art

[0002] Automatic loading and handling is the last step of AGV automation. At present, when AGV transports goods, it needs to know the world coordinate system where the goods are located in advance or the coordinates of the target point corresponding to the goods to be transported. However, due to inaccurate parking positions of trucks, unclear types of goods in cargo boxes, and unstable positions of goods in carriages, container loading at terminals is still carried out by manually driven traditional forklifts to deliver goods to containers. However, this method of transportation is often prone to confusion due to subjective human factors, and it is difficult to enter cargo information into the system for unified management. In addition, it is difficult to achieve 24-hour uninterrupted operation using manual forklifts, making it difficult for companies to achieve a fully automated process. Summary of the invention

[0003] The purpose of the present invention is to provide a method for realizing automatic loading of AGV based on deep learning technology.

[0004] The technical solution of the present invention is: an AGV automatic loading method, which is applied to a system including a control computing device, an AGV equipped with a scanning device, a visual device, and a stacking type AGV, wherein the control computing device integrates an AGVS scheduling system, a three-dimensional scene reconstruction module, a target object type detection module, and a task management system WMS. The AGV automatic loading method includes:

[0005] Step 1, dispatching the AGV equipped with a scanning device to circle the truck compartment through the AGVS dispatching system, issuing instructions to the 3D scene reconstruction module, and triggering the 3D scene reconstruction module to reconstruct the 3D scene of the compartment;

[0006] Step 2, after the three-dimensional scene graph is collected, a stop collection instruction is sent to the three-dimensional scene reconstruction module through the AGVS scheduling system, and the point cloud data is transmitted to the target object type detection module through the three-dimensional scene reconstruction module;

[0007] Step 3, identifying the type of goods in the carriage through the target object type detection module, determining whether they can be stacked, and calculating stacking information, the stacking information at least including the calculated coordinates of the goods, the available free area in the carriage, and information on whether the goods can be stacked, and packaging the stacking information through the target object type detection module and sending it to the task management system WMS;

[0008] Step 4: After the task management system WMS receives the stacking information, it allocates cargo handling or loading and unloading tasks to the stacking AGV, and assigns the stacking AGV to complete the corresponding cargo handling or loading and unloading tasks.

[0009] The carriage scene is modeled through a system including control computing equipment, AGV equipped with scanning equipment, visual equipment, and stacking AGV. The type of cargo in the carriage is identified in combination with the target object type detection module, and the stacking information of the cargo is determined. The task management system WMS dispatches the stacking AGV to complete the cargo handling or loading and unloading tasks. This not only reduces the workload of manual loading and unloading, but also solves the problem of confusion between cargo stacking types. At the same time, cargo information can be obtained for unified management, and automated handling is realized, thereby improving handling efficiency.

[0010] A further technical solution is that the scanning device at least includes a laser sensor or a visual sensor.

[0011] A further technical solution is: the laser sensor includes a single-line laser radar, and the installation direction of the single-line laser radar is a vertical direction;

[0012] The step 1 comprises:

[0013] The AGV equipped with the scanning device is dispatched by the AGVS dispatching system to control the single-line laser radar to scan the vertical surface of the vehicle body space, and the AGV circles the truck compartment to complete the scanning of the entire environment of the compartment;

[0014] Combining the real-time position coordinates of the AGV itself and the installation position of the single-line laser radar relative to the vehicle body, the collected three-dimensional point cloud of the carriage is converted into point cloud information of the world coordinate system to reconstruct the three-dimensional scene of the carriage.

[0015] By vertically installing a single-line laser radar and using the AGV to drive the single-line laser radar around the car, data on the entire environment of the car can be collected. By combining the posture coordinates of the AGV and the installation posture of the single-line laser radar, the point cloud information of the car in the world coordinate system can be obtained, so as to facilitate reconstruction of the three-dimensional scene of the car.

[0016] A further technical solution is: the visual device comprises at least one camera, the camera is fixedly mounted outside the vehicle compartment, the horizontal field of view and the vertical field of view of the camera are determined according to vehicle specifications, and the scanning range of the camera completely covers the entire vehicle compartment;

[0017] The step 3 comprises:

[0018] The target object type detection module uses the yolov3 target detection deep learning method to train a training model for the cargo to be loaded and unloaded;

[0019] For the trained model, the image scanned by the camera is divided into S*S grids through the YOLOv3 target detection deep learning method, and the goods in each grid area are detected to determine the type of goods;

[0020] Use the bounding box to output the relative pixel difference between the center pixel of each identified item and the camera focus;

[0021] The actual relative coordinates of the cargo relative to the center point of the camera are calculated using the length, width and height information of the identified cargo, and the world coordinates of each cargo in the carriage are calculated based on the coordinates of the camera in the world coordinate system;

[0022] According to the point cloud data collected by the three-dimensional scene reconstruction module, cargo location recognition is performed based on the point cloud segmentation-clustering-model growth model;

[0023] The PCA principal component analysis method is used to obtain the three main directions of the point cloud, obtain the coordinates and covariance of the center of mass of the goods, correspond to the calculated world coordinates of the goods, and correspond the types of goods being transported to the coordinates;

[0024] After the type of cargo in the carriage is identified, the information of the identified cargo type is retrieved from a cargo type database to determine whether it can be stacked; the cargo type database is used to store the volume of various cargoes and information on whether the cargoes can be stacked;

[0025] The available free space in the carriage is calculated according to the cargo coordinates, and the coordinate information, free space information, and information on whether the cargo can be stacked are packaged and sent to the task management system WMS.

[0026] Through deep learning technology, the cargo images captured by the camera can be used to identify the type of cargo, determine the location of the cargo and the stacking information, so as to realize the automated handling of the cargo.

[0027] A further technical solution is: the target object type detection module uses the yolov3 target detection deep learning method to train a training model of the cargo to be loaded and unloaded, including:

[0028] When there are new types of goods, the target object type detection module is used to retrain the new goods.

[0029] In order to ensure the accuracy of loading and unloading of goods, new goods are trained in a timely manner to ensure the accuracy of deep learning technology in identification and to ensure that no type of goods is missed.

[0030] A further technical solution is: the detection of goods in each grid area includes:

[0031] The grid whose coordinates of the center position of the cargo fall within the grid area is determined as the target grid, and the corresponding cargo is detected through the target grid.

[0032] The recognized image area is divided into multiple grids, and the goods are detected through the target grid at the center of the goods in the grid area, avoiding repeated detection of goods, reducing the detection area, and improving detection efficiency.

[0033] A further technical solution is that: the central pixel point of the identified goods refers to the coordinates of the central position of the bounding box of the object predicted by the target grid.

[0034] Its further technical solution is: the step 4 comprises:

[0035] After the task management system WMS receives the information on whether the goods can be stacked, the coordinates of the goods, and the position coordinates of the goods that can be stacked in the carriage from the target object type detection module, it sends the information on the goods that need to be loaded and unloaded to the AGVS scheduling system according to the storage location information of the corresponding carriage;

[0036] After the AGVS dispatching system receives the storage location information of the corresponding carriage sent by the task management system WMS, the AGVS dispatching system plans the corresponding route and sends the task to the idle stacking AGV, and deploys the idle stacking AGV to complete the corresponding automatic loading and unloading task.

[0037] The AGVS dispatching system allocates idle stacking AGVs to complete loading and unloading tasks based on the acquired carriage location information, unifies the dispatching, reduces the manual workload, and ensures the accuracy of stacking.

[0038] The advantages of the present invention are:

[0039] The carriage scene is modeled through a system including control computing equipment, AGV equipped with scanning equipment, visual equipment, and stacking AGV. The type of cargo in the carriage is identified in combination with the target object type detection module, and the stacking information of the cargo is determined. The task management system WMS dispatches the stacking AGV to complete the cargo handling or loading and unloading tasks. This not only reduces the workload of manual loading and unloading, but also solves the problem of confusion between cargo stacking types. At the same time, cargo information can be obtained for unified management, and automated handling is realized, thereby improving handling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0041] Figure 1 This is a system structure diagram of the AGV automatic loading method provided by the present application;

[0042] Figure 2 It is a flow chart of the AGV automatic loading method provided by the present application;

[0043] Figure 3 It is a schematic diagram of truck parking provided by this application;

[0044] Figure 4 It is a schematic diagram of the installation position of the camera provided in this application. DETAILED DESCRIPTION

[0045] Embodiment: This application provides an AGV automatic loading method, which is applied to Figure 1 In the system of the control computing device, AGV equipped with a scanning device, visual equipment, and stacking AGV shown, the control computing device integrates an AGVS scheduling system, a three-dimensional scene reconstruction module, a target object type detection module, and a task management system WMS.

[0046] The control computing device is a device with data processing and signal receiving and sending functions, which can be a computer.

[0047] Optionally, the scanning device, the AGV equipped with the scanning device, the visual device, and the stacking AGV can achieve wired or wireless communication with the control computing device.

[0048] Optionally, the scanning device includes at least a laser sensor or a visual sensor.

[0049] Optionally, the laser sensor includes a single-line laser radar, and the installation direction of the single-line laser radar is a vertical direction.

[0050] Optionally, the visual device includes at least one camera, which is fixedly mounted outside the vehicle compartment, and the horizontal and vertical field of view of the camera are determined according to vehicle specifications, and the scanning range of the camera completely covers the entire vehicle compartment. Optionally, the camera is preferably an RGB camera.

[0051] The dispatching system dispatches the AGV equipped with a single-line laser head to circle the truck compartment, and sends the instruction to the 3D scene reconstruction module, which reconstructs the 3D scene of the compartment; after collecting the 3D scene graph, the AGVS dispatching system sends a stop collection instruction to the 3D scene reconstruction module, and then the 3D scene reconstruction module transmits the point cloud data cluster to the target object type detection module, which identifies the type of cargo in the compartment, determines whether it can be stacked, and calculates the cargo coordinates and the available free area in the compartment, and packages this information and sends it to the task management system WMS; after receiving the cargo information in the compartment, the task management system WMS assigns cargo handling or loading and unloading tasks to the stacking AGV and assigns it to complete the corresponding task.

[0052] like Figure 2As shown, the AGV automatic loading method includes the following steps.

[0053] Step 1: The AGV equipped with the scanning device is dispatched by the AGVS dispatching system to circle the truck compartment, and a command is sent to the 3D scene reconstruction module to trigger the 3D scene reconstruction module to reconstruct the 3D scene of the compartment.

[0054] For an AGV equipped with a single-line laser radar, step 1 includes the following steps.

[0055] In the first step, the AGV equipped with the scanning device is dispatched by the AGVS dispatching system to control the single-line laser radar to scan the vertical surface of the vehicle body space, and the AGV circles the truck compartment to complete the scanning of the entire environment of the compartment.

[0056] Combining the real-time position coordinates of the AGV itself and the installation position of the single-line laser radar relative to the vehicle body, the collected three-dimensional point cloud of the cabin is converted into point cloud information of the world coordinate system to reconstruct the three-dimensional scene of the cabin.

[0057] Step 2: After the three-dimensional scene graph is collected, a stop collection instruction is sent to the three-dimensional scene reconstruction module through the AGVS scheduling system, and the point cloud data is transmitted to the target object type detection module through the three-dimensional scene reconstruction module.

[0058] Step 3: Identify the type of cargo in the carriage through the target object type detection module, determine whether it can be stacked, and calculate the stacking information. The target object type detection module will package the stacking information and send it to the task management system WMS.

[0059] The stacking information includes at least the calculated coordinates of the cargo, the available free space in the carriage, and information on whether the cargo can be stacked.

[0060] Step 3 includes the following steps.

[0061] The first step is to use the yolov3 target detection deep learning method through the target object type detection module to train a training model for the cargo to be loaded and unloaded.

[0062] The sample information of the collected goods is marked in advance. When the goods are identified through the YOLOv3 target detection deep learning method, the identified goods are compared with the pre-collected samples. When the pre-set similarity is reached, matching can be performed.

[0063] In actual applications, when there are new types of goods, the new goods are re-trained through the target object type detection module.

[0064] In the second step, for the trained model, the image scanned by the camera is divided into S*S grids through the yolov3 target detection deep learning method, the goods in each grid area are detected, and the type of goods is determined.

[0065] The detection of the goods in each grid area includes: determining the grid with the coordinates of the center position of the goods falling into the grid area as the target grid, and detecting the corresponding goods through the target grid.

[0066] The third step is to use the bounding box to output the relative pixel difference between the center pixel of each identified item and the camera focus.

[0067] The center pixel point of the identified goods refers to the coordinates of the center position of the bounding box of the object predicted by the target grid.

[0068] The fourth step is to use the length, width and height information of the identified goods to calculate the actual relative coordinates of the goods relative to the center point of the camera, and calculate the world coordinates of each item in the carriage based on the coordinates of the camera in the world coordinate system.

[0069] Since the image scanned by the recognition camera is two-dimensional information, the accuracy of the required world coordinates and the actual coordinates is poor. In order to accurately determine the specific coordinates of the objects to be transported in the vehicle, this application adds a method for calculating the cargo position coordinates, which is specifically implemented in steps 5 to 6.

[0070] In the fifth step, based on the point cloud data collected by the 3D scene reconstruction module, cargo location identification is performed based on the point cloud segmentation-clustering-model growth model.

[0071] In the sixth step, the PCA principal component analysis method is used to obtain the three main directions of the point cloud, obtain the center of mass coordinates and covariance of the goods, correspond to the calculated world coordinates of the goods, and correspond the type of the transported goods to the coordinates.

[0072] The seventh step is to retrieve the information of the identified cargo type from the cargo type database after the cargo type in the carriage is identified to determine whether it can be stacked.

[0073] The type of cargo in the carriage can be identified through the trained model.

[0074] The cargo type database is used to store information about the volume of various cargoes and whether the cargoes can be stacked.

[0075] The eighth step is to calculate the available free space in the carriage according to the cargo coordinates, and to package the coordinate information, free space information, and information on whether the cargo can be stacked and send them to the task management system WMS.

[0076] It should be noted that the target object type detection module needs to provide as many cargo samples as possible for deep learning training models. The more training samples there are, the higher the accuracy of cargo identification and the more accurate the size of cargo encircled by the bounding box. When there are new types of cargo, the target object type detection module needs to be informed in advance to retrain the new cargo.

[0077] Step 4: After the task management system WMS receives the stacking information, it allocates the cargo handling or loading and unloading tasks to the stacking AGV, and assigns the stacking AGV to complete the corresponding cargo handling or loading and unloading tasks.

[0078] Step 4 includes the following steps:

[0079] In the first step, after the task management system WMS receives the information on whether the goods can be stacked, the coordinates of the goods, and the coordinates of the position of the goods that can be stacked in the carriage from the target object type detection module, it sends the information on the goods that need to be loaded and unloaded to the AGVS scheduling system according to the storage location information of the corresponding carriage;

[0080] In the second step, after the AGVS dispatching system receives the storage location information of the corresponding carriage sent by the task management system WMS, the AGVS dispatching system plans the corresponding route and sends the task to the idle stacking AGV, deploying the idle stacking AGV to complete the corresponding automatic loading and unloading tasks.

[0081] Exemplary, with reference to Figure 3 In actual application, a large truck with a container is parked in the frame area. A parking block is set at the rear wheel position of the truck parking space. The truck needs to reverse along the center line as much as possible into the designated position. The error is controlled within 2cm. The error is reflected between the rear wheel and the block. The left and right deviations during parking need to be controlled within 0.5m. After the truck is parked, the compartment wings on both sides of the container are opened. The management system is notified by manual RFID, and the truck loading and unloading is ready. Then the AGVS dispatching system starts to dispatch the AGV equipped with scanning equipment to collect compartment information and subsequent modeling.

[0082] Combined with reference Figure 4 , cameras are placed on both sides of the truck parking space, and the camera's shooting range covers the entire vehicle compartment. For example, Figure 4 Four cameras are shown in the figure, with two cameras placed on each side to collect images. In actual applications, the number of cameras can be increased or decreased and the layout of the cameras can be adjusted according to the specifications of the truck.

[0083] To summarize, the AGV automatic loading method provided in the present application models the carriage scene through a system including a control computing device, an AGV equipped with a scanning device, a visual device, and a stacking AGV. The target object type detection module is used to identify the type of cargo in the carriage, and the stacking information of the cargo is determined. The task management system WMS schedules the stacking AGV to complete the cargo handling or loading and unloading tasks, which not only reduces the workload of manual loading and unloading, but also solves the problem of confusion in the types of cargo stacking. At the same time, cargo information can be obtained for unified management, and automated handling is realized, thereby improving handling efficiency.

[0084] In addition, by vertically installing a single-line laser radar and using the AGV to drive the single-line laser radar around the car, data on the entire environment of the car can be collected. Combining the position coordinates of the AGV and the installation position of the single-line laser radar, the point cloud information of the car in the world coordinate system can be obtained, so as to facilitate reconstruction of the three-dimensional scene of the car.

[0085] In addition, through deep learning technology, the cargo images captured by the camera can be used to identify the type of cargo, determine the location of the cargo, and stacking information, so as to realize the automated handling of cargo.

[0086] In addition, in order to ensure the accuracy of loading and unloading of goods, new goods are trained in a timely manner to ensure the accuracy of deep learning technology in identification and to ensure that no type of goods is missed.

[0087] In addition, the recognized image area is divided into multiple grids, and the goods are detected through the target grid at the center of the goods in the grid area, so as to avoid repeated detection of goods, reduce the detection area, and improve the detection efficiency.

[0088] In addition, the AGVS dispatching system can allocate idle stacking AGVs to complete loading and unloading tasks based on the acquired carriage storage location information, unify the dispatching, reduce the manual workload, and ensure the accuracy of stacking.

[0089] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implying the number of technical features indicated. Thus, the defined "first" and "second" features may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0090] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0091] A person of ordinary skill in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0092] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An AGV automatic loading method, characterized in that: Applied to a system including a control computing device, an AGV equipped with a scanning device, a visual device, and a stacking AGV, wherein the control computing device integrates an AGVS scheduling system, a three-dimensional scene reconstruction module, a target object type detection module, and a task management system WMS. The AGV automatic loading method includes: Step 1, dispatching the AGV equipped with a scanning device to circle the truck compartment through the AGVS dispatching system, issuing instructions to the 3D scene reconstruction module, and triggering the 3D scene reconstruction module to reconstruct the 3D scene of the compartment; Step 2, after the three-dimensional scene graph is collected, a stop collection instruction is sent to the three-dimensional scene reconstruction module through the AGVS scheduling system, and the point cloud data is transmitted to the target object type detection module through the three-dimensional scene reconstruction module; Step 3, identifying the type of goods in the carriage through the target object type detection module, determining whether they can be stacked, and calculating stacking information, the stacking information at least including the calculated coordinates of the goods, the available free area in the carriage, and information on whether the goods can be stacked, and packaging the stacking information through the target object type detection module and sending it to the task management system WMS; Step 4, after the task management system WMS receives the stacking information, it allocates cargo handling or loading and unloading tasks to the stacking AGV, and assigns the stacking AGV to complete the corresponding cargo handling or loading and unloading tasks; The step 1 comprises: The AGVS dispatching system dispatches the scanning device to scan the vertical surface of the vehicle body space, and the AGV circles the truck compartment to complete the scanning of the entire environment of the compartment; Combining the real-time position coordinates of the AGV itself and the installation position of the scanning device relative to the vehicle body, the collected three-dimensional point cloud of the carriage is converted into point cloud information of the world coordinate system to reconstruct the three-dimensional scene of the carriage; The step 3 comprises: The target object type detection module uses the yolov3 target detection deep learning method to train a training model for the cargo to be loaded and unloaded; For the trained model, the image scanned by the scanning device is divided into S*S grids through the YOLOv3 target detection deep learning method, and the goods in each grid area are detected to determine the type of goods; Use the bounding box to output the relative pixel difference between the center pixel of each identified item and the focus of the scanning device; The actual relative coordinates of the cargo relative to the center point of the scanning device are calculated using the length, width and height information of the identified cargo, and the world coordinates of each cargo in the carriage are calculated based on the coordinates of the scanning device in the world coordinate system; According to the point cloud data collected by the three-dimensional scene reconstruction module, cargo location recognition is performed based on the point cloud segmentation-clustering-model growth model; The PCA principal component analysis method is used to obtain the three main directions of the point cloud, obtain the coordinates and covariance of the center of mass of the goods, correspond to the calculated world coordinates of the goods, and correspond the types of goods being transported to the coordinates; After the type of cargo in the carriage is identified, the information of the identified cargo type is retrieved from a cargo type database to determine whether it can be stacked; the cargo type database is used to store the volume of various cargoes and information on whether the cargoes can be stacked; The available free space in the carriage is calculated according to the cargo coordinates, and the coordinate information, free space information, and information on whether the cargo can be stacked are packaged and sent to the task management system WMS.

2. The AGV automatic loading method according to claim 1, characterized in that: The scanning device at least includes a laser sensor or a visual sensor.

3. The AGV automatic loading method according to claim 2, characterized in that: The laser sensor includes a single-line laser radar, and the installation direction of the single-line laser radar is a vertical direction.

4. The AGV automatic loading method according to claim 2, characterized in that: The visual device includes at least one camera, which is fixedly mounted on the outside of the vehicle compartment. The horizontal field of view and the vertical field of view of the camera are determined according to vehicle specifications, and the scanning range of the camera completely covers the entire vehicle compartment.

5. The AGV automatic loading method according to claim 1, characterized in that: The target object type detection module uses the yolov3 target detection deep learning method to train a training model for the cargo to be loaded and unloaded, including: When there are new types of goods, the target object type detection module is used to retrain the new goods.

6. The AGV automatic loading method according to claim 1, characterized in that: The detecting of the goods in each grid area includes: The grid whose coordinates of the center position of the cargo fall within the grid area is determined as the target grid, and the corresponding cargo is detected through the target grid.

7. The AGV automatic loading method according to claim 6, characterized in that: The central pixel point of the identified goods refers to the coordinates of the center position of the bounding box of the object predicted by the target grid.

8. The AGV automatic loading method according to any one of claims 1 to 7, characterized in that: The step 4 comprises: After the task management system WMS receives the information on whether the goods can be stacked, the coordinates of the goods, and the position coordinates of the goods that can be stacked in the carriage from the target object type detection module, it sends the information on the goods that need to be loaded and unloaded to the AGVS scheduling system according to the storage location information of the corresponding carriage; After the AGVS dispatching system receives the storage location information of the corresponding carriage sent by the task management system WMS, the AGVS dispatching system plans the corresponding route and sends the task to the idle stacking AGV, and deploys the idle stacking AGV to complete the corresponding automatic loading and unloading task.

Images