A shelf detection and positioning method based on laser radar multi-frame splicing

Abstract

The present invention belongs to the technical field of large-scale shelf detection and positioning, specifically a shelf detection and positioning method based on laser radar multi-frame splicing, comprising the following steps: 1) continuously acquiring the laser point cloud of the space where the AGV is located and collecting IMU data; 2) using the lidar-imu algorithm to realize splicing of the multi-frame point cloud to obtain the overall point cloud of the shelf; 3) extracting the target cargo space shelf legs according to the acquired overall point cloud of the shelf, detecting the cargo situation of the target adjacent cargo space, calculating the target shelf center position and posture, and obtaining the target cargo space; 4) processing the information data of the target cargo space, calculating the actual space of the target shelf, and correcting the target cargo space. The present invention effectively obtains complete shelf information, makes shelf positioning more accurate, and can effectively judge the information of the shelf, judge the cargo status of the target cargo space and the adjacent cargo space, and improve the safety and flexibility of placing goods.

Description

Technical Field

[0001] The invention belongs to the technical field of large-scale shelf detection and positioning, and specifically is a shelf detection and positioning method based on laser radar multi-frame splicing. Background Art

[0002] In today's retail and warehousing industries, large shelf detection and positioning technology plays a key role. This technology uses computer vision, deep learning and sensor technology to achieve efficient and accurate detection and positioning of large shelves and the goods on them. This technology is widely used in automated warehousing systems, intelligent management of the retail industry and logistics, providing companies with more efficient inventory management, cargo tracking and order processing functions.

[0003] Shelf detection and positioning usually require a lot of manpower and time, are prone to errors, and are inefficient. However, large shelf detection technology based on computer vision and deep learning can achieve real-time monitoring and positioning of goods on shelves by installing cameras and sensors on the shelves and combining advanced image processing algorithms, greatly improving the speed and accuracy of goods recognition. Secondly, in the traditional AGV method of detecting large shelves, single-frame data is used for detection. Due to the limitations of the field of view and the installation position of the equipment, for large shelves, the use of single-frame data to detect shelves will encounter incomplete data information. Summary of the invention

[0004] The purpose of the present invention is to provide a shelf detection and positioning method based on multi-frame splicing of laser radar, solve the problem of AGV detecting large shelves, adopt a multi-frame point cloud splicing and mapping method based on lidar-imu algorithm, when the equipment goes to the shelf, continuous multi-frame splicing and mapping can effectively obtain complete shelf information, make shelf positioning more accurate, and effectively judge the shelf information, judge the cargo status of the target cargo location and the adjacent cargo location, and improve the safety and flexibility of placing goods.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a shelf detection and positioning method based on laser radar multi-frame splicing, comprising the following steps:

[0006] 1) Continuously obtain the laser point cloud of the space where the AGV is located and collect IMU data;

[0007] 2) Use lidar-imu algorithm to stitch multiple frame point clouds and obtain the overall point cloud of the shelf;

[0008] 3) Based on the overall point cloud of the shelf, extract the legs of the target shelf, detect the goods in the target adjacent shelves, calculate the center position and posture of the target shelf, and obtain the target shelf space;

[0009] 4) Process the information data of the target storage space, calculate the actual space of the target shelf, and correct the target storage space.

[0010] The step 1) comprises the following steps:

[0011] 1-1) Install a laser sensor on the AGV, wherein the laser sensor is a laser radar; scan the surrounding environment of the AGV by the laser radar to generate laser point cloud data, where each frame of point cloud represents environmental information at different times or locations; process the laser point cloud data through a point cloud library to obtain laser point cloud information;

[0012] 1-2) Install the IMU sensor on the AGV, and use the IMU sensor to measure the acceleration, angular velocity and direction information of the AGV; use the Kalman filter to fuse the accelerometer and gyroscope data to obtain the imu data;

[0013] 1-3) Set the starting and ending points of data collection. When the AGV reaches the starting point, it starts collecting laser point cloud and imu data. When it reaches the ending point, it stops collecting.

[0014] In step 1-3), the starting point and the ending point of data collection are set as follows:

[0015] According to the actual scenario, the four shelf legs and the middle space of the target shelf layer are covered, and the continuous frame data are processed to remove the invalid points and the interference points of the vehicle body in each frame.

[0016] The step 2) comprises the following steps:

[0017] 2-1) performing preprocessing operations including denoising, filtering, and coordinate transformation on the multi-frame point cloud data and IMU data obtained in step 1) to obtain preprocessed data;

[0018] 2-2) Point cloud registration: Use IMU data to perform temporal or spatial alignment on multiple point clouds to eliminate relative posture changes between different frames;

[0019] 2-3) Point cloud stitching: Based on the registered multi-frame point cloud data, the ICP closest point matching method or the NDT normal distribution transformation method is used to set the multi-frame point cloud data at a certain moment as the target shelf area, and the multi-frame point cloud data at that moment are fused into a map of the target shelf area;

[0020] 2-4) Use the IMU and lidar as the front end to build a laser inertial odometer through the lidar-imu algorithm, and obtain the dedistorted point cloud frame, that is, generate complete shelf point cloud data.

[0021] The step 3) is specifically:

[0022] 3-1) De-noise and downsample the generated complete shelf point cloud data, and obtain the coordinates of the target delivery point through the relative position relationship after the AGV reaches the fixed detection position;

[0023] 3-2) Preliminary calculation of the delivery status of the target location;

[0024] 3-3) According to the placement of goods at the target location, the columns are segmented by normal vectors, and then the non-column points are segmented by RANSAC fitting multi-line segmentation. Based on the segmentation results, the horizontal columns and columns of the shelf are obtained by normal vectors;

[0025] 3-4) According to the corresponding position relationship of the shelves, calculate the center position of the goods and the center position of the goods in the adjacent area.

[0026] The step 3-2) is specifically:

[0027] The spatial progressive search method is adopted. According to the preliminary calculated coordinates of the target delivery point, the initial space is set, the point cloud distribution quantity status in the space is detected, the space size is expanded to the set threshold, and the relationship between the searched space and the actual cargo size is compared.

[0028] The steps 3-4) are specifically:

[0029] According to the horizontal and vertical columns of the shelf, the point cloud is corrected by the normal vector, and the rotation matrix R is calculated. The intersection position is calculated by the linear equation of the horizontal and vertical columns, and the translation matrix t can be calculated by the cargo size and cargo room width.

[0030] According to the point cloud rotation formula, the center position of the converted shelf where the goods are placed is:

[0031] [x ′ ,y ′ ,z ′ ] T =R*[x,y,z] T +t

[0032] Among them, R is the rotation matrix and t is the translation matrix.

[0033] The step 4) is specifically:

[0034] 4-1) According to the current cargo center position and the cargo center of the neighboring area, the spatial progressive search method is used again to perform spatial search. According to the calculated center position of the shelf where the cargo is placed, the initial space is set, the point cloud distribution quantity state in the space is detected, the space size is expanded to the edges of the horizontal and vertical columns of the shelf, the relationship between the searched space and the actual cargo size is compared, and the actual space of the target shelf is calculated;

[0035] 4-2) According to the actual space of the target shelf, determine whether the current goods can be placed, and compare the actual size of the goods with the size of the searched space to obtain the center position of the goods and complete the shelf positioning.

[0036] A shelf detection and positioning system based on laser radar multi-frame splicing, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement a shelf detection and positioning method based on laser radar multi-frame splicing as described when executing the computer program.

[0037] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, a shelf detection and positioning method based on laser radar multi-frame splicing is implemented.

[0038] The present invention has the following beneficial effects and advantages: the present invention adopts a multi-frame point cloud stitching and mapping method based on the lidar-imu algorithm. When the equipment goes to the shelf, continuous multi-frame stitching and mapping can effectively obtain complete shelf information, make shelf positioning more accurate, and effectively judge the shelf information, judge the cargo status of the target cargo location and the adjacent cargo location, and improve the safety and flexibility of placing goods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A method flow chart of a shelf detection and positioning method based on laser radar multi-frame splicing of the present invention;

[0040] Figure 2 Flow chart of the method for shelf detection in step 3) of the present invention;

[0041] Figure 3 Flow chart of the method for shelf positioning in step 4) of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, it is a method flow chart of a shelf detection and positioning method based on laser radar multi-frame splicing of the present invention. The shelf detection and positioning method based on laser radar multi-frame splicing of the present invention comprises the following steps:

[0044] 1) Continuously obtain the laser point cloud of the space where the AGV is located and collect IMU data;

[0045] 1-1) Install a laser sensor on the AGV, wherein the laser sensor is a laser radar; scan the surrounding environment of the AGV by the laser radar to generate laser point cloud data, where each frame of point cloud represents environmental information at different times or locations; process the laser point cloud data through a point cloud library to obtain laser point cloud information;

[0046] 1-2) Install the IMU sensor on the AGV, and use the IMU sensor to measure the acceleration, angular velocity and direction information of the AGV; use the Kalman filter to fuse the accelerometer and gyroscope data to obtain the imu data;

[0047] 1-3) Set the starting and ending points of data collection. When the AGV reaches the starting point, it starts collecting laser point cloud and imu data. When it reaches the ending point, it stops collecting.

[0048] Set the start and end points of data collection, specifically:

[0049] According to the actual scenario, the four shelf legs and the middle space of the target shelf layer are covered, and the continuous frame data are processed to remove the invalid points and the interference points of the vehicle body in each frame.

[0050] 2) Use lidar-imu algorithm to stitch multiple frame point clouds and obtain the overall point cloud of the shelf;

[0051] 2-1) performing preprocessing operations including denoising, filtering, and coordinate transformation on the multi-frame point cloud data and IMU data obtained in step 1) to obtain preprocessed data;

[0052] 2-2) Point cloud registration: Use IMU data to perform temporal or spatial alignment on multiple point clouds to eliminate relative posture changes between different frames;

[0053] 2-3) Point cloud stitching: Based on the registered multi-frame point cloud data, the ICP closest point matching method or the NDT normal distribution transformation method is used to set the multi-frame point cloud data at a certain moment as the target shelf area, and the multi-frame point cloud data at that moment are fused into a map of the target shelf area;

[0054] 2-4) Use the IMU and lidar as the front end to build a laser inertial odometer through the lidar-imu algorithm, and obtain the dedistorted point cloud frame, that is, generate complete shelf point cloud data.

[0055] 3) If Figure 2 As shown, it is a flow chart of the method for shelf detection in step 3) of the present invention, according to obtaining the overall point cloud of the shelf, extracting the shelf legs of the target cargo space, detecting the cargo situation of the target adjacent cargo space, calculating the center position and posture of the target shelf, and obtaining the target cargo space;

[0056] 3-1) De-noise and downsample the generated complete shelf point cloud data, and obtain the coordinates of the target delivery point through the relative position relationship after the AGV reaches the fixed detection position;

[0057] 3-2) Preliminary calculation of the delivery status of the target location;

[0058] The spatial progressive search method is adopted. According to the preliminary calculated coordinates of the target delivery point, the initial space is set, the point cloud distribution quantity status in the space is detected, the space size is expanded to the set threshold, and the relationship between the searched space and the actual cargo size is compared.

[0059] 3-3) According to the placement of goods at the target location, the columns are segmented by normal vectors, and then the non-column points are segmented by RANSAC fitting multi-line segmentation. Based on the segmentation results, the horizontal columns and columns of the shelf are obtained by normal vectors;

[0060] 3-4) According to the corresponding position relationship of the shelves, calculate the center position of the goods and the center position of the goods in the adjacent area.

[0061] According to the horizontal and vertical columns of the shelf, the point cloud is corrected by the normal vector, and the rotation matrix R is calculated. The intersection position is calculated by the linear equation of the horizontal and vertical columns, and the translation matrix t can be calculated by the cargo size and cargo room width.

[0062] According to the point cloud rotation formula, the center position of the converted shelf where the goods are placed is:

[0063] [x ′ ,y ′ ,z ′ ] T =R*[x,y,z] T +t

[0064] Among them, R is the rotation matrix and t is the translation matrix.

[0065] 4) Process the information data of the target storage space, calculate the actual space of the target shelf, and correct the target storage space.

[0066] 4-1) According to the current cargo center position and the cargo center of the neighboring area, the spatial progressive search method is used again to perform spatial search. According to the calculated center position of the shelf where the cargo is placed, the initial space is set, the point cloud distribution quantity state in the space is detected, the space size is expanded to the edges of the horizontal and vertical columns of the shelf, the relationship between the searched space and the actual cargo size is compared, and the actual space of the target shelf is calculated;

[0067] 4-2) According to the actual space of the target shelf, determine whether the current goods can be placed, and compare the actual size of the goods with the size of the searched space to obtain the center position of the goods and complete the shelf positioning.

[0068] Embodiment 1:

[0069] Step 1: Continuously acquire the laser point cloud and imu data of the space where the AGV is located: set the starting and ending points of data collection. After the AGV reaches the starting point, it starts to collect laser point cloud and imu data. After reaching the ending point, it stops collecting. The selection of the starting and ending points needs to cover the four shelf legs and the middle space of the target shelf layer according to the actual scene, and process the continuous frame data to remove the invalid points and the interference points of the vehicle body in each frame.

[0070] Step 2: Use the lidar-imu algorithm to stitch multiple frame point clouds to obtain the overall point cloud of the shelf: set the target shelf area at a certain moment, collect data through sensors, design specific values ​​based on the data to describe the scene to form a map of the target shelf area, use the IMU and multi-line laser radar as the front end through the lidar-imu algorithm to build a laser inertial odometer, and obtain the dedistorted point cloud frame to generate complete shelf point cloud data.

[0071] Step 3: Extract the target shelf information, detect the target and adjacent shelves, and determine the target shelf space: process the overall point cloud of the shelf, perform space detection based on the approximate target placement point, preliminarily calculate the placement of the target location, segment the columns through normal vectors, and then perform RANSAC fitting multi-line segmentation on non-column points. For the segmentation results, use the normal vector to obtain the horizontal and vertical columns of the shelf, and calculate the center position of the goods and the center position of the adjacent area goods based on the corresponding position relationship. Figure 2 As shown;

[0072] Step 4: Use shelf information data processing to calculate the actual space of the target shelf and correct the target cargo location: perform space search through the current cargo center location and the adjacent cargo center, calculate the cargo space, determine whether the current cargo can be placed, and compare the actual cargo size with the searched space size to reasonably give the cargo center location. Figure 3 As shown;

[0073] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0074] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A shelf detection and positioning method based on laser radar multi-frame splicing, characterized in that: The following steps are involved: 1) Continuously obtain the laser point cloud of the space where the AGV is located and collect IMU data; 2) Use lidar-imu algorithm to stitch multiple frame point clouds and obtain the overall point cloud of the shelf; 3) Based on the overall point cloud of the shelf, extract the target shelf legs, detect the goods in the target adjacent shelves, calculate the center position and posture of the target shelf, and obtain the target shelf space; 4) Process the information data of the target storage space, calculate the actual space of the target shelf, and correct the target storage space.

2. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 1 is characterized in that: The step 1) comprises the following steps: 1-1) Install a laser sensor on the AGV, wherein the laser sensor is a laser radar; scan the surrounding environment of the AGV by the laser radar to generate laser point cloud data, where each frame of point cloud represents environmental information at different times or locations; process the laser point cloud data through a point cloud library to obtain laser point cloud information; 1-2) Install the IMU sensor on the AGV, and use the IMU sensor to measure the acceleration, angular velocity and direction information of the AGV; use the Kalman filter to fuse the accelerometer and gyroscope data to obtain the imu data; 1-3) Set the starting and ending points of data collection. When the AGV reaches the starting point, it starts collecting laser point cloud and imu data. When it reaches the ending point, it stops collecting.

3. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 2 is characterized in that: In step 1-3), the starting point and the ending point of data collection are set as follows: According to the actual scenario, the four shelf legs and the middle space of the target shelf layer are covered, and the continuous frame data are processed to remove the invalid points and the interference points of the vehicle body in each frame.

4. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 1 is characterized in that: The step 2) comprises the following steps: 2-1) performing preprocessing operations including denoising, filtering, and coordinate transformation on the multi-frame point cloud data and IMU data obtained in step 1) to obtain preprocessed data; 2-2) Point cloud registration: Use IMU data to perform temporal or spatial alignment on multiple point clouds to eliminate relative posture changes between different frames; 2-3) Point cloud stitching: Based on the registered multi-frame point cloud data, the ICP closest point matching method or the NDT normal distribution transformation method is used to set the multi-frame point cloud data at a certain moment as the target shelf area, and the multi-frame point cloud data at that moment are fused into a map of the target shelf area; 2-4) Use the IMU and lidar as the front end to build a laser inertial odometer through the lidar-imu algorithm, and obtain the dedistorted point cloud frame, that is, generate complete shelf point cloud data.

5. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 1 is characterized in that: The step 3) is specifically: 3-1) De-noise and downsample the generated complete shelf point cloud data, and obtain the coordinates of the target delivery point through the relative position relationship after the AGV reaches the fixed detection position; 3-2) Preliminary calculation of the delivery status of the target location; 3-3) According to the placement of goods at the target location, the columns are segmented by normal vectors, and then the non-column points are segmented by RANSAC fitting multi-line segmentation. Based on the segmentation results, the horizontal columns and columns of the shelf are obtained by normal vectors; 3-4) According to the corresponding position relationship of the shelves, calculate the center position of the goods and the center position of the goods in the adjacent area.

6. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 5 is characterized in that: The step 3-2) is specifically: The spatial progressive search method is adopted. According to the preliminary calculated coordinates of the target delivery point, the initial space is set, the point cloud distribution quantity status in the space is detected, the space size is expanded to the set threshold, and the relationship between the searched space and the actual cargo size is compared.

7. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 5 is characterized in that: The steps 3-4) are specifically: According to the horizontal and vertical columns of the shelf, the point cloud is corrected by the normal vector, and the rotation matrix R is calculated. The intersection position is calculated by the linear equation of the horizontal and vertical columns, and the translation matrix t can be calculated by the cargo size and cargo room width. According to the point cloud rotation formula, the center position of the converted shelf where the goods are placed is: [x ′ ,y ′ ,z ′ ] T =R*[x,y,z] T +t Among them, R is the rotation matrix and t is the translation matrix.

8. The shelf detection and positioning method based on laser radar multi-frame splicing according to claim 1 is characterized in that: The step 4) is specifically: 4-1) According to the current cargo center position and the cargo center of the neighboring area, the spatial progressive search method is used again to perform spatial search. According to the calculated center position of the shelf where the cargo is placed, the initial space is set, the point cloud distribution quantity state in the space is detected, the space size is expanded to the edges of the horizontal and vertical columns of the shelf, the relationship between the searched space and the actual cargo size is compared, and the actual space of the target shelf is calculated; 4-2) According to the actual space of the target shelf, determine whether the current goods can be placed, and compare the actual size of the goods with the size of the searched space to obtain the center position of the goods and complete the shelf positioning.

9. A shelf detection and positioning system based on laser radar multi-frame splicing, characterized in that: It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement a shelf detection and positioning method based on laser radar multi-frame splicing as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, a shelf detection and positioning method based on laser radar multi-frame splicing as described in any one of claims 1 to 8 is implemented.

Images