A method for pile positioning in a compact space based on multi-sensor fusion

Abstract

The application discloses a pallet positioning method based on multi-sensor fusion in a compact space, and a composite robot is arranged, wherein the composite robot comprises an AGV trolley and a pallet robot carried on the AGV trolley; a rotary table and an RGB camera equipped with a wide-angle lens are arranged at the front end of the AGV trolley respectively, and a laser radar is fixed on the rotary table; the pallet positioning method comprises the following steps: performing calibration processing on the rotary table, the RGB camera, the laser radar and the pallet robot respectively; and performing positioning processing on a pallet based on the calibrated rotary table, RGB camera, laser radar and pallet robot through the AGV trolley, so that the pallet robot can work. The application combines the laser radar and the RGB camera, fuses the image information of the pallet in the compact space, and guides the AGV to move into the compact space to work.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion, and more specifically to a method for positioning a stack of materials in a compact space based on multi-sensor fusion. Background Technology

[0002] Positioning materials within a compact space refers to accurately locating the stacked materials within a compact space to facilitate subsequent destacking or palletizing operations. In logistics transportation, to improve loading and unloading efficiency and reduce human error rates, it is necessary to position materials within a compact space. This ensures that upon arrival at the destination, materials requiring destacking or palletizing can be located quickly and accurately.

[0003] Currently, the main methods for locating material stacks within compact spaces are 3D structured light cameras and 3D Time-of-Flight (TOF) cameras. These methods acquire images from multiple angles to obtain RGBD images of the space. However, these methods have some drawbacks. First, the field of view of a 3D structured light camera is relatively small, insufficient to cover the entire compact space. To address this, two or more cameras are typically required, each rotating along a yaw angle to acquire images from multiple angles. This results in a complex and bulky camera mounting mechanism, and may pose a risk of interference with surrounding equipment. Second, while the field of view of a 3D TOF camera is slightly better than that of a 3D structured light camera, it still cannot cover the entire compact space with a single camera, and the positioning accuracy of TOF cameras is relatively poor.

[0004] Therefore, current methods for positioning material stacks within compact spaces cannot provide a mature and reliable automated solution for stacking and destacking within compact spaces. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, a method for positioning material stacks based on multi-sensor fusion in a compact space is provided.

[0006] The specific technical solution is as follows: A material stack positioning method based on multi-sensor fusion in a compact space, wherein a composite robot is provided, the composite robot including an AGV trolley and a material stacking robot mounted on the AGV trolley;

[0007] The front end of the AGV is equipped with a turntable and an RGB camera with a wide-angle lens, and a LiDAR is fixed on the turntable.

[0008] The stack positioning method includes:

[0009] Calibration processes are performed on the turntable, the RGB camera, the lidar, and the stacking robot respectively.

[0010] as well as

[0011] The AGV (Automated Guided Vehicle) uses the calibrated turntable, RGB camera, LiDAR, and stacking robot to locate the stack, enabling the stacking robot to perform its operations.

[0012] Preferably, the calibration process includes a first calibration process that calibrates the lidar based on the relationship between the lidar and the turntable;

[0013] Before performing the first calibration process, a first calibration plate is set up in advance within the scanning range in front of the lidar;

[0014] The first calibration process includes:

[0015] Step A1: During the process of rotating the turntable to a preset angle, randomly collect multiple rotation angles of the turntable and record them as the first rotation angle;

[0016] as well as

[0017] The first radar data obtained by the lidar scanning the first calibration plate at each of the first rotation angles is acquired respectively;

[0018] Step A2: Preprocess multiple sets of the first radar data to obtain multiple first calibration board features;

[0019] Step A3: Based on multiple features of the first calibration board, obtain multiple actual poses of the lidar during the scanning process using the inter-frame matching method;

[0020] Step A4: Establish a first calibration model based on each actual pose and the corresponding first rotation angle, and perform calibration processing on the lidar based on the first calibration model.

[0021] Preferably, the calibration process includes a second calibration process that calibrates the RGB camera based on the relationship between the lidar and the RGB camera;

[0022] Before performing the second calibration process, a second calibration board is set up in advance within the field of view of the RGB camera;

[0023] The second calibration process includes:

[0024] Step B1: Simultaneously control the lidar and the turntable to rotate 180 degrees, so that the lidar scans multiple laser point cloud data of the second calibration board during the rotation and fuses them to obtain the second radar data;

[0025] Step B2: Preprocess the second radar data to obtain the second calibration board features;

[0026] Step B3: Acquire a first image of the second calibration board using the RGB camera, and extract the features of the third calibration board based on the first image;

[0027] Step B4: Move the second calibration board and return to step B1. Repeat steps B1-B4 multiple times to obtain multiple second calibration board features and multiple third calibration board features. Take a single second calibration board feature and the corresponding third calibration board feature as a group of first-class calibration board features.

[0028] Step B5: Based on multiple sets of features of the first type of calibration board and the corresponding first image, establish a second calibration model, and calculate the first positional relationship matrix between the lidar and the RGB camera based on the second calibration model;

[0029] Step B6: Perform calibration processing on the RGB camera according to the first position relationship matrix.

[0030] Preferably, the calibration process includes a third calibration process that calibrates the lidar based on the relationship between the stacking robot and the lidar;

[0031] Before performing the third calibration process, a third calibration plate is pre-installed on the end of the robotic arm of the stacking robot, and the stacking robot is moved to the first preset calibration position;

[0032] The third calibration process includes:

[0033] Step C1: Simultaneously control the lidar and the turntable to rotate 180 degrees, obtain multiple laser point cloud data of the lidar scanning the third calibration board during the rotation process, and fuse them to obtain the third lidar data;

[0034] Step C2: Preprocess the third radar data to obtain the fourth calibration board features;

[0035] Step C3: After repeatedly changing the pose of the stacking robot and cyclically executing steps C1-C3, multiple features of the fourth calibration plate are obtained;

[0036] Step C4: Construct a third calibration model based on multiple features of the fourth calibration plate and the position and posture of the stacking robot corresponding to the features of the fourth calibration plate;

[0037] Step C5: Calculate the second positional relationship matrix between the lidar and the stacking robot based on the third calibration model;

[0038] Step C6: Perform calibration processing on the lidar according to the second position relationship matrix.

[0039] Preferably, the calibration process includes a fourth calibration process that calibrates the RGB camera based on the relationship between the stacking robot and the RGB camera;

[0040] Before performing the fourth calibration process, the stacking robot holding the second calibration plate is moved to the second preset calibration position in advance;

[0041] The fourth calibration includes:

[0042] Step D1: Acquire a second image of the second calibration board using the RGB camera, and extract the fifth calibration board features of the second calibration board based on the second image;

[0043] Step D2: Change the position and posture of the stacking robot multiple times, and return to step D1. After repeatedly executing step D1, multiple features of the fifth calibration plate are obtained.

[0044] Step D3: Construct a fourth calibration model based on multiple features of the fifth calibration plate and the position and posture of the stacking robot corresponding to the features of the fifth calibration plate;

[0045] Step D4: Calculate the positional relationship matrix between the RGB camera and the stacking robot based on the fourth calibration model; this is the third positional relationship matrix.

[0046] Step D5: Based on the third positional relationship matrix and the relationship between the stacking robot and the RGB camera, perform calibration processing on the RGB camera.

[0047] Preferably, the method for positioning the material stack using an AGV (Automated Guided Vehicle) based on the calibrated turntable, the RGB camera, the LiDAR, and the composite machine includes:

[0048] Step S1: When the AGV is located outside the compact space of the storage stack, control the AGV to face the interior of the compact space and use the pre-calibrated lidar to acquire the first lidar point cloud data of the compact space;

[0049] Step S2: Process the first laser point cloud data to obtain compact spatial information;

[0050] Step S3: Calculate the working position of the AGV based on the compact space information and the working space of the composite robot;

[0051] Step S4: Based on the working position and the second laser point cloud data obtained by scanning again after the AGV enters the compact space, calculate the AGV's travel trajectory, and control the AGV to reach the working position according to the travel trajectory;

[0052] Step S5: After the AGV arrives at the working position, the calibrated LiDAR is used to detect corner points inside the compact space and obtain corner point information;

[0053] Step S6: Obtain image information of the material stack through the calibrated RGB camera, and obtain actual information based on the second laser point cloud data;

[0054] Step S7: Based on the corner point information and the actual information, the calibrated stacking robot performs operations on the stack.

[0055] Preferably, the laser point cloud data includes the front, top, bottom, left and right sides, inner pillars, top beam, and obstacles on the bottom surface of the compact space;

[0056] The compact space information includes depth information and width information;

[0057] In step S2, the depth information of the compact space is obtained through the front and inner columns.

[0058] The width information of the compact space is obtained by the protrusions on the left and right sides.

[0059] Preferably, step S4 includes:

[0060] Step S41: After the AGV enters the interior of the compact space, the lidar is controlled to scan the interior of the compact space to obtain the second lidar point cloud data;

[0061] Step S42: Based on the second laser point cloud data and the working position, the travel trajectory of the AGV is obtained;

[0062] Step S43: Based on the second laser point cloud data and the compact space information, perform real-time collision detection, and correct the AGV's trajectory in real time based on the real-time collision detection results;

[0063] Step S44: Based on the real-time corrected travel trajectory, adjust the moving speed of the AGV in real time to control the AGV to reach and fix at the working position.

[0064] Preferably, step S5 includes:

[0065] Step S51: When the AGV arrives at the working position, control the turntable to rotate and simultaneously control the lidar to scan to obtain complete point cloud data of the interior of the compact space;

[0066] Step S52: Based on the working range of the stacking robot, extract point cloud data larger than the working range from the complete internal point cloud data;

[0067] Step S53: Fit the extracted point cloud data to obtain fitted point cloud data;

[0068] Step S54: Calculate the corner information based on the fitted point cloud data and the compact spatial information.

[0069] Preferably, step S6 includes:

[0070] Step S61: Acquire image information of the material stack using the calibrated RGB camera;

[0071] Step S62: Based on a pre-set segmentation mask algorithm, the stack of materials is segmented and identified according to the image information, and identification information is obtained;

[0072] Step S63: Obtain the actual information based on the identification information and the laser point cloud data.

[0073] The above technical solution has the following advantages or beneficial effects: The present invention combines LiDAR and RGB camera to generate image information of the material stack inside the compact space and guide the AGV to move to the specified working position in the compact space, identify the corner information inside the compact space, and guide the robot to perform operations. Attached Figure Description

[0074] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0075] Figure 1 In a preferred embodiment of the present invention, a diagram showing the relative positions of an AGV vehicle, a stacking robot, a turntable, an RGB camera, and a lidar in a stacking positioning method based on multi-sensor fusion in a compact space is provided.

[0076] Figure 2 In a preferred embodiment of the present invention, a flowchart illustrating the first calibration process for calibrating a lidar in a material stack positioning method based on multi-sensor fusion in a compact space is provided.

[0077] Figure 3In a preferred embodiment of the present invention, a flowchart illustrating the second calibration process for calibrating an RGB camera in a material stack positioning method based on multi-sensor fusion in a compact space is provided.

[0078] Figure 4 In a preferred embodiment of the present invention, a flowchart illustrating the third calibration process for calibrating the pose of a stack robot in a stack positioning method based on multi-sensor fusion in a compact space is provided.

[0079] Figure 5 In a preferred embodiment of the present invention, a flowchart illustrating the fourth calibration process for calibrating the pose of a stack robot in a stack positioning method based on multi-sensor fusion in a compact space is provided.

[0080] Figure 6 This is a schematic diagram of the process for positioning a stack of materials in a compact space based on multi-sensor fusion, as described in a preferred embodiment of the present invention.

[0081] Figure 7 This is a flowchart illustrating step S4 during the positioning process of the material stack, as described in a preferred embodiment of the present invention.

[0082] Figure 8 This is a flowchart illustrating step S5 during the positioning process of the material stack, as described in a preferred embodiment of the present invention.

[0083] Figure 9 This is a flowchart illustrating step S6 during the positioning process of the material stack, as described in a preferred embodiment of the present invention. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0086] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0087] In a preferred embodiment of the present invention, a method for stack positioning based on multi-sensor fusion in a compact space is provided, such as... Figure 1 As shown, a composite robot is provided, which includes an AGV trolley 1 and a stacking robot 2 mounted on the AGV trolley 1.

[0088] The front end of the AGV trolley 1 is equipped with a turntable 3 and an RGB camera 4 with a wide-angle lens. A lidar 5 is fixed on the turntable 3.

[0089] Stack positioning methods include:

[0090] Calibration processes were performed on the turntable 3, RGB camera 4, LiDAR 5, and stacking robot 2 respectively.

[0091] as well as

[0092] The AGV trolley 1 uses a calibrated turntable 3, RGB camera 4, LiDAR 5, and stacking robot 2 to position the stacking material for operation.

[0093] Specifically, in the above embodiments, the composite robot includes an AGV trolley 1 and a stacking robot 2 mounted on the AGV trolley 1; the front end of the AGV trolley 1 is equipped with a turntable 3, a wide-angle lens RGB camera 4, and a fixed lidar 5.

[0094] The calibration process refers to determining the relative positions of the turntable 3, LiDAR 5, RGB camera 4, and stacking robot 2 through a series of calibration steps. A single calibration process can determine the relative positions of two devices. Through this calibration process, accurate measurement results from the RGB camera and LiDAR can be obtained in the coordinate system of the stacking robot.

[0095] After calibration, AGV 1 can use turntable 3 and LiDAR 5 to locate the stack of materials. This information controls AGV 1 to move to the accurate position. Data from RGB camera 4 and LiDAR 5 can acquire the precise position and shape information of each material in the stack. This information can be transmitted to stacking robot 2 so that it can accurately perform tasks such as handling and stacking.

[0096] In a preferred embodiment of the present invention, the calibration process includes a first calibration process based on the relationship between the lidar 5 and the turntable 3 to calibrate the lidar 5.

[0097] Before performing the first calibration process, a first calibration plate is set up in advance within the scanning range in front of the lidar 5;

[0098] like Figure 2 As shown, the first calibration process includes:

[0099] Step A1: During the process of rotating the turntable 3 to the preset angle, randomly collect multiple rotation angles of the turntable 3 and record them as the first rotation angle;

[0100] as well as

[0101] The first radar data obtained by LiDAR 5 scanning the first calibration plate at each first rotation angle is acquired respectively;

[0102] Step A2: Preprocess multiple first radar data to obtain multiple first calibration board features;

[0103] Step A3: Based on the features of multiple first calibration plates, obtain multiple actual poses of the lidar 5 during the scanning process using the inter-frame matching method;

[0104] Step A4: Establish a first calibration model based on each actual pose and the corresponding first rotation angle, and perform calibration processing on the lidar 5 based on the first calibration model.

[0105] Specifically, in the above embodiments, in order to provide a known reference object for subsequent calibration processing, a first calibration plate is set in advance within the scanning range in front of the lidar 5 before performing the first calibration process.

[0106] The turntable 3 is controlled to rotate to multiple preset rotation angles, and at each angle, the lidar 5 is controlled to scan and obtain a set of radar data. The more preset rotation angles, the more comprehensive the data can be obtained, thus improving the accuracy of calibration.

[0107] By preprocessing multiple radar data, including noise removal, filtering, and outlier removal, features of the first calibration board, such as corner points and edges, can be extracted.

[0108] By using the inter-frame matching method, radar data between different frames can be matched to obtain the actual pose of the lidar 5 during the scanning process, that is, the position and attitude of the lidar 5 relative to the first calibration plate.

[0109] By establishing the first calibration model, the actual pose of the lidar 5 can be correlated with the rotation angle of the turntable 3, thereby obtaining the calibration parameters of the lidar 5, such as the rotation center and scanning angle, so as to achieve accurate calibration of the lidar 5.

[0110] In a preferred embodiment of the present invention, the calibration process includes a second calibration process for calibrating the RGB camera 4 based on the relationship between the lidar 5 and the RGB camera 4.

[0111] Before performing the second calibration process, a second calibration board is set up in advance within the field of view of the RGB camera 4;

[0112] like Figure 2 As shown, the second calibration process includes:

[0113] Step B1: Simultaneously control the lidar 5 and the turntable 3 to rotate 180 degrees, so that the lidar 5 scans multiple laser point cloud data of the second calibration board during the rotation and fuses them to obtain the second radar data;

[0114] Step B2: Preprocess the second radar data to obtain the second calibration board features;

[0115] Step B3: Acquire a first image of the second calibration board using the RGB camera 4, and extract the features of the third calibration board based on the first image;

[0116] Step B4: Move the second calibration board and return to step B1. Repeat steps B1-B4 multiple times to obtain multiple second calibration board features and multiple third calibration board features. Take a single second calibration board feature and the corresponding third calibration board feature as a group of first-class calibration board features.

[0117] Step B5: Based on multiple sets of features of the first type of calibration board and the corresponding first image, establish a second calibration model, and calculate the first positional relationship matrix between the lidar 5 and the RGB camera 4 based on the second calibration model;

[0118] Step B6: Perform calibration processing on the RGB camera 4 according to the first position relationship matrix.

[0119] Specifically, in the above embodiments, the positional relationship between the LiDAR 5 and the RGB camera 4 enables the data between the LiDAR 5 and the RGB camera 4 to correspond accurately.

[0120] During the second calibration process, a second calibration board needs to be placed within the field of view of the RGB camera 4 so that the RGB camera 4 can be calibrated using the second calibration board.

[0121] By rotating the lidar 5 multiple times and acquiring images, multiple sets of second calibration plate features and corresponding third calibration plate features are obtained to establish a second calibration model.

[0122] Based on multiple sets of first-class calibration plate features and corresponding first images, a second calibration model is established, and the first positional relationship matrix between the LiDAR 5 and the RGB camera 4 is calculated based on the model. This allows for the establishment of an accurate positional relationship between the LiDAR 5 and the RGB camera 4, thereby achieving registration between them.

[0123] The first positional relationship matrix enables accurate correspondence between the data of the LiDAR 5 and the RGB camera 4, and subsequent measurements and analyses can be performed based on the calibration results.

[0124] Furthermore, in the above embodiments, the dynamic lidar 5 collects point clouds in two ways: one is that the turntable 3 rotates continuously, and the lidar 5 continuously collects point clouds during the rotation until the turntable 3 rotates 180°.

[0125] Another approach involves non-continuous rotation of turntable 3, for example, stopping every 10° while lidar 5 acquires one frame of point cloud data, until turntable 3 has rotated 180°. Continuous rotation of turntable 3 results in a faster point cloud acquisition cycle and denser point cloud data, but motion compensation for the data acquired by lidar 5 during rotation needs to be considered.

[0126] The process of fusing multiple laser point cloud data is as follows:

[0127] First, during data acquisition, the rotation angle of turntable 3 and the data from lidar 5 are collected simultaneously.

[0128] For cases where the turntable 3 rotates discontinuously, the rotation angle corresponding to each frame of lidar 5 data is the fixed rotation angle of the turntable 3 when the data is collected.

[0129] For cases where turntable 3 rotates continuously, the data of LiDAR 5 for each frame and the rotation angle of turntable 3 at the corresponding time are matched according to the timestamp.

[0130] Then, based on the rotation angle corresponding to each frame of LiDAR data, initial alignment is performed, and the point cloud data is rotated and transformed.

[0131] Next, iterative optimization is performed to calculate the distance error between corresponding feature points after the rotation transformation of two frames of LiDAR 5 data, and the attitude transformation matrix between the two frames of data is optimized by minimizing the error.

[0132] During the iteration process, the error changes are monitored. When the error decreases to below an acceptable threshold, convergence is determined, the iteration is stopped, and the attitude transformation matrix is ​​recorded.

[0133] After completing the iterative optimization, a reference coordinate system is selected (e.g., the position where the rotation angle of turntable 3 is 0°), the attitude transformation matrix of each frame of LiDAR 5 data relative to the reference coordinate system is calculated, and the data is converted to the same reference coordinate system for point cloud stitching and fusion.

[0134] Finally, the stitched point cloud data is post-processed, including noise removal, selection of effective regions, filtering, downsampling, and other methods to optimize the data.

[0135] In this invention, the process of fusing multiple laser point cloud data is the same as described above, so it will not be elaborated further in the following content.

[0136] In a preferred embodiment of the present invention, the calibration process includes a third calibration process for calibrating the lidar 5 based on the relationship between the stacking robot 2 and the lidar 5.

[0137] Before performing the third calibration process, a third calibration plate is pre-installed on the end of the robotic arm of the stacking robot 2, and the stacking robot 2 is moved to the first preset calibration position;

[0138] like Figure 3 As shown, the third calibration process includes:

[0139] Step C1: Simultaneously control the lidar 5 and the turntable 3 to rotate 180 degrees, obtain multiple laser point cloud data of the lidar 5 scanning the third calibration board during the rotation process, and fuse them to obtain the third lidar data;

[0140] Step C2: Preprocess the third radar data to obtain the features of the fourth calibration board;

[0141] Step C3: After repeatedly changing the pose of the stacking robot 2 and cyclically executing steps C1-C3, multiple fourth calibration plate features are obtained;

[0142] Step C4: Construct a third calibration model based on the features of multiple fourth calibration plates and the position and posture of the stacking robot 2 corresponding to the features of the fourth calibration plates;

[0143] Step C5: Calculate the second positional relationship matrix between the lidar 5 and the stacking robot 2 based on the third calibration model;

[0144] Step C6: Calibrate the lidar 5 according to the second position relationship matrix.

[0145] Specifically, in the above embodiments, before the third calibration process, it is necessary to pre-install the third calibration plate and move the stacking robot 2 to the first preset calibration position to ensure the accuracy of the calibration;

[0146] The calibration process involves simultaneously controlling the lidar 5 and the turntable 3 to rotate 180 degrees. During the rotation, the lidar 5 scans multiple laser point cloud data of the third calibration board and fuses them to obtain the third lidar data.

[0147] The third radar data is preprocessed to obtain the characteristics of the fourth calibration board;

[0148] In addition, by repeatedly changing the pose of the stacking robot 2 and executing steps C1-C3 multiple times, multiple fourth calibration plate features can be obtained. These fourth calibration plate features are related to the position and pose of the stacking robot 2.

[0149] Based on the features of multiple fourth calibration plates and their corresponding robot positions and postures, a third calibration model can be constructed to calculate the second positional relationship matrix between the lidar 5 and the stacking robot 2.

[0150] In a preferred embodiment of the present invention, the calibration process includes a fourth calibration process based on the relationship between the stacking robot 2 and the RGB camera 4 to calibrate the RGB camera 4.

[0151] Before performing the fourth calibration process, the stacking robot 2 holding the second calibration plate is moved to the second preset calibration position in advance;

[0152] like Figure 4 As shown, the fourth calibration includes:

[0153] Step D1: Acquire a second image of the second calibration board using RGB camera 4, and extract the fifth calibration board features of the second calibration board based on the second image;

[0154] Step D2: Change the position and orientation of the stacking robot 2 multiple times, and return to step D1. After repeatedly executing step D1, multiple fifth calibration plate features are obtained.

[0155] Step D3: Construct the fourth calibration model based on the features of multiple fifth calibration plates and the position and posture of the stacking robot 2 corresponding to the features of the fifth calibration plates;

[0156] Step D4: Calculate the positional relationship matrix between RGB camera 4 and stacking robot 2 based on the fourth calibration model, which is the third positional relationship matrix;

[0157] Step D5: Based on the third position relationship matrix and the relationship between the stacking robot 2 and the RGB camera 4, calibrate the RGB camera 4.

[0158] Specifically, in the above embodiments, by calibrating the relationship between the stacking robot 2 and the RGB camera 4, the image information obtained by the RGB camera 4 can be converted to the coordinate system of the stacking robot 2.

[0159] Furthermore, the fourth calibration process includes: capturing images of the second calibration board using the RGB camera 4 and extracting features of the fifth calibration board from the images; repeatedly changing the position and orientation of the stacking robot 2 and repeating step D1 to obtain features of multiple fifth calibration boards; constructing a fourth calibration model using the features of multiple fifth calibration boards and the corresponding position and orientation of the stacking robot 2; calculating the positional relationship matrix between the RGB camera 4 and the stacking robot 2 using the fourth calibration model, as the third positional relationship matrix; and calibrating the RGB camera 4 based on the relationship between the stacking robot 2 and the RGB camera 4 according to the second positional relationship matrix.

[0160] In a preferred embodiment of the present invention, the method for positioning the material stack using the AGV trolley 1 based on the calibrated turntable 3, RGB camera 4, LiDAR 5, and composite machine is as follows: Figure 5 As shown, it includes:

[0161] Step S1: When the AGV trolley 1 is located outside the compact space of the storage stack, control the AGV trolley 1 to face the inside of the compact space, and use the pre-calibrated lidar 5 to acquire the first lidar point cloud data of the compact space;

[0162] Step S2: Obtain compact spatial information based on the first laser point cloud data;

[0163] Step S3: Calculate the working position of AGV 1 based on the compact space information and the working space of the composite robot;

[0164] Step S4: Based on the working position and the second laser point cloud data obtained by scanning again after the AGV 1 enters the compact space, calculate the travel trajectory of the AGV 1, and control the AGV 1 to reach the working position according to the travel trajectory.

[0165] Step S5: After the AGV trolley 1 arrives at the working position, the calibrated LiDAR 5 is used to detect corner points inside the compact space and obtain corner point information;

[0166] Step S6: Obtain image information of the material stack through the calibrated RGB camera 4, and obtain actual information based on the second laser point cloud data;

[0167] Step S7: Based on the corner information and actual information, the calibrated stacking robot 2 performs operations on the stack.

[0168] Specifically, in the above embodiments, the combination of turntable 3, lidar 5 and RGB camera 4 equipped with wide-angle lens enables accurate positioning and operation of the material stack inside the compact space.

[0169] The pre-calibrated LiDAR 5 is used to acquire laser point cloud data in a compact space. The data is preprocessed and features are extracted to obtain compact space information. The working position of the AGV is calculated based on the compact space information and the AGV's working space. The AGV's trajectory is calculated based on the working position and the laser point cloud data, and the AGV is controlled to reach the working position. The pre-calibrated LiDAR 5 is used to detect corners in the compact space and acquire corner information. The pre-calibrated RGB camera 4 is used to acquire image information of the material stack and combine it with the laser point cloud information to obtain actual information. Finally, the pre-calibrated composite robot is used to perform operations on the material stack.

[0170] By combining multiple sensors and precise calibration, accurate positioning and operation of material stacks can be achieved, improving work efficiency and accuracy.

[0171] Furthermore, the compact space in this invention, such as a carriage...

[0172] In a preferred embodiment of the present invention, the compact space information includes depth information and width information.

[0173] Specifically, in the above embodiments, the compact space in this invention, such as a carriage or container, obtains its depth information through the support columns at the front and inner sides of the compact space; and its width information is obtained from the protruding positions on the left and right sides.

[0174] The process of extracting information from the compact space is as follows: Start the lidar 5 and turntable 3. Rotate turntable 3 180°. Obtain the lidar point cloud data in the compact space by using the point cloud method obtained by splicing and fusion at different rotation angles. Perform lidar point cloud data preprocessing and feature extraction. Obtain the depth information of the compact space by using the positions of the pillars at the front and rear entrances of the compact space. Obtain the width information by using the positions of the corrugated plates on the left and right sides of the compact space.

[0175] The preprocessing process for laser point cloud data is as follows: Based on the maximum effective distance of parking in the compact space and the size of the compact space, the effective range of the point cloud within the compact space is determined, noise points outside the compact space are filtered out, and the first laser point cloud data within the compact space is obtained.

[0176] Furthermore, the simulation process for the compact space is as follows: the compact space is fitted with a plane based on the first laser point cloud data within the compact space, and planes belonging to the top / bottom of the compact space are selected based on the normal from all valid planes. Clustering is then performed to determine the equations of the top / bottom planes of the compact space.

[0177] Planar filtering is performed on point cloud I within the compact space to obtain laser point cloud data between the top and bottom planes of the compact space, which is the second laser point cloud data. With a fixed constraint angle θ in the AGV's travel direction, the second laser point cloud data is filtered, and the laser point cloud data within this constraint range is the third laser point cloud data, which belongs to the front plane of the compact space. The front plane equation is fitted based on the third laser point cloud data, and all points in the third laser point cloud data whose distance to this plane is less than the corrugated plate thickness are selected. The front plane equation is fitted again, and the second laser point cloud data is plane filtered again to obtain the fourth laser point cloud data outside the plane.

[0178] Plane fitting is performed on the fourth laser point cloud data, and two parallel planes are selected to determine the equations of the left / right planes. 5. The left / right sides of the compact space are corrugated plates with a thickness of about 20cm. The width of the compact space is calculated based on the convex point of the corrugated plate in the compact space.

[0179] Identify the pillars at the front and rear doors of the compact space, and calculate the length of the compact space using the rear end face of the front pillar and the front end face of the rear pillar as references.

[0180] In a preferred embodiment of the present invention, such as Figure 6 As shown, step S4 includes:

[0181] Step S41: After the AGV trolley 1 enters the interior of the compact space, the lidar 5 is controlled to scan the interior of the compact space to obtain the second lidar point cloud data.

[0182] Step S42: Based on the second laser point cloud data and the working position, the travel trajectory of AGV 1 is obtained;

[0183] Step S43: Based on the second laser point cloud data and compact space information, perform real-time collision detection, and correct the AGV vehicle 1's trajectory in real time based on the real-time collision detection results;

[0184] Step S44: Based on the real-time corrected travel trajectory, adjust the moving speed of AGV 1 in real time to control AGV 1 to reach and fix at the working position.

[0185] Specifically, in the above embodiments, the lidar 5 collects point clouds in two ways: one is that the turntable 3 rotates continuously, and the lidar 5 continuously collects point clouds during the rotation until the turntable 3 rotates 180°.

[0186] Another method is to rotate the turntable 3 discontinuously, for example, stopping once every 10° of rotation, while the lidar 5 collects a frame of point cloud data, until the turntable 3 rotates 180°.

[0187] The turntable 3 rotates continuously to collect point clouds at a faster pace and collects denser point clouds, but motion compensation for the data collected by the lidar 5 during rotation needs to be considered.

[0188] Furthermore, it is necessary to stitch and fuse point clouds with different rotation angles according to the rotation angle of turntable 3: First, simultaneously collect the rotation angle of turntable 3 and the laser point cloud data collected by lidar 5 corresponding to the rotation angle.

[0189] For cases where the turntable 3 rotates discontinuously, the rotation angle corresponding to the laser point cloud data collected by the lidar 5 in each frame is the fixed rotation angle of the turntable 3 when the data is collected.

[0190] For the case of continuous rotation of turntable 3, the laser point cloud data collected by lidar 5 in each frame and the rotation angle of turntable 3 at the corresponding time are matched according to the timestamp.

[0191] And according to the rotation angle corresponding to the laser point cloud data collected by each frame of lidar 5, the laser point cloud data is rotated and transformed.

[0192] After rotating and transforming the laser point cloud data collected by the two frames of lidar 5, the distance error between the corresponding feature points is calculated, and the attitude transformation matrix between the two frames of data is optimized by minimizing the error.

[0193] A reference coordinate system is selected, and the attitude transformation matrix of each frame of LiDAR data relative to the reference coordinate system is calculated. The data is then converted to the same reference coordinate system for point cloud stitching and fusion.

[0194] In this embodiment, the reference coordinates are set to the position where the rotation angle of turntable 3 is 0°; after stitching, the amount of laser point cloud data is large, so the data is optimized by removing noise, selecting effective areas, filtering, downsampling and other methods.

[0195] In a preferred embodiment of the present invention, such as Figure 7 As shown, step S5 includes:

[0196] Step S51: When the AGV trolley 1 arrives at the working position, control the turntable 3 to rotate and simultaneously control the lidar 5 to scan in order to obtain complete point cloud data of the compact space.

[0197] Step S52: Based on the working range of the stacking robot 2, extract point cloud data that is larger than the working range from the complete internal point cloud data;

[0198] Step S53: Fit the extracted point cloud data to obtain fitted point cloud data;

[0199] Step S54: Calculate the corner information based on the fitted point cloud data and compact spatial information.

[0200] Specifically, in the above embodiment, after the AGV is parked at the target working position, the lidar 5 is activated to start scanning, the turntable 3 rotates 180° to obtain complete point cloud data inside the compact space, and according to the current position of the composite robot and its working range, a portion slightly larger than the robot's working area is extracted from the complete lidar point cloud data for calculation.

[0201] The blob tool is used to divide the overall point cloud into several point cloud clusters, and point cloud clusters with too few points are removed. Then, a plane fitting tool is used to try to fit all the remaining point cloud clusters.

[0202] After fitting, based on the obtained planar pose, flatness, area and other information, it is determined whether it is a compact space sidewall, bottom surface and material stack surface, and the intersection of the three planes is calculated to obtain the required corner point information.

[0203] In a preferred embodiment of the present invention, such as Figure 8 As shown, step S6 includes:

[0204] Step S61: Acquire image information of the material stack using the calibrated RGB camera 4;

[0205] Step S62: Based on the pre-set segmentation mask algorithm, the material stack is segmented according to the image information, and the identification information is obtained;

[0206] Step S63: Obtain actual information based on the recognition information and laser point cloud data.

[0207] Specifically, in the above embodiments, after the composite robot completes the stacking action of a material, it uses RGB camera 4 and LiDAR 5 to acquire RGBD images of the working area, segments and identifies the placed material based on the segmentation mask algorithm, and outputs its position information to compare with the theoretical stacking data to confirm the stacking effect.

[0208] In summary, the technical solution of this invention includes a composite robot comprising an AGV trolley 1 and a stacking robot mounted on the AGV trolley 1. The front of the AGV trolley 1 is equipped with a LiDAR 5 and an RGB camera 4. The LiDAR 5 is mounted on the AGV trolley 1 via a turntable 3. A stacking robot 2 is also mounted at the rear of the AGV trolley 1. For the configured composite robot, the following calibration operations are performed on its components:

[0209] Based on the relative positional relationship between turntable 3 and lidar 5, lidar 5 is calibrated.

[0210] Based on the relative positional relationship between LiDAR 5 and RGB camera 4, RGB camera 4 is calibrated;

[0211] Based on the relative positional relationship between LiDAR 5 and the stacking robot 2, LiDAR 5 is calibrated.

[0212] The RGB camera 4 is calibrated based on its relative position to the stacking robot 2.

[0213] Subsequently, using a calibrated composite robot, the following main steps are followed to enable the composite robot to operate within the compact space of the storage stack.

[0214] First, before the composite robot enters the compact space, the LiDAR 5 is used to acquire the internal point cloud of the compact space. Based on the internal point cloud, the internal information of the compact space is obtained, and further processing is performed to obtain the movement trajectory of the AGV 1.

[0215] Subsequently, after the AGV 1 is driven to the designated position according to the movement trajectory, the lidar 5 is used again to scan the interior of the compact space to obtain the corner information of the compact space.

[0216] Next, the stacking robot 2 performs its work based on the corner information.

[0217] Finally, the stacking robot 2 determines the operation result based on the images obtained by the RGB camera and the point cloud data collected by the LiDAR.

[0218] Without human intervention, the material stack information is assumed to be known, and the robot can perform operations based on corner point information. RGB cameras and LiDAR are used to identify and locate the materials in the stack, which are then compared with the theoretical stack information.

[0219] Compared with existing technologies, the above technical solution adopts a unique dynamic rotation scanning design of LiDAR 5 to obtain point clouds inside the compact space, and combines it with RGB camera 4 to generate RGBD images of the compact space. This technology can guide the AGV trolley 1 carrying the palletizing robot 2 to move to the designated working position according to the internal conditions of the compact space where the pallet is stored. Based on the corner point information of the compact space identified by LiDAR 5, the palletizing robot 2 is guided to perform palletizing or depalletizing operations.

[0220] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for positioning a stack of materials in a compact space based on multi-sensor fusion, characterized in that, A composite robot is provided, which includes an AGV trolley and a stacking robot mounted on the AGV trolley; The front end of the AGV is equipped with a turntable and an RGB camera with a wide-angle lens, and a lidar is fixed on the turntable. The stack positioning method includes: Calibration processes are performed on the turntable, the RGB camera, the lidar, and the stacking robot respectively. as well as The AGV (Automated Guided Vehicle) uses the calibrated turntable, RGB camera, LiDAR, and stacking robot to locate the stacking robot for operation. The calibration process includes a second calibration process that calibrates the RGB camera based on the relationship between the LiDAR and the RGB camera; Before performing the second calibration process, a second calibration board is set up in advance within the field of view of the RGB camera; The second calibration process includes: Step B1: Simultaneously control the lidar and the turntable to rotate 180 degrees, so that the lidar scans multiple laser point cloud data of the second calibration board during the rotation and fuses them to obtain the second radar data; Step B2: Preprocess the second radar data to obtain the second calibration board features; Step B3: Acquire a first image of the second calibration board using the RGB camera, and extract the features of the third calibration board based on the first image; Step B4: Move the second calibration board and return to step B1. Repeat steps B1-B4 multiple times to obtain multiple second calibration board features and multiple third calibration board features. Take a single second calibration board feature and the corresponding third calibration board feature as a group of first-class calibration board features. Step B5: Based on multiple sets of features of the first type of calibration board and the corresponding first image, establish a second calibration model, and calculate the first positional relationship matrix between the lidar and the RGB camera based on the second calibration model; Step B6: Perform calibration processing on the RGB camera according to the first position relationship matrix.

2. The stack positioning method according to claim 1, characterized in that, The calibration process includes a first calibration process that calibrates the lidar based on the relationship between the lidar and the turntable; Before performing the first calibration process, a first calibration plate is set up in advance within the scanning range in front of the lidar; The first calibration process includes: Step A1: During the process of rotating the turntable to a preset angle, randomly collect multiple rotation angles of the turntable and record them as the first rotation angle; as well as The first radar data obtained by the lidar scanning the first calibration plate at each of the first rotation angles is acquired respectively; Step A2: Preprocess multiple sets of the first radar data to obtain multiple first calibration board features; Step A3: Based on multiple features of the first calibration board, obtain multiple actual poses of the lidar during the scanning process using the inter-frame matching method; Step A4: Establish a first calibration model based on each actual pose and the corresponding first rotation angle, and perform calibration processing on the lidar based on the first calibration model.

3. The stack positioning method according to claim 1, characterized in that, The calibration process includes a third calibration process that calibrates the lidar based on the relationship between the stacking robot and the lidar. Before performing the third calibration process, a third calibration plate is pre-installed on the end of the robotic arm of the stacking robot, and the stacking robot is moved to the first preset calibration position; The third calibration process includes: Step C1: Simultaneously control the lidar and the turntable to rotate 180 degrees, obtain multiple laser point cloud data of the lidar scanning the third calibration board during the rotation process, and fuse them to obtain the third lidar data; Step C2: Preprocess the third radar data to obtain the fourth calibration board features; Step C3: After repeatedly changing the pose of the stacking robot and cyclically executing steps C1-C3, multiple features of the fourth calibration plate are obtained; Step C4: Construct a third calibration model based on multiple features of the fourth calibration plate and the position and posture of the stacking robot corresponding to the features of the fourth calibration plate; Step C5: Calculate the second positional relationship matrix between the lidar and the stacking robot based on the third calibration model; Step C6: Perform calibration processing on the lidar according to the second position relationship matrix.

4. The stack positioning method according to claim 1, characterized in that, The calibration process includes a fourth calibration process, which calibrates the RGB camera based on the relationship between the stacking robot and the RGB camera. Before performing the fourth calibration process, the stacking robot holding the second calibration plate is moved to the second preset calibration position in advance; The fourth calibration includes: Step D1: Acquire a second image of the second calibration board using the RGB camera, and extract the fifth calibration board features of the second calibration board based on the second image; Step D2: Change the position and posture of the stacking robot multiple times, and return to step D1. After repeatedly executing step D1, multiple features of the fifth calibration plate are obtained. Step D3: Construct a fourth calibration model based on multiple features of the fifth calibration plate and the position and posture of the stacking robot corresponding to the features of the fifth calibration plate; Step D4: Calculate the positional relationship matrix between the RGB camera and the stacking robot based on the fourth calibration model; this is the third positional relationship matrix. Step D5: Based on the third positional relationship matrix and the relationship between the stacking robot and the RGB camera, perform calibration processing on the RGB camera.

5. The stack positioning method according to claim 1, characterized in that, The method for positioning the stack of materials using an AGV (Automated Guided Vehicle) based on the calibrated turntable, the RGB camera, the LiDAR, and the composite machine includes: Step S1: When the AGV is located outside the compact space of the storage stack, control the AGV to face the interior of the compact space and use the pre-calibrated lidar to acquire the first lidar point cloud data of the compact space; Step S2: Process the first laser point cloud data to obtain compact spatial information; Step S3: Calculate the working position of the AGV based on the compact space information and the working space of the composite robot; Step S4: Based on the working position and the second laser point cloud data obtained by scanning again after the AGV enters the compact space, calculate the AGV's travel trajectory, and control the AGV to reach the working position according to the travel trajectory; Step S5: After the AGV arrives at the working position, the calibrated LiDAR is used to detect corner points inside the compact space and obtain corner point information; Step S6: Obtain image information of the material stack through the calibrated RGB camera, and obtain actual information based on the second laser point cloud data; Step S7: Based on the corner point information and the actual information, the calibrated stacking robot performs operations on the stack.

6. The stack positioning method according to claim 5, characterized in that, The compact spatial information includes depth information and width information.

7. The stack positioning method according to claim 5, characterized in that, Step S4 includes: Step S41: After the AGV enters the interior of the compact space, the lidar is controlled to scan the interior of the compact space to obtain the second lidar point cloud data; Step S42: Based on the second laser point cloud data and the working position, the travel trajectory of the AGV is obtained; Step S43: Based on the second laser point cloud data and the compact space information, perform real-time collision detection, and correct the AGV's trajectory in real time based on the real-time collision detection results; Step S44: Based on the real-time corrected travel trajectory, adjust the moving speed of the AGV in real time to control the AGV to reach and fix at the working position.

8. The stack positioning method according to claim 5, characterized in that, Step S5 includes: Step S51: When the AGV arrives at the working position, control the turntable to rotate and simultaneously control the lidar to scan to obtain complete point cloud data of the interior of the compact space; Step S52: Based on the working range of the stacking robot, extract point cloud data larger than the working range from the complete internal point cloud data; Step S53: Fit the extracted point cloud data to obtain fitted point cloud data; Step S54: Calculate the corner information based on the fitted point cloud data and the compact spatial information.

9. The stack positioning method according to claim 5, characterized in that, Step S6 includes: Step S61: Acquire image information of the material stack using the calibrated RGB camera; Step S62: Based on a pre-set segmentation mask algorithm, the stack of materials is segmented and identified according to the image information, and identification information is obtained; Step S63: Obtain the actual information based on the identification information and the laser point cloud data.

Images