An automatic loading system based on visual detection and a control method thereof

Abstract

The application discloses an automatic loading system based on visual detection and a control method thereof, and belongs to the technical field of loading of goods, in particular to an automatic loading system based on visual detection and a control method thereof. The application aims to provide an automatic loading system based on visual detection and a control method thereof, and the automatic loading system is convenient to use.

Description

Technical Field

[0001] This invention relates to the field of cargo loading technology, and more specifically, to an automated loading system and control method based on vision detection. Background Technology

[0002] In modern production processes, the loading and unloading of goods is becoming increasingly mechanized and procedural, leading to a growing demand for mechanization for a large number of repetitive tasks. In recent years, with the development of industrial robots, enterprise production processes have seen significant advancements, making industrial robots an essential component of modern machinery manufacturing. Industrial robots have significantly improved labor productivity and product quality, playing a crucial role in accelerating the mechanization and automation of industrial production.

[0003] Existing truck loading machines typically employ industrial robots, whose loading targets are pre-taught based on the truck's length, width, height, and vehicle posture. This requires the truck driver to adjust the truck to a designated position and posture before automated loading begins. If the truck's parking position or posture differs from the pre-taught posture, the automated loading system cannot complete the process. Furthermore, loading sites are often extremely confined, making it difficult for drivers to adjust the truck's position or posture to the designated location. This further limits the capabilities of existing automated loading systems based on teach-in industrial robots. The inability to determine the truck's position information hinders improvements in loading efficiency. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an automatic loading system and control method based on vision detection, which facilitates automatic loading.

[0005] The present invention achieves its objective by employing the following technical solution:

[0006] An automated loading system and control method based on vision detection are disclosed. The invention includes a vehicle height measurement system, a liftable reference rod system, a vehicle body position detection system, an industrial robot movement system, and a conveyor belt cargo transportation system. The vehicle height measurement system comprises a fixed-height gantry frame, a freight truck, and a laser sensor measuring device. The laser height measuring device is fixed at the middle position of the gantry frame crossbar and is perpendicular to the freight truck body. The liftable reference rod system includes a linear motor-controlled liftable platform and a reference rod, with the reference rod fixedly connected to a lifting rail of the liftable platform. The body position detection system includes multiple gantry cameras and the truck. The cameras are fixed at the middle crossbar of the gantry and placed from top to bottom. The truck is parked within the area of ​​the gantry. The industrial robot movement system consists of a slide rail system and an industrial robot. The slide rail system is located on one side of the truck body. A drive motor drives the industrial robot to move on the slide rail system. The base of the industrial robot is on the same horizontal plane as the base of the liftable platform. The conveyor belt transport system includes a conveyor belt, a sensor detection device, and goods. The sensor detection device is installed at the end of the conveyor belt.

[0007] A control method for an automated loading system based on vision detection, characterized by comprising the following steps:

[0008] Step 1: Measure the height of the bottom of the cargo compartment of the vehicle to be loaded from the ground and proceed to the next step;

[0009] Step 2: Operate the lifting slide rail of the lifting platform to raise the reference rod to the same height as the bottom of the carriage, and determine the distance between unit pixels through the reference rod;

[0010] Step 3: The images of the truck captured by the camera are stitched together to obtain a complete image of the truck body. The corner points of the cargo compartment are located, and the coordinates of the cargo placement position are calculated and sent to the industrial robot.

[0011] Step 4: The conveyor belt transports goods. When the goods on the conveyor belt reach the end of the conveyor belt, the sensor detection device detects that the goods have reached the end of the conveyor belt and transmits a signal to the industrial robot. The end effector of the industrial robot moves to the end of the conveyor belt to grip the goods.

[0012] Step 5: Use the industrial robot to load the goods onto the vehicle. During the continuous loading of the goods, the industrial robot moves on the slide rail system to continuously adjust the coordinate position of the goods in the vehicle, and repeats steps 4 and 5.

[0013] As a further limitation of this technical solution, the specific steps for measuring the height of the bottom of the cargo compartment of the truck to be loaded from the ground in step 1 are as follows:

[0014] Step 1.1: The truck enters the area of ​​the gantry, and the camera monitors the movement of the truck in real time;

[0015] Step 1.2: When the truck is detected to have moved to the gantry and stopped moving, the laser rangefinder is activated and the distance L0 between the top of the truck bed and the camera is obtained. Let the height of the gantry be L, then the value of L-L0 is the height of the upper surface of the truck bed from the ground.

[0016] As a further limitation of this technical solution, in step 2, the length of the unit pixel is determined by the reference rod, and the specific steps are as follows:

[0017] Step 2.1: Set three reference points with known distances on the two patterns of the reference rod, which are called the starting reference point, the horizontal ending reference point, and the vertical ending reference point, respectively. The camera above the reference rod obtains the image coordinates of the reference points in real time through the reference point detection algorithm.

[0018] The coordinate system is the world coordinate system of the loading machine system. The coordinate system is the image coordinate system with the industrial robot as the origin. A is the starting reference point, B is the ending reference point in the horizontal direction, E is the ending reference point in the vertical direction, C and D are the two corner points of the car body, P is the actual distance from the industrial robot to the starting reference point, α is the angle between the line connecting the industrial robot and the corner point D of the car body and the horizontal direction, and β is the angle between the line connecting the starting reference point A and the corner point D of the car body and the horizontal direction.

[0019] Step 2.2: Raise the reference rod to the same height as the bottom of the carriage;

[0020] Step 2.3: Using the image coordinates of the starting reference point and the horizontal ending reference point, we can obtain the image distance between the two image coordinates. Knowing the actual distance between the two points and the image distance, we can obtain the length of the image unit pixel in the horizontal direction. Similarly, we can obtain the length of the image unit pixel in the vertical direction.

[0021] Step 2.3.1: Select the image coordinates of the starting reference point as (x1, y1), the image coordinates of the ending reference point in the horizontal direction as (x2, y2), and the image coordinates of the ending reference point in the vertical direction as (x3, y3).

[0022] Step 2.3.2: Let the image distance between two points be S, and calculate the image distance between the two points. The formula for calculating the image distance is as follows:

[0023]

[0024] Step 2.3.3: Let the length of a unit pixel in the horizontal direction be d. x The length of a unit pixel in the vertical direction is d. y The actual distance between the starting and ending reference points is f, and the length of a unit pixel is s. The calculation formula is as follows:

[0025]

[0026] The length d of a unit pixel in the horizontal direction can be obtained through the calculation formula. x The length d of a unit pixel in the vertical direction y During the process of the reference rod rising, the distance between the reference point and the camera is constantly changing, and the length of the unit pixel is also constantly changing. Therefore, the reference point detection algorithm needs to detect the reference point in real time and calculate the length of the unit pixel.

[0027] As a further limitation of this technical solution, in step 3, a complete vehicle image is obtained through image stitching. The corner positions of the cargo compartment are then located using the complete vehicle image, and the coordinates of the cargo placement position are calculated. The specific steps are as follows:

[0028] Step 3.1: After the truck enters the gantry parking area, multiple cameras begin to capture vehicle images;

[0029] Step 3.2: Stitch together the vehicle images acquired from multiple locations to obtain a complete vehicle image;

[0030] Step 3.3: Perform image detection on the complete vehicle image and determine the four corner points of the truck compartment using the corner detection algorithm for the compartment area;

[0031] Step 3.3.1: Locate the boundary of the entire carriage in the complete image to obtain the boundary image of the carriage;

[0032] Step 3.3.1.1: Select a set of original seed points at the center of the stitched carriage image;

[0033] Step 3.3.1.2: Sequentially merge pixels with the same or similar properties around the seed pixel into the region where the seed pixel is located to form new seed points;

[0034] Step 3.3.1.3: Repeat step 3.3.1.2 until no new seed points can be formed, and display the boundary image of the carriage;

[0035] Step 3.3.2: Perform corner detection on the boundary image of the carriage to determine the positions of the four corner points of the carriage;

[0036] The carriage boundary image obtained in step 3.3.1.3 is processed, and the positions of the four corner points in the image are determined by a corner detection algorithm;

[0037] In the edge region, pixels parallel to the edge direction show high similarity; however, nearby pixels perpendicular to the edge direction exhibit significant differences in grayscale values. For the corner region, the grayscale intensity of surrounding pixels changes significantly in all directions. The gradient values ​​I in the two directions of the boundary image I are calculated. x with I y The Gaussian autocorrelation matrix M is constructed by multiplying the Gaussian function G with its respective gradient and then using Gaussian weighting. Finally, the response value R is calculated using this matrix M to make the final corner point decision. The calculation formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042] This represents the convolution operator, where σ is the Gaussian scale, and G... x，y Let G be the two gradient directions of the Gaussian function G. If the two eigenvalues ​​of the autocorrelation matrix M are not significantly different, are both small and approximately equal, then they represent homogeneous locations in the image. Conversely, if they are both large and approximately equal, then they indicate corner regions.

[0043] Step 3.4: After determining the corner points, calculate the coordinates of the corner points, and calculate the stacking coordinates based on the length and width of the carriage calculated from the corner point coordinates and the dimensions of the goods to be loaded;

[0044] Step 3.4.1: Select any corner point. Based on the number of pixels and the size of the unit pixel between the corner point and the starting reference point, the actual distance between the corner point and the starting reference point can be obtained. Similarly, the actual distance between the corner point and the ending reference point can be calculated.

[0045] In step 3.4.1, based on the number of pixels at the corner point and the starting reference point, and the included angle β, the number of pixels in the horizontal and vertical directions can be obtained. Let m be the number of pixels at the corner point and the starting reference point in the horizontal direction, and let N be the actual distance between them. Then N = m * d xLet n be the number of pixels at the corner point and the starting reference point in the vertical direction, and let Q be the actual distance between them. Then Q = n * d y , where m, d x ,n,d y All data is known. Therefore, the coordinates (m, n) of the corner point in the image can be obtained. If the coordinates of the industrial robot in the world coordinate system are (x′, y′, 0), then the coordinates of the corner point in the world coordinate system are (x′+P+N, y′-Q, L-L0). Similarly, the image coordinates and world coordinates of other corner points can be obtained.

[0046] Step 3.4.2: Based on the length, height, and width information of the goods, as well as the height, length, and width of the carriage, calculate the number of stacking layers and quantity to realize the change in the quantity and type of goods placed;

[0047] Step 3.4.2.1: Let the height of the carriage be H, the width of the carriage be Y, the length of the carriage be X, the length of the cargo be x, the width of the cargo be y, and the height of the cargo be h. Then... The number of goods that can be placed in the length direction of the carriage, k1, the number of goods that can be placed in the width direction of the carriage, k2, and the number of stacking layers of goods, k3, are obtained.

[0048] Step 3.4.2.2: Determine the spatial coordinates of the center point where each item needs to be placed. The spatial coordinates are... Where (k1 = 0, 1, ... k1; k2 = 0, 1, ... k2; k3 = 0, 1, ... k3);), the calculation order of the cargo midpoint starts from the cargo at the corner position. The coordinates of the center point can be used as the position of the cargo in the carriage. If k1 = 0, k2 = 0, k3 = 0, and let m2 be the number of pixels between the center point and the origin of the image coordinate system in the horizontal direction, and let N2 be the actual distance between them, then N2 = m2 * d x Let n2 be the number of pixels between the center point and the origin of the image coordinate system in the vertical direction, and let Q2 be the actual distance between them. Then Q2 = n2 * d y Where m2, d x n2,d y Given the known data, we can obtain the coordinates (m2, n2) of the center point in the image coordinate system. Let the coordinates of the industrial robot in the world coordinate system be (x′, y′, 0), then the coordinates of the center point in the world coordinate system are (x′+N2, y′-Q2, L-L0). Similarly, we can obtain the image coordinates and world coordinates of the other placement positions of the first layer of goods. When placing the k3rd layer of goods, its world coordinates are...

[0049] As a further limitation of this technical solution, in step 4, the conveyor belt transports the goods, and the specific steps are as follows:

[0050] Step 4.1: The conveyor belt transports the goods to the end of the conveyor belt. After the laser sensor detects the goods, the conveyor belt stops and sends a signal to the industrial robot.

[0051] Step 4.2: The industrial robot receives a signal and picks up the goods. After the laser sensor detects that there are no goods at the end of the conveyor belt, the conveyor belt continues to run and repeats the operation when the goods reach the end again.

[0052] As a further limitation of this technical solution, in step 5, the industrial robot is used to load the goods onto the vehicle. The specific steps are as follows:

[0053] Step 5.1: Before loading the industrial robot onto the vehicle, the industrial robot needs to be powered on.

[0054] Step 5.2: After obtaining the location where the truck is parked and the location where the goods need to be placed in the truck bed, the industrial robot moves to the vicinity of the location where the goods need to be placed via the slide rail system;

[0055] Step 5.2.1: Establish a coordinate system with the industrial robot as the center. When the industrial robot is located at the origin, obtain the coordinates of the location where the goods need to be placed in the carriage from step 3.4.2.2.

[0056] Step 5.2.2: Control the industrial robot according to the coordinates of the goods to grasp and stack the goods, and place the goods at the target position;

[0057] Step 5.2.3: When the industrial robot moves to the starting reference point, let m3 be the number of pixels between the position of the placed goods and the starting reference point in the horizontal direction, and let N3 be the actual distance between them. Then N3 = m3 * d x Let n3 be the number of pixels between the location of the goods and the starting reference point in the vertical direction, and let Q3 be the actual distance between them. Then Q3 = n3 * d y , where m3, d x n3,d y Given known data, let the coordinates of the industrial robot in the world coordinate system be (x′, y′, 0). Then the coordinates of the corner point in the world coordinate system are (x′+P+N3, y′-Q3, L-L0). After determining the corner point coordinates, replace the origin of the image coordinate system in step 3.4.2.2 with the starting reference point to obtain the coordinates of the location where the goods need to be placed inside the carriage.

[0058] Step 5.3: The industrial robot picks up the goods and loads them onto the vehicle. The conveyor belt continues to run, and steps 4 and 5 are repeated.

[0059] Compared with the prior art, the advantages and positive effects of the present invention are:

[0060] 1. This device employs a laser sensor measuring device, a liftable platform, a reference rod, and a camera to perform visual inspection. The coordinate system is the world coordinate system of the loading machine system, specifically the image coordinate system with the industrial robot 8 as the origin. A is the starting reference point, B is the horizontal ending reference point, E is the vertical ending reference point, C and D are the two corner points of the truck bed, P is the actual distance from the industrial robot 8 to the starting reference point, α is the angle between the line connecting the industrial robot 8 and corner point D of the truck bed and the horizontal direction, and β is the angle between the line connecting the starting reference point A and corner point D of the truck bed and the horizontal direction. This achieves automatic positioning of the truck, facilitating subsequent automatic loading. It avoids situations where, in extremely narrow spaces, the driver is sometimes unable to adjust the truck's position or posture to the designated position, thus improving loading efficiency.

[0061] 2. When calculating the length of a unit pixel, the distance between the reference point and the camera changes continuously as the reference rod rises, and the length of the unit pixel also changes continuously. Therefore, the reference point detection algorithm is used to detect the reference point in real time, which improves the accuracy of corner coordinate calculation and cargo placement position coordinate calculation, thereby achieving precise loading of cargo.

[0062] 3. This invention directly calculates the number of stacking layers and quantities based on the length, height, and width of the goods, as well as the height, length, and width of the wagon. This allows for changes in the quantity and type of goods placed, greatly liberating labor, reducing the burden on workers, improving loading efficiency, and reducing costs. It also speeds up the operational process and achieves modern artificial intelligence.

[0063] 4. When using industrial robots to load goods, the industrial robots move on the slide rail system according to the required position of the goods, achieving the shortest distance from the goods to the loading position, which greatly improves loading efficiency, saves labor, and accelerates the automation process of loading. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the automatic loading machine control system of the present invention.

[0065] Figure 2 This is a schematic diagram of the laser sensor measuring device and camera of the present invention.

[0066] Figure 3This is a schematic diagram of the device for determining pixel distance using a liftable reference rod according to the present invention.

[0067] Figure 4 This is a schematic diagram of the industrial robot slide rail system of the present invention.

[0068] Figure 5 The present invention relates to a conveyor belt system for transporting goods.

[0069] Figure 6 This is a partial schematic diagram of the industrial robot slide rail system of the present invention.

[0070] Figure 7 This is a schematic diagram of the automatic loading machine control system of the present invention.

[0071] Figure 8 This is a flowchart of the automatic loading machine control system of the present invention.

[0072] Figure 9 This is a schematic diagram illustrating the transformation between the image coordinate system and the world coordinate system according to the present invention.

[0073] In the diagram: 1. Gantry, 2. Truck, 3. Laser sensor measuring device, 4. Liftable platform, 5. Reference rod, 6. Camera, 7. Slide rail system, 8. Industrial robot, 9. Conveyor belt, 10. Cargo, 11. Sensor detection device. Detailed Implementation

[0074] The following detailed description of a specific embodiment of the present invention is provided in conjunction with the accompanying drawings. However, it should be understood that the scope of protection of the present invention is not limited to the specific embodiment.

[0075] like Figures 1-9As shown, the present invention includes a vehicle height measurement system, a liftable reference rod system, a vehicle body position detection system, an industrial robot movement system, and a conveyor belt cargo transportation system. The vehicle height measurement system includes a fixed-height gantry frame 1, a truck 2, and a laser sensor measuring device 3. The laser height measuring device 3 is fixed at the middle position of the crossbar of the gantry frame 1 and is perpendicular to the vehicle body of the truck 2. The liftable reference rod system includes a linear motor-controlled liftable platform 4 and a reference rod 5. The reference rod 5 is fixedly connected to the lifting rail of the liftable platform 4. The vehicle body position detection system includes multiple gantry frame cameras 6. The truck 2 and the camera 6 are fixed at the position of the middle crossbar of the gantry 1 and placed from top to bottom. The truck 2 is parked within the area of ​​the gantry 1. The industrial robot moving system consists of a slide rail system 7 and an industrial robot 8. The slide rail system 7 is located on one side of the truck 2 body. The drive motor drives the industrial robot 8 to move on the slide rail system 7. The base of the industrial robot 8 is on the same horizontal plane as the base of the lifting platform 4. The conveyor belt transport system includes a conveyor belt 9, a sensor detection device 11 and cargo 10. The sensor detection device 11 is installed at the end of the conveyor belt 9.

[0076] A control method for an automated loading system based on vision detection includes the following steps:

[0077] Step 1: Measure the height of the bottom of the cargo compartment of the vehicle to be loaded from the ground and proceed to the next step;

[0078] Step 2: Operate the lifting slide rail of the lifting platform (4) to raise the reference rod (5) to the same height as the bottom of the carriage, and determine the distance between unit pixels through the reference rod 5;

[0079] Step 3: The images of the truck captured by the camera 6 are stitched together to obtain a complete vehicle image. The corner points of the cargo compartment are located, and the coordinates of the cargo 10 are calculated and sent to the industrial robot 8.

[0080] Step 4: The conveyor belt 9 transports goods. When the goods on the conveyor belt reach the end of the conveyor belt, the sensor detection device 11 detects that the goods have reached the end of the conveyor belt and transmits a signal to the industrial robot 8. The end effector of the industrial robot 8 moves to the end of the conveyor belt 9 to grip the goods 10.

[0081] Step 5: Use the industrial robot 8 to load the goods onto the vehicle. As the goods 10 are continuously loaded, the industrial robot 8 moves on the slide rail system 7 to continuously adjust the coordinate position of the goods 10 in the vehicle, and repeat steps 4 and 5.

[0082] In step 1, the specific steps for measuring the height of the bottom of the cargo compartment of the truck 2 to be loaded from the ground are as follows:

[0083] Step 1.1: The truck 2 enters the area of ​​the gantry 1, and the camera 6 monitors the movement of the truck 2 in real time;

[0084] Step 1.2: When the truck 2 is detected to have moved to the gantry 1 and stopped moving, the laser rangefinder 3 is activated and the distance L0 between the top of the truck 2 and the camera 6 is obtained. Let the height of the gantry be L, then the value of L-L0 is the height of the upper surface of the truck body from the ground.

[0085] In step 2, the length of the unit pixel is determined by the reference rod 5. The specific steps are as follows:

[0086] Step 2.1: Set three reference points with known distances on the two patterns of the reference rod 5, which are called the starting reference point, the horizontal ending reference point and the vertical ending reference point, respectively. The camera 6 above the reference rod 5 obtains the image coordinates of the reference points in real time through the reference point detection algorithm.

[0087] The coordinate system is the world coordinate system of the loading machine system. The coordinate system is the image coordinate system with the industrial robot 8 as the origin. A is the starting reference point, B is the ending reference point in the horizontal direction, E is the ending reference point in the vertical direction, C and D are the two corner points of the carriage, P is the actual distance from the industrial robot 8 to the starting reference point, α is the angle between the line connecting the industrial robot 8 and the corner point D of the carriage and the horizontal direction, and β is the angle between the line connecting the starting reference point A and the corner point D of the carriage and the horizontal direction.

[0088] Step 2.2: Raise the reference rod 5 to the same height as the bottom of the carriage;

[0089] Step 2.3: Using the image coordinates of the starting reference point and the horizontal ending reference point, we can obtain the image distance between the two image coordinates. Knowing the actual distance between the two points and the image distance, we can obtain the length of the image unit pixel in the horizontal direction. Similarly, we can obtain the length of the image unit pixel in the vertical direction.

[0090] Step 2.3.1: Select the image coordinates of the starting reference point as (x1, y1), the image coordinates of the ending reference point in the horizontal direction as (x2, y2), and the image coordinates of the ending reference point in the vertical direction as (x3, y3).

[0091] Step 2.3.2: Let the image distance between two points be S, and calculate the image distance between the two points. The formula for calculating the image distance is as follows:

[0092]

[0093] Step 2.3.3: Let the length of a unit pixel in the horizontal direction be d. x The length of a unit pixel in the vertical direction is d. y The actual distance between the starting and ending reference points is f, and the length of a unit pixel is s. The calculation formula is as follows:

[0094]

[0095] The length d of a unit pixel in the horizontal direction can be obtained through the calculation formula. x The length d of a unit pixel in the vertical direction y During the process of the reference rod 5 rising, the distance between the reference point and the camera 6 is constantly changing, and the length of the unit pixel is also constantly changing. Therefore, the reference point detection algorithm needs to detect the reference point in real time and calculate the length of the unit pixel.

[0096] In step 3, a complete vehicle image is obtained through image stitching. The corner points of the cargo compartment are located using the complete vehicle image, and the coordinates of the cargo placement position are calculated. The specific steps are as follows:

[0097] Step 3.1: After the truck 2 enters the parking area of ​​the gantry 1, the multiple cameras 6 begin to collect vehicle images;

[0098] Step 3.2: Stitch together the vehicle images acquired from multiple locations to obtain a complete vehicle image;

[0099] Step 3.3: Perform image detection on the complete vehicle image and determine the four corner points of the truck compartment using the corner detection algorithm for the compartment area;

[0100] Step 3.3.1: Locate the boundary of the entire carriage in the complete image to obtain the boundary image of the carriage;

[0101] Step 3.3.1.1: Select a set of original seed points at the center of the stitched carriage image;

[0102] Step 3.3.1.2: Sequentially merge pixels with the same or similar properties around the seed pixel into the region where the seed pixel is located to form new seed points;

[0103] Step 3.3.1.3: Repeat step 3.3.1.2 until no new seed points can be formed, and display the boundary image of the carriage;

[0104] Step 3.3.2: Perform corner detection on the boundary image of the carriage to determine the positions of the four corner points of the carriage;

[0105] The carriage boundary image obtained in step 3.3.1.3 is processed, and the positions of the four corner points in the image are determined by a corner detection algorithm;

[0106] In the edge region, pixels parallel to the edge direction show high similarity; however, nearby pixels perpendicular to the edge direction exhibit significant differences in grayscale values. For the corner region, the grayscale intensity of surrounding pixels changes significantly in all directions. The gradient values ​​I in the two directions of the boundary image I are calculated. x with I y The Gaussian autocorrelation matrix M is constructed by multiplying the Gaussian function G with its respective gradient and then using Gaussian weighting. Finally, the response value R is calculated using this matrix M to make the final corner point decision. The calculation formula is as follows:

[0107]

[0108]

[0109]

[0110]

[0111] This represents the convolution operator, where σ is the Gaussian scale, and G... x，y Let G be the two gradient directions of the Gaussian function G. If the two eigenvalues ​​of the autocorrelation matrix M are not significantly different, are both small and approximately equal, then they represent homogeneous locations in the image. Conversely, if they are both large and approximately equal, then they indicate corner regions.

[0112] Step 3.4: After determining the corner points, calculate the coordinates of the corner points, and calculate the stacking coordinates based on the length and width of the carriage calculated from the corner point coordinates and the dimensions of the goods to be loaded;

[0113] Step 3.4.1: Select any corner point. Based on the number of pixels and the size of the unit pixel between the corner point and the starting reference point, the actual distance between the corner point and the starting reference point can be obtained. Similarly, the actual distance between the corner point and the ending reference point can be calculated.

[0114] In step 3.4.1, based on the number of pixels at the corner point and the starting reference point, and the included angle β, the number of pixels in the horizontal and vertical directions can be obtained. Let m be the number of pixels at the corner point and the starting reference point in the horizontal direction, and let N be the actual distance between them. Then N = m * d x Let n be the number of pixels at the corner point and the starting reference point in the vertical direction, and let Q be the actual distance between them. Then Q = n * d y , where m, d x ,n,d y All data is known. Therefore, the coordinates (m, n) of the corner point in the image can be obtained. If the coordinates of the industrial robot in the world coordinate system are (x′, y′, 0), then the coordinates of the corner point in the world coordinate system are (x′+P+N, y′-Q, L-L0). Similarly, the image coordinates and world coordinates of other corner points can be obtained.

[0115] Step 3.4.2: Based on the length, height, and width information of the goods, as well as the height, length, and width of the carriage, calculate the number of stacking layers and quantity to realize the change in the quantity and type of goods placed;

[0116] Step 3.4.2.1: Let the height of the carriage be H, the width of the carriage be Y, the length of the carriage be X, the length of the cargo be x, the width of the cargo be y, and the height of the cargo be h. Then... The number of goods that can be placed in the length direction of the carriage, k1, the number of goods that can be placed in the width direction of the carriage, k2, and the number of stacking layers of goods, k3, are obtained.

[0117] Step 3.4.2.2: Determine the spatial coordinates of the center point where each item needs to be placed. The spatial coordinates are... Where (k1 = 0, 1, ... k1; k2 = 0, 1, ... k2; k3 = 0, 1, ... k3);), the calculation order of the cargo midpoint starts from the cargo at the corner position. The coordinates of the center point can be used as the position of the cargo in the carriage. If k1 = 0, k2 = 0, k3 = 0, and let m2 be the number of pixels between the center point and the origin of the image coordinate system in the horizontal direction, and let N2 be the actual distance between them, then N2 = m2 * d x Let n2 be the number of pixels between the center point and the origin of the image coordinate system in the vertical direction, and let Q2 be the actual distance between them. Then Q2 = n2 * d y Where m2, d x n2,d yGiven the known data, we can obtain the coordinates (m2, n2) of the center point in the image coordinate system. Let the coordinates of the industrial robot in the world coordinate system be (x′, y′, 0), then the coordinates of the center point in the world coordinate system are (x′+N2, y′-Q2, L-L0). Similarly, we can obtain the image coordinates and world coordinates of the other placement positions of the first layer of goods. When placing the k3rd layer of goods, its world coordinates are...

[0118] In step 4, the conveyor belt 9 transports the goods 10, and the specific steps are as follows:

[0119] Step 4.1: The conveyor belt 9 transports the goods 10 to the end of the conveyor belt. After the laser sensor 11 detects the goods 10, the conveyor belt 9 stops and sends a signal to the industrial robot 8.

[0120] Step 4.2: After the industrial robot 8 receives the signal and picks up the cargo 10, the laser sensor detects that there is no cargo 10 at the end of the conveyor belt. The conveyor belt 9 continues to run and repeats the operation when the cargo 10 reaches the end again.

[0121] In step 5, the industrial robot 8 is used to load the goods 10 onto the vehicle. The specific steps are as follows:

[0122] Step 5.1: Before loading the industrial robot 8 onto the vehicle, the industrial robot 8 needs to be powered on.

[0123] Step 5.2: After obtaining the parking location of the truck 2 and the location where the cargo 10 needs to be placed in the truck bed, the industrial robot 8 moves to the vicinity of the location where the cargo needs to be placed via the slide rail system 7;

[0124] Step 5.2.1: Establish a coordinate system with the industrial robot 8 as the center. When the industrial robot 8 is located at the origin, obtain the coordinates of the location where the goods need to be placed in the carriage from step 3.4.2.2.

[0125] Step 5.2.2: Control the industrial robot 8 according to the coordinates of the goods to grasp and stack the goods 10, and place the goods 10 at the target position;

[0126] Step 5.2.3: When the industrial robot 8 moves to the starting reference point, let m3 be the number of pixels between the position point of the placed goods and the starting reference point in the horizontal direction, and let N3 be the actual distance between them. Then N3 = m3 * d x Let n3 be the number of pixels between the location of the goods and the starting reference point in the vertical direction, and let Q3 be the actual distance between them. Then Q3 = n3 * dy , where m3, d x n3,d y Given known data, let the coordinates of the industrial robot 8 in the world coordinate system be (x′, y′, 0). Then the coordinates of the corner point in the world coordinate system are (x′+P+N3, y′-Q3, L-L0). After determining the corner point coordinates, replace the origin of the image coordinate system in step 3.4.2.2 with the starting reference point to obtain the coordinates of the location where the goods need to be placed inside the carriage.

[0127] Step 5.3: The industrial robot 8 picks up the goods 10 and loads them onto the vehicle. The conveyor belt 9 continues to run and repeats steps 4 and 5.

[0128] The above-disclosed embodiments are merely specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A control method for an automated loading system based on vision detection, characterized in that, Includes the following steps: Step 1: Measure the height of the bottom of the cargo compartment of the vehicle to be loaded from the ground and proceed to the next step; Step 2: Operate the lifting slide rail of the lifting platform (4) to raise the reference rod (5) to the same height as the bottom of the carriage, and determine the distance between unit pixels through the reference rod (5); Step 3: The images of the truck captured by the camera (6) are stitched together to obtain a complete vehicle image. The corner points of the truck are located, and the coordinates of the cargo (10) are calculated and sent to the industrial robot (8). Step 4: The conveyor belt (9) transports goods. When the goods on the conveyor belt reach the end of the conveyor belt, the laser sensor (11) detects that the goods have reached the end of the conveyor belt and transmits a signal to the industrial robot (8). The end effector of the industrial robot (8) moves to the end of the conveyor belt (9) to grip the goods (10). Step 5: Use industrial robot (8) to load goods onto the vehicle. During the continuous loading of goods (10), determine the coordinates of the position where goods (10) need to be placed in the vehicle by the number of pixels and the distance between pixels. Industrial robot (8) moves on the slide rail system (7) to continuously adjust the coordinate position of goods (10) in the vehicle and repeat steps 4 and 5. In step 5, an industrial robot (8) is used to load the goods (10) onto the vehicle. The specific steps are as follows: Step 5.1: Before loading the industrial robot (8) onto the vehicle, the industrial robot (8) needs to be powered on. Step 5.2: After obtaining the location where the truck (2) is parked and the location where the goods (10) need to be placed in the truck bed, the industrial robot (8) moves to the vicinity of the location where the goods need to be placed via the slide rail system (7); Step 5.2.1: Establish a coordinate system with the industrial robot (8) as the center. When the industrial robot (8) is located at the origin, obtain the coordinates of the position where the goods need to be placed in the carriage. Step 5.2.2: Control the industrial robot (8) according to the coordinates of the goods to grasp and stack the goods (10) and place the goods (10) at the target position; Step 5.2.3: When the industrial robot (8) moves to the starting reference point, let m3 be the number of pixels between the position point of the goods and the starting reference point in the horizontal direction, and let N3 be the actual distance between them. Then N3 = m3 d x Let n3 be the number of pixels between the location of the goods and the starting reference point in the vertical direction, and let Q3 be the actual distance between them. Then Q3 = n3 d y , where m3, d x n3,d y All of these are known data. Let the coordinates of the industrial robot (8) in the world coordinate system be (x', y', 0). Then the coordinates of the corner point in the world coordinate system are (x'+P+N3, y'-Q3, L-L0). After determining the corner point coordinates, the origin of the image coordinate system is replaced by the starting reference point to obtain the coordinates of the position where the goods need to be placed in the carriage. Step 5.3: The industrial robot (8) picks up the goods (10) and loads them onto the vehicle. The conveyor belt (9) continues to run and repeats steps 4 and 5. L0 is the distance between the top of the truck (2) and the camera (6); L is the height of the gantry. The value of L-L0 is the height of the upper surface of the carriage from the ground; P is the actual distance from the industrial robot (8) to the starting reference point.

2. The control method for an automated loading system based on vision detection according to claim 1, characterized in that: In step 1, the specific steps for measuring the height of the bottom of the cargo compartment of the truck (2) to be loaded from the ground are as follows: Step 1.1: The truck (2) enters the area of ​​the gantry (1), and the camera (6) monitors the movement of the truck (2) in real time; Step 1.2: When the truck (2) is detected to have moved to the gantry (1) and stopped moving, the laser height measuring device (3) is turned on and the distance from the top of the truck (2) to the camera (6) is obtained as L0. Let the height of the gantry be L, then the value of L-L0 is the height of the upper surface of the truck body from the ground.

3. The control method for an automated loading system based on vision detection according to claim 1, characterized in that: In step 2, the length of the unit pixel is determined by the reference rod (5). The specific steps are as follows: Step 2.1: Set three reference points with known distances on the two patterns of the reference rod (5), which are called the starting reference point, the horizontal ending reference point and the vertical ending reference point respectively. The camera (6) above the reference rod (5) obtains the image coordinates of the reference points in real time through the reference point detection algorithm. Step 2.2: Raise the reference rod (5) to the same height as the bottom of the carriage; Step 2.3: Using the image coordinates of the starting reference point and the horizontal ending reference point, we can obtain the image distance between the two image coordinates. Knowing the actual distance between the two points and the image distance, we can obtain the length of the image unit pixel in the horizontal direction. Similarly, we can obtain the length of the image unit pixel in the vertical direction. Step 2.3.1: Select the image coordinates of the starting reference point as (x1, y1), the image coordinates of the ending reference point in the horizontal direction as (x2, y2), and the image coordinates of the ending reference point in the vertical direction as (x3, y3). Step 2.3.2: Let the image distance between two points be S. Calculate the image distance between the two points using the following formula: Step 2.3.3: Let the length of a unit pixel in the horizontal direction be d. x The length of a unit pixel in the vertical direction is d. y The actual distance between the starting and ending reference points is f, and the length of a unit pixel is s. The calculation formula is as follows: The length d of a unit pixel in the horizontal direction can be obtained through the calculation formula. x The length d of a unit pixel in the vertical direction y ; The benchmark detection algorithm detects benchmarks in real time and calculates the length of each unit pixel.

4. The control method for an automated loading system based on vision detection according to claim 1, characterized in that: In step 3, a complete vehicle image is obtained through image stitching. The corner points of the cargo compartment are located using the complete vehicle image, and the coordinates of the cargo placement position are calculated. The specific steps are as follows: Step 3.1: After the truck (2) enters the parking area of ​​the gantry (1), the multiple cameras (6) begin to collect vehicle images; Step 3.2: Stitch together the vehicle images acquired from multiple locations to obtain a complete vehicle image; Step 3.3: Perform image detection on the complete vehicle image and determine the four corner points of the truck compartment using the corner detection algorithm for the compartment area; Step 3.3.1: Locate the boundary of the entire carriage in the complete image to obtain the boundary image of the carriage; Step 3.3.1.1: Select a set of original seed points at the center of the stitched carriage image; Step 3.3.1.2: Sequentially merge pixels with the same or similar properties around the seed pixel into the region where the seed pixel is located to form new seed points; Step 3.3.1.3: Repeat step 3.3.1.2 until no new seed points can be formed, and display the boundary image of the carriage; Step 3.3.2: Perform corner detection on the boundary image of the carriage to determine the positions of the four corner points of the carriage; The carriage boundary image obtained in step 3.3.1.3 is processed, and the positions of the four corner points in the image are determined by a corner detection algorithm; The gradient values ​​I in two directions of the boundary image I are obtained by calculation. x with I y The Gaussian autocorrelation matrix M is constructed by multiplying the Gaussian function G with its respective gradient and then using Gaussian weighting. Finally, the response value R is calculated using this matrix M to make the final corner point decision. The calculation formula is as follows: This represents the convolution operator, where σ is the Gaussian scale, and G... x,y Let G be the two gradient directions of the Gaussian function G. If the two eigenvalues ​​of the autocorrelation matrix M are not significantly different, are both small and approximately equal, then they represent homogeneous locations in the image. Conversely, if they are both large and approximately equal, then they indicate corner regions. Step 3.4: After determining the corner points, calculate the coordinates of the corner points, and calculate the stacking coordinates based on the length and width of the carriage calculated from the corner point coordinates and the dimensions of the goods to be loaded; Step 3.4.1: Select any corner point. Based on the number of pixels and the size of the unit pixel between the corner point and the starting reference point, the actual distance between the corner point and the starting reference point can be obtained. Similarly, the actual distance between the corner point and the ending reference point can be calculated. Based on the number of pixels at the corner point and the starting reference point, and the angle between the line connecting the starting reference point and the corner point and the horizontal direction, we can obtain the number of pixels in the horizontal and vertical directions. Let m be the number of pixels at the corner point and the starting reference point in the horizontal direction, and let N be the actual distance between them. Then N = m d x Let n be the number of pixels at the corner point and the starting reference point in the vertical direction, and let Q be the actual distance between them. Then Q = n d y , where m, d x ,n,d y All the data are known, meaning we can obtain the coordinates (m,n) of the corner point in the image. If the coordinates of the industrial robot in the world coordinate system are (x',y',0), then the coordinates of the corner point in the world coordinate system are (x'+P+N,y'-Q,L-L0). Similarly, we can obtain the image coordinates and world coordinates of other corner points. Step 3.4.2: Based on the length, height, and width information of the goods, as well as the height, length, and width of the carriage, calculate the number of stacking layers and quantity to realize the change in the quantity and type of goods placed; Step 3.4.2.1: Let the height of the carriage be H, the width of the carriage be Y, the length of the carriage be X, the length of the cargo be x, the width of the cargo be y, and the height of the cargo be h. Then... The number of goods that can be placed in the length direction of the carriage, k1, the number of goods that can be placed in the width direction of the carriage, k2, and the number of stacking layers of goods, k3, are obtained. Step 3.4.2.2: Determine the spatial coordinates of the center point where each item needs to be placed. The spatial coordinates are... Where k1 = 0, 1, ..., k1; k2 = 0, 1, ..., k2; k3 = 0, 1, ..., k3, the calculation order of the cargo midpoint starts from the cargo at the corner position. The coordinates of the center point can be used as the position of the cargo in the carriage, k1 = 0, k2 = 0, k3 = 0. Let m2 be the number of pixels between the center point and the origin of the image coordinate system in the horizontal direction, and let N2 be the actual distance between them, then N2 = m2. d x Let n be the number of pixels between the center point and the origin of the image coordinate system in the vertical direction, and let Q2 be the actual distance between them. Then Q2 = n2. d y Where m2, d x n2,d y Given known data, we can obtain the coordinates (m2, n2) of the center point in the image coordinate system. Let the coordinates of the industrial robot in the world coordinate system be (x', y', 0). Then the coordinates of the center point in the world coordinate system are (x'+N2, y'-Q2, L-L0). Similarly, we can obtain the image coordinates and world coordinates of other placement positions of the first layer of goods. When placing the k3rd layer of goods, its world coordinates are... .

5. The control method for an automated loading system based on vision detection according to claim 1, characterized in that: In step 4, the conveyor belt (9) transports the goods (10), and the specific steps are as follows: Step 4.1: The conveyor belt (9) transports the goods (10) to the end of the conveyor belt. After the laser sensor (11) detects the goods (10), the conveyor belt (9) stops and sends a signal to the industrial robot (8). Step 4.2: The industrial robot (8) receives a signal and picks up the goods (10). After the laser sensor (11) detects that there are no goods (10) at the end of the conveyor belt, the conveyor belt (9) continues to run and repeats the operation when goods (10) arrive at the end of the conveyor belt (9) again.

6. A vision-based automatic loading system implementing the vision-based automatic loading system control method according to any one of claims 1-5, comprising a vehicle height measurement system, a liftable reference pole system, a vehicle position detection system, an industrial robot movement system, and a conveyor belt cargo transportation system, characterized in that: The vehicle height measurement system includes a fixed-height gantry frame (1), a truck (2), and a laser height measurement device (3). The laser height measurement device (3) is fixed at the middle position of the crossbar of the gantry frame (1), and the laser height measurement device (3) is perpendicular to the body of the truck (2). The liftable reference rod system includes a liftable platform (4) controlled by a linear motor and a reference rod (5), wherein the reference rod (5) is fixedly connected to the lifting slide rail of the liftable platform (4); The vehicle position detection system includes multiple gantry cameras (6) and the truck (2). The cameras (6) are fixed at the position of the middle crossbar of the gantry (1) and placed from top to bottom. The truck (2) is parked within the area of ​​the gantry (1). The industrial robot mobile system consists of a slide rail system (7) and an industrial robot (8). The slide rail system (7) is located on one side of the truck (2) body. The drive motor drives the industrial robot (8) to move on the slide rail system (7). The base of the industrial robot (8) is on the same horizontal plane as the base of the liftable platform (4). The conveyor belt transport system includes a conveyor belt (9), a laser sensor (11), and cargo (10), wherein the laser sensor (11) is installed at the end of the conveyor belt (9).

Images