A forklift reversing obstacle avoidance method based on RGB-D

Abstract

The present invention discloses a forklift reversing obstacle avoidance method based on RGB-D. The method adopts an RGB-D camera to obtain depth information and RGB information of the surrounding environment, and is implemented by using a producer-consumer model. The producer is responsible for the acquisition of depth maps and RGB maps and the preprocessing of depth maps to improve the quality of depth maps. The consumer is responsible for the detection and avoidance of rear obstacles. First, ROI is used to remove most of the irrelevant data, and then the position of the obstacle is roughly extracted using background difference. Finally, the obstacle is accurately segmented using a three-dimensional threshold, and the obstacle RGB corner point information is combined to improve the detection accuracy of the obstacle and reduce the false detection rate. The method has the advantages of simple implementation, low computational consumption, fast computational speed, and low false detection rate.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and mobile robots, and in particular to an RGB-D-based forklift backward obstacle avoidance method. Background Art

[0002] Since the beginning of the 21st century, robotics has made great progress, especially in the field of intelligent unmanned handling robots. Mobile handling robots are currently making great progress in the industrial field, such as intelligent unmanned lifting AGV, intelligent unmanned stacker, and intelligent unmanned forklift. In a complex factory environment, how to detect and avoid obstacles to ensure the safety of its own operation is the prerequisite for the robot to complete its work tasks. In the obstacle avoidance function, the detection and avoidance of obstacles behind the mobile robot is an important part. At present, the detection and avoidance of obstacles behind the autonomous mobile robot generally rely on ultrasonic sensors and obstacle avoidance laser sensors. However, the ultrasonic sensor has a short detection distance, and cannot obtain the outline information of the obstacle or detect obstacles that can absorb ultrasonic waves. The obstacle avoidance laser sensor is expensive and can only obtain the laser point cloud information in one plane. For obstacles that do not appear in the plane, the obstacle avoidance laser sensor is powerless.

[0003] In summary, both ultrasonic sensors and obstacle avoidance laser sensors have certain limitations in detecting obstacles. Compared with these two sensors, RGB-D cameras have great advantages. First of all, RGB-D cameras can simultaneously obtain three-dimensional depth information and color information in the surrounding space, and can obtain the outline and point cloud information of obstacles. This also requires RGB-D cameras to be widely used in obstacle detection in autonomous mobile robots. The current RGB-D-based forklift reversing obstacle detection method has the disadvantages of large consumption of computing resources, slow processing speed, susceptibility to the influence of external lighting changes, and high false detection rate. Therefore, in view of the existing forklift reversing obstacle detection method, it is necessary to provide a forklift reversing obstacle avoidance method based on RGB-D camera. Summary of the invention

[0004] In the existing actual factory environment, in view of the problems of low real-time performance, susceptibility to external influences, high false detection rate, etc. of the existing forklift handling robot backward obstacle detection and avoidance methods, the present invention proposes a forklift backward obstacle detection method based on RGB-D. The backward obstacle detection and avoidance method based on RGB-D can detect three-dimensional obstacles in space, detect obstacles more accurately and output the coordinates of obstacles in the vehicle body coordinate system, and output the obstacle avoidance level, finally realizing the whole process of obstacle detection and avoidance.

[0005] The specific scheme of the forklift backward obstacle detection method based on RGB-D is as follows: A forklift reverse obstacle avoidance method based on RGB-D, the method comprising the following steps:

[0006] Step 1: Use the RGB-D camera installed on the forklift to obtain a three-dimensional depth image and an RGB image, and perform preprocessing on the depth image;

[0007] Step 2: Calibrate the depth information of the depth image background, specifically: obtain several pre-processed depth maps, record the maximum value of each pixel position fluctuation, obtain the threshold file; and calculate the weighted average of these depth maps to obtain the background image D background , and at the same time, the ROI area set by the current depth image is differentially operated with the background image to generate a binary image B;

[0008] Step 3: Calculate the three-dimensional spatial coordinates of the part marked as an obstacle in the binary image B, and mark the pixels that do not meet the spatial threshold condition on the binary image B as non-obstacles according to the spatial threshold set according to the state of the forklift leg;

[0009] Step 4: Calculate the convex hull of the obstacle area in the binary image and generate the minimum orthogonal rectangular frame of the convex hull;

[0010] Step 5: Define the Z axis of the camera coordinate system as the front of the camera, the X axis as the right of the camera, and the Y axis as the bottom of the camera; calculate the size of the obstacle on the Z axis and the maximum value of the obstacle in the positive direction of the Y axis;

[0011] Step 6: Set different convex hull area thresholds according to different distance intervals on the Z axis. If the convex hull area of ​​the obstacle is smaller than the convex hull area threshold of the interval in which it is located, the obstacle is discarded, otherwise it is retained;

[0012] Step 7: Select obstacles whose maximum value in the positive direction of the Y axis is close to the ground from the remaining obstacles, map the minimum orthogonal rectangle of these obstacles into the RGB image, and perform corner point detection on the corresponding content in the RGB image. If the number of detected corner points is less than the corner point detection threshold, the obstacle is discarded, otherwise it is retained.

[0013] Step 8: Convert the Z-axis value of the last remaining obstacle from the camera coordinate system to the vehicle coordinate system, and decelerate the forklift according to the obstacle coordinates in the vehicle coordinate system and the forklift model.

[0014] Further, the camera is horizontally mounted on the forklift leg, parallel to the horizontal ground, wherein the camera coordinate system is a right-hand coordinate system.

[0015] Furthermore, the depth map preprocessing in step 1 specifically includes:

[0016] Step 1-1: downsample the original depth image D twice to obtain a downsampled depth image D1;

[0017] Step 1-2: Use the improved EMA algorithm to filter D1 to obtain the processed depth image D2; the specific calculation process is: use the following improved formula to process each row and column of data on the U axis and V axis of the depth image:

[0018]

[0019] Where S h Represents the output result; Z h represents the depth value at sequence position h, where h represents the sequence position where the processing is located, and δ thread is the threshold value, which is set to 10; α is the attenuation coefficient, which is set to 0.6;

[0020] Step 1-3: Fill in some missing data of D2 based on historical data to obtain depth map D3;

[0021] Step 1-4: Use the hole filling technique to supplement the hole data of D3 to obtain the processed depth map D4.

[0022] Furthermore, the ROI area in step 2 is a preset field of view that the RGB-D camera needs to process.

[0023] Furthermore, the calculation process of the binary image in step 2 is:

[0024]

[0025] Among them, D(u,v) is the value of the acquired depth image at the pixel point (u,v), D background (u,v) is the value of the calibrated background image at the pixel (u,v), th(u,v) is the value of the threshold file at the pixel (u,v), B(u,v) is the value of the result binary image at the pixel (u,v), and the result value of 255 indicates that it is marked as an obstacle part.

[0026] Furthermore, the calculation formula for the three-dimensional space coordinates of the obstacle in step 3 is:

[0027]

[0028] in is the depth camera intrinsic parameter matrix, which is used to transform the camera coordinate system to the pixel coordinate system. x ,f y ) represents the relationship between the camera coordinate system and the imaging plane, representing the scaling on the U and V axes. (C x ,C y) is the optical center of the camera, is the pixel point (u d ,v d ) is the three-dimensional coordinate in the horizontal depth camera coordinate system, Z d is the preprocessed depth image in (u d ,v d ) pixel value.

[0029] Furthermore, in step 3, the depth pixel point (u d ,v d ) After obtaining the three-dimensional coordinates in the horizontal depth camera coordinate system, the specific formula for removing the points of obstacles that do not meet the conditions according to the spatial threshold set according to the state of the forklift legs is:

[0030]

[0031] Where B(u,v) is the value of the binary image at the pixel (u,v); X (u,v) , Y (u,v) 、z (u,v) are the spatial three-dimensional coordinates of the pixel point (u, v) in the depth map in the camera coordinate system; th1 is the threshold in the positive X direction in the camera coordinate system, X th2 is the threshold in the negative X direction in the camera coordinate system; Y th1 is the threshold in the positive Y direction in the camera coordinate system, Y th2 is the threshold in the negative Y direction in the camera coordinate system; Z th1 is the threshold in the positive Z direction in the camera coordinate system, Z th2 is the threshold in the negative Z direction in the camera coordinate system.

[0032] Furthermore, in step 5, the size of the obstacle on the Z axis is calculated using the histogram statistics method. The specific process is:

[0033] Step 5-1: Set the maximum detection distance to KMAX, and calculate the pixel point in the three-dimensional space coordinate system Z d The value is compared with KMAX, and the values ​​greater than KMAX are set to KMAX, and those less than KMAX remain unchanged;

[0034] Step 5-2: Set the distance calculation interval to [100, KMAX], divide the interval into 100 equal parts, and perform histogram statistics on the values ​​of the obstacle part on the Z axis in the binary image;

[0035] Step 5-3: Select the part where the cumulative proportion reaches more than 75% for the first time, and calculate the distance F between the obstacle and the depth camera based on this part. The calculation formula is F=0.029*i+0.01, where i is the index of the part where the cumulative proportion reaches 75%, and the calculation result is in meters.

[0036] Furthermore, the mapping formula for mapping the minimum orthogonal rectangular frame to the RGB image in step 7 is:

[0037]

[0038]

[0039]

[0040] Where Z d is the preprocessed depth image in (u d ,v d )The value of the pixel; is the pixel (u d ,v d ) in the three-dimensional coordinates of the depth camera coordinate system; is the pixel (u d ,v d ) is the three-dimensional coordinate in the RGB camera coordinate system; (u c ,v c )yes The pixel coordinates on the RGB image; R and T are the rotation matrix and translation matrix from the depth camera to the RGB camera respectively; K d is the depth camera intrinsic parameter matrix, K c is the intrinsic parameter matrix of the RGB camera.

[0041] The specific process of mapping the minimum orthogonal rectangle to the RGB image is:

[0042] Step 7-1: Set the pixel coordinates of the upper left corner of the minimum orthogonal rectangle (u tl ,v tl ) and the lower right corner coordinate (u br ,v br ) extracted;

[0043] Step 7-2: Convert the upper left pixel and the lower right pixel in the depth image into the RGB image through the mapping formula.

[0044] Step 7-3: Trim the coordinates of the corresponding upper left corner and lower right corner in the RGB image. The trimming formula is:

[0045]

[0046]

[0047] Where (u c ,v c ) is the pixel point mapped to the RGB image, width is the pixel width of the RGB image, and height is the pixel height of the RGB image.

[0048] Furthermore, the conversion formula from the camera coordinate system to the vehicle coordinate system in step 8 is:

[0049] (X o ,Y o ,Z o )=(X d ,Y d ,Z d )*R+T

[0050] Where (X o ,Y o ,Z o ) is the three-dimensional coordinate of the obstacle in the forklift body coordinate system, (X d ,Y d ,Z d ) is the three-dimensional coordinate of the obstacle in the camera coordinate system, R is the rotation matrix from the camera coordinate system to the vehicle coordinate system, and T is the translation matrix from the camera coordinate system to the vehicle coordinate system.

[0051] The beneficial effects of the present invention are as follows: the present invention provides a reliable, fast, and low-false-detection forklift backward obstacle avoidance method. The system can not only improve the quality of the depth map by preprocessing the depth map, but also perform secondary judgment on low obstacles in combination with the RGB map to improve the obstacle detection accuracy and reduce the false-detection rate. The implementation of the method adopts a producer-consumer processing mode, can make good use of multi-threading technology to process data in parallel, and has good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The figure is a general flow chart of the method of the present invention.

[0053] Figure 2 This is the depth map preprocessing flowchart.

[0054] Figure 3 Background calibration flow chart.

[0055] Figure 4 Schematic diagram of RGB-D camera installation. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0057] like Figure 1 As shown, the present invention provides a forklift reversing obstacle avoidance method based on RGB-D. The camera installation position in the method is as follows Figure 4 As shown, the angle of the camera is kept horizontal relative to the ground. This method first needs to preprocess the acquired depth map to improve the quality of the depth map. The preprocessing steps are as follows: Figure 2 As shown; the specific steps of the present invention are as follows:

[0058] Step 1: The camera is installed horizontally on the forklift leg, parallel to the horizontal ground. The RGB-D camera installed on the forklift is used to obtain a three-dimensional depth image and an RGB image, and the depth image is preprocessed to improve the quality of the depth image; specifically:

[0059] Step 1-1: Downsample the original depth map D twice to obtain a downsampled depth map D1.

[0060] Step 1-2: Use the improved EMA algorithm to filter D1 to obtain the processed depth map D2.

[0061] Step 1-3: Fill in some missing data in D2 based on historical data to obtain the depth map D3.

[0062] Step 1-4: Use the hole filling technique to supplement the hole data of D3 to obtain the processed depth map D4.

[0063] Specifically, the specific calculation process of the step-improved EMA algorithm is:

[0064] The following improved formula is used to process each row and column of data on the U axis and V axis of the depth image:

[0065]

[0066] Where S h Represents the output result; Z h represents the depth value at sequence position h, where h represents the sequence position where the processing is located, and δ thread is the threshold value, which is set to 10; α is the attenuation coefficient, which is set to 0.6;

[0067] Specifically, the hole filling technique in steps 1-4 is: selecting the largest value in the eight regions around the vacant pixel point for filling.

[0068] Step 2: Calibrate the depth information of the depth image background, specifically: obtain several pre-processed depth maps, record the maximum value of each pixel position fluctuation, obtain the threshold file; and calculate the weighted average of these depth maps to obtain the background image D background ,like Figure 3The figure shows the specific process of ground depth information calibration. At the same time, the ROI area set by the current depth image is differentially operated with the background image to generate a binary image B. The ROI area is the pre-set field of view that the camera needs to process;

[0069] , the calculation process of the binary image is:

[0070]

[0071] Among them, D(u,v) is the value of the depth map obtained at the pixel point (u,v), D background (u,v) is the value of the calibrated background image at the pixel (u,v); th(u,v) is the value of the threshold file at the pixel (u,v); B(u,v) is the value of the resulting binary image at the pixel (u,v), and a result value of 255 indicates that it is marked as an obstacle part.

[0072] Step 3: When D is obtained background And the threshold file starts to detect obstacles and calculate the distance to the obstacle. Calculate the three-dimensional spatial coordinates of the depth pixel points marked as obstacles in the binary image B, and mark the positions of the pixels that do not meet the threshold conditions on B as non-obstacles according to the state of the fork leg and the spatial thresholds set under different fork leg states. There are three states of the fork leg: raised state (UP), lowered state (DOWN), and unknown state (UNKNOW). The calculation formula for the three-dimensional spatial coordinates of the obstacle is:

[0073]

[0074] in is the depth camera intrinsic parameter matrix, which is used to transform the camera coordinate system to the pixel coordinate system. x ,f y ) represents the relationship between the camera coordinate system and the imaging plane, representing the scaling on the U and V axes. (C x ,C y ) is the optical center of the camera, is the pixel point (u d ,v d ) is the three-dimensional coordinate in the horizontal depth camera coordinate system, Z d is the preprocessed depth image in (u d ,v d ) pixel value.

[0075] Get the depth pixel point (u d ,v d ) After obtaining the three-dimensional coordinates in the horizontal depth camera coordinate system, the specific formula for removing the points of obstacles that do not meet the conditions according to the spatial threshold set according to the state of the forklift legs is:

[0076]

[0077] Where B(u,v) is the value of the binary image at the pixel (u,v); X (u,v) , Y (u,v) , Z (u,v) are the spatial three-dimensional coordinates of the pixel point (u, v) in the depth map in the camera coordinate system; th1 is the threshold in the positive X direction in the camera coordinate system, X th2 is the threshold in the negative X direction in the camera coordinate system; Y th1 is the threshold in the positive Y direction in the camera coordinate system, Y th2 is the threshold in the negative Y direction in the camera coordinate system; Z th1 is the threshold in the positive Z direction in the camera coordinate system, Z th2 is the threshold in the negative Z direction in the camera coordinate system.

[0078] Step 4: Calculate the convex hull of the obstacle area in the binary image and generate the minimum orthogonal rectangular frame of the convex hull;

[0079] Step 5: The camera coordinate system is a right-handed coordinate system. Define the Z axis of the camera coordinate system as the front of the camera, the X axis as the right of the camera, and the Y axis as the bottom of the camera. Use the histogram statistics method to calculate the size of the obstacle on the Z axis, and at the same time calculate the minimum value of the obstacle on the Y axis. The specific process of using the histogram statistics method to calculate the size of the obstacle on the Z axis is as follows:

[0080] Step 5-1: Set the maximum detection distance to KMAX, and calculate the pixel point in the three-dimensional space coordinate system Z d The value is compared with KMAX, and the values ​​greater than KMAX are set to KMAX, and those less than KMAX remain unchanged;

[0081] Step 5-2: Set the distance calculation interval to [100, KMAX], divide the interval into 100 equal parts, and perform histogram statistics on the values ​​of the obstacle part on the Z axis in the binary image;

[0082] Step 5-3: Select the part where the cumulative proportion reaches more than 75% for the first time, and calculate the distance F between the obstacle and the depth camera based on this part. The calculation formula is F=0.029*i+0.01, where i is the index of the part where the cumulative proportion reaches 75%, and the calculation result is in meters.

[0083] Step 6: Different convex hull area thresholds are set according to different distance intervals on the Z axis. If the convex hull area of ​​the obstacle is smaller than the convex hull area threshold of the interval in which it is located, the obstacle is discarded, otherwise it is retained.

[0084] Step 7: Select obstacles whose minimum Y-axis value is close to the ground from the remaining obstacles, map the minimum orthogonal rectangle of these obstacles into the RGB image, and perform corner point detection on the corresponding content in the RGB image. If the number of detected corner points is less than the corner point detection threshold, the obstacle is discarded, otherwise it is retained.

[0085] The specific calculation formula of the minimum orthogonal rectangular frame is:

[0086]

[0087]

[0088] Where (u lt ,v lt ) is the coordinate of the upper left corner of the minimum orthogonal rectangle in the depth map, (u rb ,v rb ) is the coordinate of the lower right corner of the minimum orthogonal rectangle in the depth map; P is the set of points marked as obstacles.

[0089] The mapping formula for mapping the minimum orthogonal rectangle in the depth map to the RGB map is:

[0090]

[0091]

[0092]

[0093] Where Z d is the depth map D4 in (u d ,v d )The value of the pixel; is the pixel (u d ,v d ) in the three-dimensional coordinates of the depth camera coordinate system; is the pixel (u d ,v d ) is the three-dimensional coordinate in the RGB camera coordinate system; (u c ,v c )yes The pixel coordinates on the RGB image. R and T are the rotation matrix and translation matrix from the depth camera to the RGB camera, respectively. d is the depth camera intrinsic parameter matrix, K c is the intrinsic parameter matrix of the RGB camera.

[0094] After obtaining the minimum orthogonal rectangular frame, further processing is required because the RGB camera and the depth camera have inconsistent fields of view. The specific steps of the processing are as follows: tl ,vtl ) and the lower right corner coordinate (u br ,v br ) is extracted. The upper left corner pixel and the lower right corner pixel in the depth map are converted into the RGB map. The corresponding upper left corner and lower right corner coordinates in the RGB map are trimmed. The trimming formula is:

[0095]

[0096]

[0097] Where (u c ,v c ) is the pixel mapped to the RGB image, width is the pixel width of the RGB image, and height is the pixel height of the RGB image.

[0098] Finally, the FAST corner detection algorithm is performed on the mapped RGB data. The specific processing process is: setting the corner detection threshold and performing FAST corner detection; performing maximum suppression on the number of corner points; retaining obstacles whose number of corner points is less than the threshold, otherwise discarding them.

[0099] Step 8: Convert the Z-axis value of the last remaining obstacle from the camera coordinate system to the vehicle coordinate system, and decelerate the forklift according to the obstacle coordinates in the vehicle coordinate system and the forklift model.

[0100] Specifically, the conversion formula from the camera coordinate system to the vehicle coordinate system is:

[0101] (X o ,Y o ,Z o )=(X d ,Y d ,Z d )*R+T

[0102] Where (X o ,Y o ,Z o ) is the three-dimensional coordinate of the obstacle in the forklift body coordinate system, (X d ,Y d ,Z d ) is the three-dimensional coordinate of the obstacle in the camera coordinate system, R is the rotation matrix from the camera coordinate system to the vehicle coordinate system, and T is the translation matrix from the camera coordinate system to the vehicle coordinate system.

[0103] An example of a deceleration control process in step 8 of the present invention is as follows: after obtaining the coordinates of the obstacle relative to the forklift body coordinate system, the following strategy is executed: when the distance between the obstacle and the forklift center is calculated to be 4m, it is a first-level obstacle avoidance, and the forklift is decelerated to 60% of the current speed; when the distance between the obstacle and the forklift center is calculated to be 2m, it is a second-level obstacle avoidance, and the forklift is decelerated to 30% of the current speed; when the distance between the obstacle and the forklift center is calculated to be 1.2m, it is a third-level obstacle avoidance, and the forklift is decelerated to 0% of the current speed. Finally, the required angular velocity and linear velocity are calculated according to the obstacle avoidance level and the forklift model to control the deceleration of the forklift, and finally the obstacle detection and obstacle avoidance functions are realized.

[0104] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A forklift reversing obstacle avoidance method based on RGB-D, characterized in that , the method described comprises the following steps: Step 1: Use the RGB-D camera installed on the forklift to obtain a three-dimensional depth image and an RGB image, and perform preprocessing on the depth image; Step 2: Calibrate the depth information of the depth image background, specifically: obtain several pre-processed depth maps, record the maximum value of each pixel position fluctuation, obtain the threshold file; and calculate the weighted average of these depth maps to obtain the background image D background , and at the same time, the ROI area set by the current depth image is differentially operated with the background image to generate a binary image B; Step 3: Calculate the three-dimensional spatial coordinates of the part marked as an obstacle in the binary image B, and mark the pixels that do not meet the spatial threshold condition on the binary image B as non-obstacles according to the spatial threshold set according to the state of the forklift leg; Step 4: Calculate the convex hull of the obstacle area in the binary image and generate the minimum orthogonal rectangular frame of the convex hull; Step 5: Define the Z axis of the camera coordinate system as the front of the camera, the X axis as the right of the camera, and the Y axis as the bottom of the camera; calculate the size of the obstacle on the Z axis and the maximum value of the obstacle in the positive direction of the Y axis; Step 6: Set different convex hull area thresholds according to different distance intervals on the Z axis. If the convex hull area of ​​the obstacle is smaller than the convex hull area threshold of the interval in which it is located, the obstacle is discarded, otherwise it is retained; Step 7: Select obstacles whose maximum value in the positive direction of the Y axis is close to the ground from the remaining obstacles, map the minimum orthogonal rectangle of these obstacles into the RGB image, and perform corner point detection on the corresponding content in the RGB image. If the number of detected corner points is less than the corner point detection threshold, the obstacle is discarded, otherwise it is retained. Step 8: Convert the Z-axis value of the last remaining obstacle from the camera coordinate system to the vehicle coordinate system, and decelerate the forklift according to the obstacle coordinates in the vehicle coordinate system and the forklift model.

2. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The camera is installed horizontally on the forklift leg, parallel to the horizontal ground, where the camera coordinate system is a right-hand coordinate system.

3. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The depth map preprocessing in step 1 specifically includes: Step 1-1: downsample the original depth image D twice to obtain a downsampled depth image D1; Step 1-2: Use the improved EMA algorithm to filter D1 to obtain the processed depth image D2; the specific calculation process is: use the following improved formula to process each row and column of data on the U axis and V axis of the depth image: Where S h Represents the output result; Z h represents the depth value at sequence position h, where h represents the sequence position where the processing is located, and δ thread is the threshold value, which is set to 10; α is the attenuation coefficient, which is set to 0.6; Step 1-3: Fill in some missing data of D2 based on historical data to obtain depth map D3; Step 1-4: Use the hole filling technique to supplement the hole data of D3 to obtain the processed depth map D4.

4. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The ROI area in step 2 is the pre-set field of view that the RGB-D camera needs to process.

5. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The calculation process of the binary image in step 2 is: Among them, D(u,v) is the value of the acquired depth image at the pixel point (u,v), D background (u,v) is the value of the calibrated background image at the pixel (u,v), th(u,v) is the value of the threshold file at the pixel (u,v), B(u,v) is the value of the result binary image at the pixel (u,v), and the result value of 255 indicates that it is marked as an obstacle part.

6. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The calculation formula for the three-dimensional space coordinates of the obstacle in step 3 is: in is the depth camera intrinsic parameter matrix, which is used to transform the camera coordinate system to the pixel coordinate system. x ,f y ) represents the relationship between the camera coordinate system and the imaging plane, representing the scaling on the U and V axes. (C x ,C y ) is the optical center of the camera, is the pixel point (u d ,v d ) is the three-dimensional coordinate in the horizontal depth camera coordinate system, Z d is the preprocessed depth image in (u d ,v d ) pixel value.

7. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: In step 3, obtain the depth pixel point (u d ,v d ) After obtaining the three-dimensional coordinates in the horizontal depth camera coordinate system, the specific formula for removing the points of obstacles that do not meet the conditions according to the spatial threshold set according to the state of the forklift legs is: Where B(u,v) is the value of the binary image at the pixel (u,v); X (u,v) , Y (u,v) , Z (u,v) are the spatial three-dimensional coordinates of the pixel point (u, v) in the depth map in the camera coordinate system; th1 is the threshold in the positive X direction in the camera coordinate system, X th2 is the threshold in the negative X direction in the camera coordinate system; Y th1 is the threshold in the positive Y direction in the camera coordinate system, Y th2 is the threshold in the negative Y direction in the camera coordinate system; Z th1 is the threshold in the positive Z direction in the camera coordinate system, Z th2 is the threshold in the negative Z direction in the camera coordinate system.

8. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: In step 5, the size of the obstacle on the Z axis is calculated using the histogram statistics method. The specific process is as follows: Step 5-1: Set the maximum detection distance to KMAX, and calculate the pixel point in the three-dimensional space coordinate system Z d The value is compared with KMAX, and the values ​​greater than KMAX are set to KMAX, and those less than KMAX remain unchanged; Step 5-2: Set the distance calculation interval to [100, KMAX], divide the interval into 100 equal parts, and perform histogram statistics on the values ​​of the obstacle part on the Z axis in the binary image; Step 5-3: Select the part where the cumulative proportion reaches more than 75% for the first time, and calculate the distance F between the obstacle and the depth camera based on this part. The calculation formula is F=0.029*i+0.01, where i is the index of the part where the cumulative proportion reaches 75%, and the calculation result is in meters.

9. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The mapping formula for mapping the minimum orthogonal rectangle to the RGB image in step 7 is: Where Z d is the preprocessed depth image in (u d ,v d )The value of the pixel; is the pixel (u d ,v d ) in the three-dimensional coordinates of the depth camera coordinate system; is the pixel (u d ,v d ) is the three-dimensional coordinate in the RGB camera coordinate system; (u c ,v c )yes The pixel coordinates on the RGB image; R and T are the rotation matrix and translation matrix from the depth camera to the RGB camera respectively; K d is the depth camera intrinsic parameter matrix, K c is the intrinsic parameter matrix of the RGB camera; The specific process of mapping the minimum orthogonal rectangle to the RGB image is: Step 7-1: Set the pixel coordinates of the upper left corner of the minimum orthogonal rectangle (u tl ,v tl ) and the lower right corner coordinate (u br ,v br ) extracted; Step 7-2: Convert the upper left corner pixel and the lower right corner pixel in the depth image into the RGB image through the mapping formula; Step 7-3: Trim the coordinates of the corresponding upper left corner and lower right corner in the RGB image. The trimming formula is: Where (u c ,v c ) is the pixel point mapped to the RGB image, width is the pixel width of the RGB image, and height is the pixel height of the RGB image.

10. The RGB-D based forklift reversing obstacle avoidance method according to claim 1, characterized in that: The conversion formula from the camera coordinate system to the vehicle coordinate system in step 8 is: (X o ,Y o ,Z o )=(X d ,Y d ,Z d )*R+T Where (X o ,Y o ,Z o ) is the three-dimensional coordinate of the obstacle in the forklift body coordinate system, (X d ,Y d ,Z d ) is the three-dimensional coordinate of the obstacle in the camera coordinate system, R is the rotation matrix from the camera coordinate system to the vehicle coordinate system, and T is the translation matrix from the camera coordinate system to the vehicle coordinate system.

Images