Computer vision based transport device alignment guidance system and method

Abstract

The application discloses a kind of transport equipment alignment guide system and method based on computer vision, guide system includes camera module, network module, server module and display module, wherein: the camera module is used to real-time acquisition transport equipment video stream data, and is transmitted to server module by network module;The network module includes network switch and gigabit optical fiber, and network switch is connected with camera module, server module, display module and hoisting equipment PLC respectively by gigabit optical fiber;The server module calculates the transport equipment guide information using the video stream data photographed by camera module and outputs hoisting equipment PLC and display module.The application can accurately position the position information of transport equipment and container, and output the accurate distance of its arrival specified location to hoisting equipment PLC and external display screen in time.The application can realize full-automatic guide, greatly reduce the cost and security risk in manual guide process.

Description

TECHNICAL FIELD

[0001] The present application relates to a computer vision-based transportation equipment alignment guiding system and method. BACKGROUND

[0002] In the process of container loading and unloading at the wharf, in order to ensure the efficiency of loading and unloading and the accuracy of the stopping position of the transportation equipment during loading and unloading, the following methods are usually adopted: 1. Draw guiding lines on both sides of the hoisting equipment, and guide the transportation equipment in the lane direction by the driver observing the guiding lines. 2. Guide by the command hand of the transportation equipment. It can be seen that the above two guiding methods both rely on manual guidance, and therefore have the following disadvantages: 1. The guiding position is uncertain each time, and there is often a little misalignment between the spreader lock head and the container lock hole after the guiding is completed, which causes the operator to repeatedly adjust the position of the spreader when grabbing the container. In severe cases, the spreader may repeatedly collide with the container, causing damage to the lock hole of the container and the lock head of the spreader. 2. The traditional manual guiding method requires the command hand of the transportation equipment to be exposed to the working environment for a long time, which not only consumes labor but also has safety risks. However, the current market is dominated by laser-based guiding systems, which generally have high hardware costs, long positioning times, and limited sensing range. The visual-based guiding system can reduce costs while ensuring performance, has a large sensing range, rich information, limited application scenarios, and the advantages of easy installation and debugging. SUMMARY

[0003] In order to overcome the above-mentioned shortcomings of the prior art, the present application proposes a computer vision-based transportation equipment alignment guiding system and method, aiming to solve the problems of inaccurate manual guidance, labor consumption, and potential safety risks in the prior art.

[0004] The technical solution adopted by the present application to solve its technical problems is: a computer vision-based transportation equipment alignment guiding system, comprising a camera module, a network module, a server module, and a display module, wherein:

[0005] The camera module is used to collect video stream data of the transportation equipment in real time and transmit it to the server module through the network module;

[0006] The network module includes a network switch and a gigabit optical fiber, and the network switch is connected with the camera module, the server module, the display module, and the PLC of the hoisting equipment through the gigabit optical fiber;

[0007] The server module calculates the transportation equipment guiding information from the video stream data captured by the camera module and outputs it to the PLC of the hoisting equipment and the display module.

[0008] The application also provides a computer vision-based alignment guiding method for a transport device, comprising the following steps:

[0009] Step one, the camera module collects video stream data of the transport device on the work lane in real time, and transmits the data to the server module in real time through the network module;

[0010] Step two, the server module splits the obtained video stream data, and corrects the data frame by frame;

[0011] Step three, the server module splices the images of the two corrected cameras to obtain a God's eye view image with a shooting effect approximately above the lane;

[0012] Step four, the server module performs instance segmentation on the spliced image to detect the position information of the transport device and the container in the image;

[0013] Step five, the server module determines the guiding distance by using the fitted outer rectangle of the transport device and the container in step four;

[0014] Step six, the double-container spacing when loading double containers is calculated:

[0015] When the current operation is two containers, the detected front and rear edges of the two containers are used to detect the double-container spacing;

[0016] Step seven, the container off-loading angle when the transport device and the container are parked is calculated:

[0017] The long side is used as a reference for calculating the off-loading angle of the transport device and the container, the off-loading angle of the current transport device and container is calculated according to the slope a of the fitted linear equation, and the off-loading angle corresponding to the current slope a is calculated using the inverse trigonometric function arctan(a);

[0018] Step eight, the server module outputs the calculated transport device guiding information to the crane PLC and the display module through the network module;

[0019] Step nine, the server module determines whether the minimum distance of the transport device guiding supporting the operation is reached according to the current guiding information: if not, the step two is returned; if yes, the guiding completion information is output.

[0020] Compared with the prior art, the application has the following positive effects:

[0021] The present application adopts computer vision-based image processing algorithm, and combines advanced image segmentation technology to realize regional detection and segmentation of operation transport equipment and containers. The present application collects video images in real time through a camera installed above the lane of the hoisting equipment. Through detection of the transport equipment and containers in the picture, the required transport equipment guide information during operation is output, effectively solving the problem of production efficiency caused by manual multiple guidance.

[0022] The present application fully considers and combines the actual operation situation of hoisting equipment, and based on computer vision detection and segmentation technology of video stream, combines the application of server, Ethernet and camera. The transport equipment guidance process of the automated wharf is analyzed and detected in real time. It is ensured that the transport equipment can efficiently and accurately reach the designated operation position in the automated process, and the crane operator can more directly adjust the attitude of the spreader. The present application can replace the traditional manual guidance mode to realize full-automatic guidance.

[0023] The present application fully utilizes computer vision technology to realize real-time detection and tracking of transport equipment and containers, and outputs key information such as target position and attitude. Through image processing and calculation, the position information of the transport equipment and containers can be accurately positioned, and the accurate distance from the transport equipment and containers to the designated position is output to the hoisting equipment PLC and external display screen in time. This can replace the traditional manual guidance mode to realize full-automatic guidance, greatly reducing the cost and safety hazards in the manual guidance process. BRIEF DESCRIPTION OF DRAWINGS

[0024] The present application will be described by way of example and with reference to the accompanying drawings, in which:

[0025] Figure 1 is a schematic diagram of the system structure of the present application. DETAILED DESCRIPTION

[0026] As shown in Figure 1 , a transport equipment guidance system and method based on computer vision includes a camera module, a network module, a server module and a display module, wherein:

[0027] The camera module is two high-definition cameras for collecting video stream data of the transport equipment in real time and transmitting to the server module through the network module. The camera is installed at the height of the hoisting equipment's trolley platform of the automated wharf, which makes the camera's field of view completely cover the transport lane below. The angle of the camera is adjusted so that it can shoot the operation lane below the hoisting equipment. The camera is installed diagonally, which can collect lane information in all directions.

[0028] The network module comprises a network switch and a gigabit optical fiber, and the network switch is connected with the camera module, the server module, the display module and the hoisting equipment PLC through the gigabit optical fiber.

[0029] The server module analyzes and detects the video stream data photographed by the camera module and applies calculation on the detection result, and outputs the guide distance required in the operation guide process of the transportation equipment, the spreader guide distance of the hoisting equipment, the double-box distance, the container load angle and other information to the device PLC and the display module.

[0030] The display module comprises two LED display screens.

[0031] The application also provides a transportation equipment guide method based on computer vision, which comprises the following steps:

[0032] Step one, the camera module collects the video stream data of the transportation equipment on the operation lane in real time, and transmits the video stream data to the server module in real time through the network module;

[0033] Step two, the server module splits the obtained video stream data and corrects frame by frame:

[0034] The correction mainly adopts a correction method for radial distortion: the first several terms of Taylor series expansion around the optical axis center are used for description. Since the distortion of the camera used in the application is small, the first two terms are used for description, so that the coordinates before correction and the coordinates after correction are corresponded:

[0035] x0=x(1+k1r 2 +k2r 4 )

[0036] y0=y(1+k1r 2 +k2r 4 )

[0037] Wherein x and y represent the coordinates in the original picture, and x0 and y0 represent the coordinates in the corrected picture.

[0038] Step three, the server module splices the images of the two cameras after correction to obtain the God's eye view image of the approximate photographing effect above the lane:

[0039] The main steps of splicing are as follows: 1. SIFT algorithm is used to detect feature points of the transport equipment and containers in the two images captured by the two cameras on both sides. 2. The feature points of the transport equipment and containers detected in the two images are matched. 3. The RANSAC algorithm is used to filter the matched feature point set, and the abnormal points caused by changes in information such as light are excluded. 4. The remaining similar feature points are used to perform perspective matrix transformation on the images, so that the transport equipment and containers in the two images are changed to the same view angle and size. 5. The two images with the same view angle are spliced in front of and behind each other according to the running direction of the transport equipment. 6. The joint part of the spliced container is processed, mainly the part with large color difference, to make it smooth transition.

[0040] Step four, the server module performs instance segmentation on the corrected and spliced image, and uses linear fitting to find the minimum circumscribed rectangle of the segmentation box, detects the position information of the transport equipment and container targets in the image, and the specific method is as follows:

[0041] First, the transport equipment and container are detected by the segmentation algorithm, and the detection box is filtered. When filtering, the standard non-maximum suppression algorithm is used to filter and remove the candidate boxes with low similarity. The standard non-maximum suppression function is as follows:

[0042]

[0043] Where N t represents the set threshold. Iou is the intersection over union, and iou(M, b i ) is the intersection over union of M (the maximum box of the current category) and b i (other boxes of the current category). S i is the score of the current box.

[0044] Second, the motion position of the transport equipment in each frame is recorded for the remaining detection boxes after filtering, and the running direction of the current transport equipment is determined through the change information of the transport equipment motion trajectory.

[0045] Third, the current work lane is calculated according to the information of the detected transport equipment and container in the current image.

[0046] Fourth, the transport equipment and container are segmented, and the minimum circumscribed rectangle of the transport equipment and container is found in a linear fitting manner. The specific operation is as follows: according to the detection point data set of the transport equipment and container output by the detection algorithm, all points of the single side of the container are found, and the coordinates of these points are assumed to be (x1, y1), (x2, y2), (x3, y3),..., (xn, yn). The specific steps of the RANSAC algorithm are as follows:

[0047] 1) Randomly select a certain number of data points as the sample set for the current fitting. Assume there are m points in the sample set, where m ≥ 2.

[0048] 2) Calculate the initial parameters a and b of the fitting straight line using two points in the sample set. This can be obtained by calculating the slope and intercept between the two points.

[0049] a = (y2 - y1) / (x2 - x1)

[0050] b = y1 - a * x1

[0051] 3) Bring other data points into the equation of the fitting straight line one by one, and calculate their distances to the fitting straight line. The formula for calculating the distance is:

[0052] distance_i = |y_i - (a * x_i + b)|

[0053] distance_i: represents the distance of the ith data point to the fitting straight line;

[0054] 4) Set a distance threshold threshold, and divide the data points into an inlier set and an outlier set according to the threshold. For data points with a distance less than the threshold, they are divided into the inlier set, otherwise they are divided into the outlier set.

[0055] 5) If the number of inliers exceeds a certain threshold, consider the current fitting result to be reliable. The coordinates of the points in the inlier set can be used to calculate new fitting straight line parameters a_new and b_new. The method can be to take the average of the points in the inlier set.

[0056] a_new = (∑(x_i) / num_inliers)

[0057] b_new = (∑(y_i) / num_inliers)

[0058] where a_new, b_new: represent the new fitting straight line parameters calculated using the inlier set

[0059] num_inliers represents the number of inliers, i.e. the number of data points with a distance less than the threshold.

[0060] Repeat the above steps several times to obtain multiple sets of fitting straight line parameters a and b. Select the set of parameters with the most inliers as the final fitting result.

[0061] Step five, the server module uses the bounding rectangle of the transport equipment and the container fitted out in step four to judge the distance of the guide line using the single side near the guide line and parallel to the guide line.

[0062] 1) In the current edge and guide line, randomly select 40 groups of points, respectively calculate the distance between points, and calculate the average distance by the form of mean square deviation. The specific mean square deviation formula is as follows:

[0063]

[0064] Where: n represents the number of points, x i represents the value of the current point, The average value of 40 points is represented by 40 40

[0065] 2) Use the median: the mean square deviation is affected by the abnormal value to a certain extent. If there are many abnormal values in the data, the mean square deviation may not be stable enough. It can be considered to use the median instead of the mean value calculation, and the median is less affected by the abnormal value.

[0066] The median calculation formula is: L = median(|x_i-x - |)

[0067] Where, x - represents the median of the point.

[0068] 3) Remove abnormal points: before calculating the average distance, abnormal point detection can be performed first, and points with distance far from the mean value are considered as abnormal points and are removed. This can effectively exclude the interference of abnormal values and obtain more stable average distance.

[0069] 4) Adjust the sampling number: currently, 40 groups of points are randomly selected for distance calculation. The sampling number can be adjusted according to the specific situation. Increasing the sampling number may increase the complexity and calculation time of the calculation, so the balance between performance and accuracy needs to be considered.

[0070] 5) Consider different sides: in actual situation, the transportation equipment and container may have different shapes and offset conditions on different sides. Therefore, the distance of different sides of the transportation equipment and container can be judged respectively, and then the distance information of multiple sides is integrated to make a more comprehensive guide distance judgment.

[0071] For distance judgment of different sides, the same mean square deviation formula can be used to calculate the average distance of each side.

[0072] When integrating distance information, the weight of distance of different sides can be set according to the specific application requirements, so as to more reasonably determine the guide distance.

[0073] Step six, calculate the double container spacing when loading double containers:

[0074] ​When the current operation is two containers, the detected front and rear edges of the two containers are used to detect the double-container spacing. The specific calculation method also uses the mean square error formula.

[0075] Step seven, calculate the container offloading angle when the transport equipment is parked:

[0076] Using the long side as the reference for calculating the container offloading angle, the offloading angle of the current two containers is calculated according to the slope a in the fitted linear equation: y = ax + b. The specific calculation method is: using the inverse trigonometric function arctan(a) to calculate the offloading angle corresponding to the current slope a.

[0077] Step eight, the server module outputs the calculated transport equipment guide information (including: guide distance when the transport equipment is running, spreader guide distance, double-container spacing when loading double containers, container offloading angle when the transport equipment is parked, etc.) to the spreader PLC (Programmable Logic Controller) and display module through the network module; the display module displays the transport equipment guide information, spreader guide information, double-container spacing and container offloading angle data through the external display screen for real-time viewing by relevant operating personnel.

[0078] Step nine, the server module judges whether the minimum transport equipment guide distance supporting the operation is reached according to the current guide information: if not, return to step two; if yes, output the guide completion information.

Claims

1. A guiding method of a computer vision-based guiding system for aligning a transport device, characterized by: The guiding system comprises a camera module, a network module, a server module and a display module, wherein: the camera module is used for collecting video stream data of the transportation equipment in real time and transmitting the video stream data to the server module through the network module; the network module comprises a network switch and a gigabit optical fiber, the network switch is connected with the camera module, the server module, the display module and the PLC of the hoisting equipment through the gigabit optical fiber; the server module calculates the alignment guiding information of the transportation equipment by using the video stream data shot by the camera module and outputs the alignment guiding information to the PLC of the hoisting equipment and the display module; the guiding method comprises the following steps: Step one, the camera module collects the video stream data of the transportation equipment on the work lane in real time, and transmits the video stream data to the server module in real time through the network module; Step two, the server module splits the obtained video stream data, and corrects frame by frame; Step three, the server module splices the images of the two cameras after correction, and obtains the god's view image with the effect of shooting directly above the lane; Step four, the server module performs instance segmentation on the image after correction and splicing, and detects the position information of the transportation equipment and the container in the image: Firstly, the transportation equipment and the container are detected through a segmentation algorithm, and the detection frame is screened to remove the candidate frame with similarity less than a set threshold; Secondly, the motion position of the transportation equipment in each frame is recorded, and the running direction of the current transportation equipment is determined through the change information of the motion track of the transportation equipment; Thirdly, the current work lane is calculated according to the information of the transportation equipment and the container detected in the current image; Fourthly, the minimum circumscribed rectangle of the transportation equipment and the container is found in a linear fitting manner: 1) find all points of the single side edge of the container from the detection point position data set of the transportation equipment and the container, and the coordinates are (x1, y1), (x2, y2), (x3, y3),... (xn, yn); randomly select m points as the current fitting sample set, wherein m ≥ 2; 2) calculate the initial parameters a and b of the fitting straight line using two points in the sample set: a = (y2 - y1) / (x2 - x1) b = y1 - a * x1 3) bring other data points into the equation of the fitting straight line one by one, and calculate their distances to the fitting straight line: distance_i = |y_i - (a * x_i + b)| distance_i: represents the distance of the ith data point to the fitting straight line; 4) set a distance threshold threshold, and divide the data points into an inner point set and an outer point set according to the threshold; for the data points with a distance less than the threshold, they are divided into the inner point set, otherwise they are divided into the outer point set; 5) if the number of the inner point set exceeds a set threshold, calculate the new fitting straight line parameters a_new and b_new using the coordinates of the points in the inner point set: a_new = (∑(x_i) / num_inliers) b_new = (∑(y_i) / num_inliers) Wherein, a_new, b_new: represent the new fitting line parameters calculated using the inlier set, num_inliers represents the number of inliers; Repeat the above steps several times to obtain multiple sets of fitting line parameters a and b, and select the set of parameters with the largest number of inliers as the final fitting result; Step five, the server module determines the guide distance using the fitted outer rectangle of the transport equipment and the container in step four; Step six, calculate the double container spacing when loading double containers: When the current operation is double container operation, use the detected front and rear edges of the two containers to detect the double container spacing; Step seven, calculate the container off-loading angle when the transport equipment is parked with the container: Use the long side as the reference for calculating the container off-loading angle, and calculate the off-loading angle of the current transport equipment and container according to the slope a of the fitted linear equation, and use the inverse trigonometric function arctan(a) to calculate the off-loading angle corresponding to the current slope a; Step eight, the server module outputs the calculated transport equipment guide information to the PLC and display module of the hoisting equipment through the network module; Step nine, the server module determines whether the minimum distance of the transport equipment guide supporting the operation is reached according to the current guide information: if not, return to step two; if yes, output the guide completion information.

2. The guiding method of a computer vision-based transport equipment alignment guide system according to claim 1, characterized by: The camera module is two high-definition cameras installed in a diagonal manner at the height of the hoisting equipment's winding point platform on the automated wharf, used to shoot the operation data below the hoisting equipment.

3. The method of claim 1, wherein: The transport equipment alignment guide information includes: guide distance when the transport equipment is running, spreader guide distance of the hoisting equipment, double container spacing when loading double containers, and container off-loading angle when the transport equipment is parked with the container.

4. The method of claim 1, wherein: When correcting, the correction method for radial distortion is used, and the first two items of the Taylor series expansion around the optical axis center are used to describe.

5. The method of claim 1, wherein: The method for splicing is: ① Use SIFT algorithm to detect the feature points of the transport equipment and the container in the two images shot by the two cameras on both sides; ② Match the detected feature points of the transport equipment and the container in the two images; ③ Use RANSAC algorithm to filter the matched feature point set to exclude abnormal points; ④ Change the perspective matrix of the images to change the transport equipment and the container in the two images to the same perspective and size through the remaining similar feature points; ⑤ Splice the two images with the same perspective according to the running direction of the transport equipment; ⑥ Smooth the transition of the joint part of the spliced container.

6. The method of guiding a computer vision-based transport device alignment guidance system of claim 1, wherein: The method for determining the guide distance in step five is: randomly selecting several groups of points on each single side of the guide line and the side near the guide line and parallel to the guide line, respectively calculating the distance between the points, and calculating the average distance in the form of mean square deviation; when the number of abnormal values in the data is greater than a set threshold, using the median to replace the mean for calculation; when the container or the bracket has different shapes and offset conditions on different sides, using the same mean square deviation formula for different sides of the container or the bracket to calculate the average distance of each side, then setting the weight of the distance of different sides according to the specific application requirement, and comprehensively considering the distance information of multiple sides, so as to more reasonably determine the guide distance.

7. The method of guiding a computer vision-based transport device alignment guide system of claim 6, wherein: Before calculating the average distance, the abnormal point detection is carried out, and the point with a distance far away from the mean is regarded as an abnormal point and is eliminated.

Images