Positioning and Tracking Method and System for Multi-ocular Vision AGV

Abstract

The present invention discloses a method and system for AGV positioning and tracking based on multi-camera vision. The method and system include the following steps: S1, a video acquisition step; S2, a video image processing step; S3, a video image feature extraction step; S4, a map stitching step; S5, an object recognition step; S6, a target positioning and tracking step. By using the present invention, the problems that the AGV based on global vision navigation is blocked by obstacles or the positioning fails when a single camera is used can be solved. It can more accurately position and track the AGV, effectively improve the transportation efficiency, reduce the transportation cost, and is of great significance for the intelligentization of industries such as intelligent warehousing and unmanned handling.

Description

Technical Field

[0001] The present invention designs a positioning and tracking method for an AGV based on multi - vision, and particularly relates to a positioning and tracking method and system for an AGV based on multi - vision. Background Art

[0002] At present, the automation and intelligence of industrial production are still important directions for the development of our country. The automatic guided vehicle (AGV) is an important branch of the robot industry. An AGV is a transport vehicle with autonomous navigation function, which has the characteristics of high flexibility, low cost, intelligence and high efficiency, and is widely used in occasions such as warehouses, catering, medical treatment, cleaning, etc., improving people's quality of life and work efficiency.

[0003] In today's society, cameras for monitoring have spread throughout people's living scenes. Controlling the work of AGVs in the way of visual positioning and guiding has become the mainstream. However, currently, the monocular vision guiding method is generally used to control AGVs. This method has a limited viewing field, cannot meet the requirements of running in a large space, and cannot solve the problems that the vision of AGVs is blocked by obstacles during the navigation process or the positioning fails to be obtained by a single camera, and the map is inaccurate with large errors, and cannot complete tasks better. Therefore, a multi - vision guiding method is needed to solve this problem. Summary of the Invention

[0004] To overcome the problems existing in the prior art, the present invention provides a tracking method and system for an AGV based on multi - vision. By using the technical solution provided by the present invention, the problems that the vision of an AGV based on visual navigation is blocked by obstacles or the positioning fails to be obtained by a single camera, as well as the small viewing field and large map error, can be solved.

[0005] The object of the present invention can be achieved by the following technical solutions: A positioning and tracking method for an AGV based on multi - vision, comprising the following steps:

[0006] S1. Video acquisition step: capturing the required video images through cameras installed in three different directions;

[0007] S2. Video image processing step: performing distortion correction on the acquired video images by using Zhang Zhengyou calibration method;

[0008] S3. Video image feature extraction step: converting the specific rectangular calibration plate in the video image into HSV mode, performing binaryzation on it, performing a closing operation to obtain a calibration plate mask, detecting corner points on it by using the canny operator to obtain corner points, and performing perspective transformation on the image;

[0009] S4. Map stitching step. Three markers, namely square, circular, and triangular, are designed within the calibration board. The markers are used to automatically identify each corner point. The video images are stitched in the overlapping area by means of a stitching line to form a global map.

[0010] S5. Object recognition step. Images of the AGV are captured by three cameras in different directions. Then, the Labeling tool is used to label the AGV to form a dataset, which is divided into a training set and a validation set at a certain ratio and undergoes model training to recognize the AGV.

[0011] S6. Target positioning and tracking step. One large and one small color patch are set at the front and back of the AGV roof. The centroid coordinates of the two color patches are extracted to obtain the center point and direction of the AGV. The information obtained from the three cameras is fused, and then a path control algorithm for globally guiding the AGV based on the ROS platform is used to control the AGV to reach the specified location.

[0012] In the above-mentioned positioning and tracking method for an AGV based on multi-view vision, in the S3 step, the canny operator uses a Gaussian function to smooth the image, and its Gaussian function is as follows:

[0013]

[0014] where σ is the standard deviation of the Gaussian filtering function.

[0015] In the above-mentioned positioning and tracking method for an AGV based on multi-view vision, the S4 specifically includes the following steps:

[0016] S41. In the perspective-transformed image, select the upper left corner as the origin, and use the side length of the calibration board as the side length to select the ROI area of the calibration board.

[0017] S42. After binarizing the ROI area through a grayscale threshold, use a contour extraction algorithm.

[0018] S43. Obtain the vertices of the contour through a contour approximation algorithm, determine the shape according to the contour vertices. If the number of fixed points is 3, it is a triangle; if it is 4, it is a rectangle; if it is 0, it is the center of the circle.

[0019] S44. Obtain the centroid of the shape by calculating the spatial distance of the contour. Determine the corresponding points in the calibration board through the distance between the corner points and the centroid. The video images are stitched in the overlapping area by means of a stitching line to form a global map.

[0020] In the above-mentioned positioning and tracking method for an AGV based on multi-view vision, in the S4 step, the expression of the spatial distance is as follows:

[0021] M ij =∑ x，yCt(x, y) x i y j (2)

[0022] Among them, Ct(x, y) represents the coordinate of the point set of the contour.

[0023] In the above method for positioning and tracking an AGV based on multi - vision, the specific steps of S6 are as follows:

[0024] S61. In the video image of the recognized AGV, segment the ROI area of the AGV and perform binary processing on it according to the HSV interval of the color block.

[0025] S62. Extract the contours of two color blocks, calculate the spatial distance of the contours to obtain the centroids of the two color blocks, and find the center point through the centroid coordinates of the two color blocks. The center point is the center point of the AGV.

[0026] S63. Calculate the contour areas of the two color blocks through a function, corresponding to the centroid coordinates of the corresponding color blocks according to the contour area size. The direction of the AGV is the angle between the ray from the centroid of the small color block to the centroid of the large color block and the positive direction of the x - axis of the image coordinate system, and sum and average the positioning information.

[0027] S64. Calculate the transformation relationship between the AGV and the path - point coordinate system, control the AGV to rotate to the target path - point direction, first send half of the number of straight - line control commands, calculate the translation and rotation amounts to the target path - point again by listening to the transformation between the AGV coordinate system and the path - point coordinate system, and repeatedly adjust the straight - line direction in this way. When the distance between the AGV and the target path - point is less than the set threshold, it is considered that the target point has been reached.

[0028] Based on the inventive concept of the above method, the present invention also provides an AGV positioning and tracking system based on multi - vision. The system includes:

[0029] A video acquisition module, which is used to acquire video images with a target area and an overlapping area by using cameras installed in three different directions.

[0030] A video image processing module, which is used to perform distortion correction on the video image to reduce errors.

[0031] A video image feature extraction module, which is used to convert the specific rectangular calibration plate in the video image to the HSV mode, perform binary processing on it, execute a closing operation to obtain a calibration plate mask, use the canny operator to detect corner points on it, and perform perspective transformation on the image.

[0032] A map stitching module, which is used to automatically identify each corner point by using markers, and stitch the video images in the overlapping area in a suture - like manner to form a global map.

[0033] An object recognition module, which is used to collect images of the AGV by cameras in three different directions, and then use the Labeling tool to label the AGV to form a data set. The data set is divided into a training set and a validation set at a certain ratio and model training is performed on them to recognize the AGV;

[0034] A target positioning and tracking module, which is used to obtain the center point and direction of the AGV by extracting the centroid coordinates of two color blocks, fuse the information obtained by the three cameras, and then use a path control algorithm designed based on the ROS platform to globally guide the AGV to reach the specified location.

[0035] Compared with the prior art, the advantages of a positioning and tracking method and system for an AGV based on multi-camera vision of the present invention are as follows: the video image is corrected for distortion through a video image processing module to reduce errors; the specific rectangular calibration board in the video image is converted to the HSV mode through a video image feature extraction module, and it is binarized, a closing operation is performed to obtain a calibration board mask, the canny operator is used on it to detect corner points to obtain corner points, and a perspective transformation is performed on the image; the map stitching module is used to automatically identify each corner point by using identification objects, and the video images are stitched in the overlapping area in a suture line manner to form a global map. First, it can automatically identify corner points without manual selection, and second, it increases the observation field of view and improves the map accuracy; through the target positioning and tracking module, first, the center point of the AGV can be obtained more accurately, improving the positioning accuracy. Description of the Drawings

[0036] Figure 1 It is a schematic flow chart of the method described in the embodiment of the present invention

[0037] Figure 2 It is a schematic diagram of the map stitching process

[0038] Figure 3 It is a schematic diagram of the target positioning and tracking process

[0039] Figure 4 It is a schematic diagram of the module structure of the system described in the embodiment of the present invention Detailed Embodiments

[0040] The drawings are only for illustrative purposes;

[0041] For better illustration of this example, some components in the drawings are omitted, enlarged or reduced, and do not represent the actual size of the product.

[0042] The technical solutions of the present invention will be further described below with reference to the drawings and examples, but the present invention is not limited to these embodiments.

[0043] As shown in the atta Figure 1As shown in the figure, a positioning and tracking method for a multi-ocular vision AGV according to the present invention includes the following steps:

[0044] S1. Video acquisition step: Capture the required video images through cameras installed in three different directions.

[0045] S2. Video image processing step: Perform distortion correction on the acquired video images through the Zhang Zhengyou calibration method.

[0046] S3. Video image feature extraction step: Convert the specific rectangular calibration board in the video image to the HSV mode, binarize it, perform a closing operation to obtain the calibration board mask, use the canny operator to detect corner points to obtain corner points, and perform a perspective transformation on the image.

[0047] Among them, in the content of S33, the canny operator uses a Gaussian function to smooth the image, and its Gaussian function is as follows:

[0048]

[0049] Among them, σ is the standard deviation of the Gaussian filtering function.

[0050] S4. Map stitching step: Three markers, namely a square, a circle, and a triangle, are designed inside the calibration board. Use the markers to automatically identify each corner point, and stitch the video images in the overlapping area in the form of a stitching line to form a global map.

[0051] As shown in the appendix Figure 2 It includes the following specific content:

[0052] S41. In the perspective-transformed image, select the upper left corner as the origin, and use the side length of the calibration board as the side length to select the ROI area of the calibration board.

[0053] S42. After binarizing the ROI area through the gray threshold, use the contour extraction algorithm.

[0054] S43. Obtain the vertices of the contour through the contour approximation algorithm, determine the shape according to the contour vertices, where the number of fixed points is 3 for a triangle, 4 for a rectangle, and 0 for the center of the circle.

[0055] S44. Obtain the centroid of the shape by calculating the spatial distance of the contour. Determine the corresponding points in the calibration board through the distance between the corner points and the centroid, and stitch the video images in the overlapping area in the form of a stitching line to form a global map.

[0056] Among them, in the content of S44, the expression of the spatial distance is as follows:

[0057] M ij =∑ x，yCt(x, y) x i y j (2)

[0058] Among them, Ct(x, y) represents the coordinate of the point set of the contour.

[0059] S5. Object recognition step: Collect images of the AGV through cameras in three different directions, then use the Labeling tool to label the AGV to form a data set, divide it into a training set and a validation set in a certain proportion, and perform model training on them to recognize the AGV.

[0060] S6. Target positioning and tracking step: Set two color blocks, one large and one small, at the front and back of the AGV roof. Obtain the center point and direction of the AGV by extracting the centroid coordinates of the two color blocks, fuse the information obtained by the three cameras, and then use the path control algorithm designed based on the ROS platform to globally guide the AGV to reach the specified location.

[0061] As shown in the appendix Figure 3 shown, which includes the following specific content:

[0062] S61. In the video image of the recognized AGV, segment the ROI area of the AGV and perform binary processing on it according to the HSV interval of the color block.

[0063] S62. Extract the contours of the two color blocks, calculate the spatial distance of the contours to obtain the centroids of the two color blocks, and find the center point through the centroid coordinates of the two color blocks. The center point is the center point of the AGV.

[0064] S63. Calculate the contour areas of the two color blocks through a function, correspond the centroid coordinates of the corresponding color blocks according to the contour area size. The direction of the AGV is the angle between the ray from the centroid of the small color block to the centroid of the large color block and the positive direction of the x-axis of the image coordinate system, and sum and average the positioning information.

[0065] S64. Calculate the transformation relationship between the AGV and the path point coordinate system, control the AGV to rotate to the direction of the target path point, first send half of the number of straight-line control commands, listen to the transformation between the AGV coordinate system and the path point coordinate system, calculate the translation and rotation amounts to the target path point again, and repeatedly adjust the straight-line direction in this way. When the distance between the AGV and the target path point is less than the set threshold, it is considered to have reached the target point.

[0066] As shown in the appendix Figure 4 shown, a positioning and tracking system for a multi-vision AGV, the system includes a video acquisition module, a video image processing module, a video image feature extraction module, a map stitching module, an object recognition module, and a target positioning and tracking module, among which:

[0067] A video acquisition module, which is used to acquire video images with a target area and an overlapping area by using cameras installed in three different directions;

[0068] A video image processing module, which is used to correct the distortion of the video image to reduce errors;

[0069] A video image feature extraction module, which is used to convert the specific rectangular calibration plate in the video image to the HSV mode, binarize it, perform a closing operation to obtain a calibration plate mask, use the canny operator to detect corners to obtain corners, and perform a perspective transformation on the image;

[0070] A map stitching module, which is used to automatically identify each corner point by using identification objects, and stitch the video images in the overlapping area in the way of stitching lines to form a global map;

[0071] An object recognition module, which is used to collect images of the AGV by cameras in three different directions, then use the Labeling tool to label the AGV to form a data set, divide it into a training set and a validation set in a certain proportion and perform model training to recognize the AGV;

[0072] A target positioning and tracking module, which is used to obtain the center point and direction of the AGV by extracting the centroid coordinates of two color blocks, fuse the information obtained by the three cameras, and then use a path control algorithm for globally guiding the AGV designed based on the ROS platform to control the AGV to reach a specified location.

[0073] By using the technical solution provided by the present invention, the problems that the vision of the AGV based on global vision navigation is blocked by obstacles or the positioning fails when a single camera is used are solved. The AGV can be positioned and tracked more accurately, the transportation efficiency can be effectively improved, and the transportation cost can be reduced, which is of great significance to the intelligence of industries such as intelligent warehousing and unmanned handling.

[0074] The above embodiments are preferred implementation manners of the present invention and are not intended to limit the present invention. Without departing from the inventive concept of the present invention, any obvious replacement is within the protection scope of the present invention.

Claims

1. A method for AGV positioning and tracking based on multi-view vision, characterized in that, The method includes the following steps: S1. Video acquisition step: capturing the required video images through cameras installed in three different directions; S2. Video image processing step: performing distortion correction on the acquired video images by Zhang Zhengyou calibration method; S3. Video image feature extraction step: converting the specific rectangular calibration board in the video image to HSV mode, binarizing it, performing a closing operation to obtain a calibration board mask, using the canny operator to detect corner points to obtain corner points, and performing perspective transformation on the image; S4. Map stitching step: designing three markers, namely square, circular and triangular, inside the calibration board, automatically identifying each corner point by using the markers, and stitching the video images in the overlapping area in the way of stitching lines to form a global map; S5. Object recognition step: collecting images of the AGV through cameras in three different directions, then using the Labeling tool to label the AGV to form a data set, dividing it into a training set and a validation set in a certain proportion and performing model training to recognize the AGV; S6. Target positioning and tracking step: setting two color blocks, one large and one small, at the front and back of the AGV roof, obtaining the center point and direction of the AGV by extracting the centroid coordinates of the two color blocks, fusing the information obtained by the three cameras, and then using the path control algorithm designed based on the ROS platform to guide the AGV globally to control the AGV to reach the specified location. The detailed steps of S6 are as follows: S61. In the video image where the AGV is recognized, segment the ROI area of the AGV and perform binarization processing on it according to the HSV interval of the color block; S62. Extract the contours of the two color blocks, calculate the spatial distance of the contours to obtain the centroids of the two color blocks, and calculate the center point through the centroid coordinates of the two color blocks. The center point is the center point of the AGV; S63. Calculate the contour areas of the two color blocks through a function, corresponding to the centroid coordinates of the corresponding color blocks according to the contour area size. The direction of the AGV is the angle between the ray from the centroid of the small color block to the centroid of the large color block and the positive direction of the x-axis of the image coordinate system, and sum and average the positioning information; S64. Calculate the transformation relationship between the AGV and the path point coordinate system, control the AGV to rotate to the direction of the target path point, first send half of the number of straight-line control commands, monitor the transformation between the AGV coordinate system and the path point coordinate system, calculate the translation and rotation amounts to the target path point again, and adjust the straight-line direction repeatedly in this way. When the distance between the AGV and the target path point is less than the set threshold, it is considered that the target point has been reached.

2. The positioning and tracking method for multi-camera vision-based AGV according to claim 1, characterized in that In the S3 step, the canny operator smooths the image using a Gaussian function, and the Gaussian function is as follows: where o is the standard deviation of the Gaussian filter function.

3. The positioning and tracking method based on multi-vision AGV according to claim 1, characterized in that The S4 includes the following steps: S41. In the perspective transformation image, select the upper left corner as the origin, and frame the ROI area of the calibration board with the side length of the calibration board; S42. After binarizing the ROI area by the gray threshold, use the contour extraction algorithm; S43. Obtain the vertices of the contour through the contour approximation algorithm, determine the shape according to the contour vertices, if the number of fixed points is 3, it is a triangle, if it is 4, it is a rectangle, and if it is 0, it is the center of the circle; S44. Obtain the centroid of the shape by calculating the spatial distance of the contour. Determine the corresponding points in the calibration board based on the distance between the corner points and the centroid, and perform corresponding matching. Divide and splice the video images in the overlapping area in the form of suture lines to form a global map.

4. The positioning and tracking method of the multi-ocular vision-based AGV according to claim 3, characterized in that In the step S44, the expression of the spatial distance is as follows: M i,j = ∑ x,y Ct(x,y)x i y j (2) where Ct(x, y) represents the coordinate set of the points of the contour.

5. A positioning and tracking system for an AGV based on multi-view vision, characterized in that applying the AGV positioning and tracking method based on multi-view vision as described in any one of claims 1-3, which includes a video acquisition module, a video image processing module, a video image feature extraction module, a map splicing module, an object recognition module, and a target positioning and tracking module; The video acquisition module is used to obtain video images with a target area and an overlapping area by using cameras installed in three different directions; The video image processing module is used to correct the distortion of the video image to reduce errors; The video image feature extraction module is used to convert the specific rectangular calibration board in the video image to the HSV mode, perform binarization on it, execute a closing operation to obtain a calibration board mask, detect corner points using the canny operator on it, and perform a perspective transformation on the image; The map splicing module is used to automatically identify each corner point by using markers, and splice the video images in the overlapping area in the form of suture lines to form a global map; The object recognition module is used to collect images of the AGV by cameras in three different directions, then use the Labeling tool to label the AGV to form a data set, divide it into a training set and a validation set at a certain ratio and perform model training on it to identify the AGV; The target positioning and tracking module is used to obtain the center point and direction of the AGV by extracting the centroid coordinates of two color patches, fuse the information obtained by the three cameras, and then use the path control algorithm for globally guiding the AGV designed based on the ROS platform to control the AGV to reach the specified location.

Images