A method for robot following pedestrians based on binocular stereo vision

Abstract

The present invention provides a method for following pedestrians based on binocular stereo vision robots, which specifically includes the following steps: establishing a pedestrian data set; training a pedestrian detector, and detecting pedestrians with binocular vision; a detection method based on a YOLO network unifies feature extraction, target classification, and target positioning in a neural network; adopting a traditional single hypothesis tracking method of recursive Kalman filtering and frame-by-frame data association to track the target pedestrian; obtaining the detected followed pedestrian, obtaining the pedestrian outline, extracting pedestrian feature points from two images at the same time, matching binocular feature points, and calculating the depth information of the feature points on the image using similar triangles; calculating the speed and direction of the pedestrian through the depth information and the robot encoder information, and predicting the movement direction trajectory of the pedestrian at the next moment. The method can calculate the speed and direction of the pedestrian, predict the movement direction trajectory of the pedestrian at the next moment, enable the robot to maintain continuous tracking capability, and quickly find the target pedestrian.

Description

Technical Field

[0001] The invention belongs to the technical field of robots, and in particular relates to a method for a robot to follow pedestrians based on binocular stereo vision. Background Art

[0002] With the rapid development of the robotics industry, various service robots are emerging one after another, and robots are being used more and more widely in our lives and work. In service industry buildings such as hotels and banks, welcoming robots can assist in guest identity registration, help guests store their backpacks and other items, and follow guests until the end of the service. In actual scenarios, especially in scenarios with large traffic and people walking between each other, robots are prone to misidentifying target guests or missing target guests when they turn. Therefore, it is necessary to improve the existing robot following method to ensure the robot's following effect in various complex environments. Summary of the invention

[0003] The purpose of the present invention is to provide a method for a robot to follow pedestrians based on binocular stereo vision, so as to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for following pedestrians by a binocular stereo vision robot, which specifically comprises the following steps:

[0005] Step 1: Establish a pedestrian dataset;

[0006] Step 2: Train pedestrian detectors and detect pedestrians with binocular vision: The detection method based on the YOLO network integrates candidate box extraction, feature extraction, target classification, and target positioning in a neural network; the YOLO network structure consists of 24 convolutional layers and 2 fully connected layers. The network entry is 448x448 (v2 is 416x416). The image is resized before entering the network. The output result of the network is a tensor with the dimension:

[0007] S*S*(B*5+C)

[0008] Among them, S is the number of divided grids, B is the number of targets each grid is responsible for, and C is the number of categories;

[0009] Step 3: Track the target pedestrian using the traditional single hypothesis tracking method of recursive Kalman filtering and frame-by-frame data association;

[0010] Step 4: Get the detected pedestrian being followed, get the pedestrian outline, extract the pedestrian feature points from the two images at the same time, match the binocular feature points, and use similar triangles to calculate the depth information of the feature points on the image;

[0011] Step 5: Calculate the speed and direction of the pedestrian through the depth information and the robot encoder information, and predict the pedestrian's movement direction trajectory at the next moment.

[0012] Preferably, the meaning of the expression in step 2 is:

[0013] (1) Each small grid corresponds to B bounding boxes. The width and height of the bounding box are the entire image, indicating the position of the bounding box of the object centered on the small grid;

[0014] (2) Each bounding box corresponds to a score, indicating whether there is an object and the positioning accuracy:

[0015] (3) Each small grid corresponds to C probability values. Find the category P (Class|object) corresponding to the maximum probability, and consider that the small grid contains the object or part of the object.

[0016] Preferably, the traditional single hypothesis tracking method of recursive Kalman filtering and frame-by-frame data association in step 3 specifically includes:

[0017] (1) Define 8-dimensional state space Where (u, v) is the center coordinate of the detected pedestrian outline, γ is the aspect ratio, h is the height, and the remaining four variables represent the speed information in the image coordinate system; then the extended Kalman filter is used for state estimation;

[0018] (2) Use the squared Mahalanobis distance to measure the distance between the Kalman state of the predicted track and the newly arrived detection. The metric prediction formula is:

[0019]

[0020] (3) Use cosine distance to measure the distance between the appearance feature (128 dimensions) and the detection feature of each track to accurately predict the ID. The metric prediction formula is:

[0021]

[0022] (4) Two binary functions are introduced to restrict the assignment matrix. The squared Mahalanobis distance and cosine distance are compared with the threshold to make a judgment. The two functions are combined to restrict the matrix. The restriction formula is:

[0023]

[0024]

[0025]

[0026] (5) Combined distance is used as the cost matrix to measure the distance between each track and detection. In this paper, only cosine distance is used for measurement (that is, lambda is set to 0), and Mahalanobis distance is used to exclude impossible situations, that is, the possible object positions inferred by the Kalman filter are ignored. The cosine distance measurement formula is:

[0027] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0028] (6) After an object is blocked for a period of time, the uncertainty of the Kalman filter prediction increases greatly and the observability in the state space becomes very low, and the Mahalanobis distance tends to favor tracks with greater uncertainty. Cascade matching is introduced to prioritize matching detections with the most recently appeared tracks.

[0029] (7) At the end of the algorithm, the IOU distance in SORT is used to solve the problem of local occlusion by calculating the IOU distance of unmatched tracks (only the previous frame is unmatched) and unmatched detection.

[0030] The technical effects and advantages of the present invention are as follows: the present method can calculate the speed and direction of pedestrians, predict the movement direction trajectory of pedestrians at the next moment, enable the robot to maintain continuous tracking capability, and quickly find the target pedestrians. DETAILED DESCRIPTION

[0031] Example

[0032] A method for following pedestrians by a binocular stereo vision robot comprises the following steps:

[0033] Step 1: Establish a pedestrian dataset;

[0034] Step 2: Train pedestrian detectors and detect pedestrians with binocular vision. The detection method based on the YOLO network integrates candidate box extraction, feature extraction, target classification, and target positioning in a neural network. The YOLO network structure consists of 24 convolutional layers and 2 fully connected layers. The network entry is 448x448 (v2 is 416x416). The image is resized before entering the network. The output of the network is a tensor with the dimension:

[0035] S*S*(B*5+C)

[0036] Among them, S is the number of divided grids, B is the number of targets in each grid, and C is the number of categories; the meaning of the above expression is:

[0037] (1) Each small grid corresponds to B bounding boxes. The width and height of the bounding box are the entire image, indicating the position of the bounding box of the object centered on the small grid;

[0038] (2) Each bounding box corresponds to a score, indicating whether there is an object and the positioning accuracy:

[0039] (3) Each small grid corresponds to C probability values. Find the category P (Class|object) corresponding to the maximum probability, and consider that the small grid contains the object or part of the object.

[0040] Step 3: Track the target pedestrian using the traditional single hypothesis tracking method of recursive Kalman filtering and frame-by-frame data association, including:

[0041] (1) Define 8-dimensional state space Where (u, v) is the center coordinate of the detected pedestrian profile, γ is the aspect ratio, h is the height, and the remaining four variables represent the speed information in the image coordinate system; then the extended Kalman filter is used for state estimation;

[0042] (2) Use the squared Mahalanobis distance to measure the distance between the Kalman state of the predicted track and the newly arrived detection. The metric prediction formula is:

[0043]

[0044] (3) Use cosine distance to measure the distance between the appearance feature (128 dimensions) and the detection feature of each track to accurately predict the ID. The metric prediction formula is:

[0045]

[0046] (4) Two binary functions are introduced to restrict the assignment matrix. The squared Mahalanobis distance and cosine distance are compared with the threshold to make a judgment. The two functions are combined to restrict the matrix. The restriction formula is:

[0047]

[0048]

[0049]

[0050] (5) Combined distance is used as the cost matrix to measure the distance between each track and detection. In this paper, only cosine distance is used for measurement (that is, lambda is set to 0), and Mahalanobis distance is used to exclude impossible situations, that is, the possible object positions inferred by the Kalman filter are ignored. The cosine distance measurement formula is:

[0051] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0052] (6) After an object is blocked for a period of time, the uncertainty of the Kalman filter prediction increases greatly and the observability in the state space becomes very low, and the Mahalanobis distance tends to favor tracks with greater uncertainty. Cascade matching is introduced to prioritize matching detections with the most recently appeared tracks.

[0053] (7) At the end of the algorithm, the IOU distance in SORT is used to solve the problem of local occlusion by calculating the IOU distance of unmatched tracks (only the previous frame is unmatched) and unmatched detection;

[0054] Step 4: Get the detected pedestrian being followed, get the pedestrian outline, extract the pedestrian feature points from the two images at the same time, match the binocular feature points, and use similar triangles to calculate the depth information of the feature points on the image;

[0055] Step 5: Calculate the speed and direction of the pedestrian through the depth information and the robot encoder information, and predict the pedestrian's movement direction trajectory at the next moment.

[0056] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for following pedestrians by a robot based on binocular stereo vision, characterized in that: The specific steps include: Step 1: Establish a pedestrian dataset; Step 2: Train pedestrian detectors and detect pedestrians with binocular vision: The detection method based on the YOLO network integrates candidate box extraction, feature extraction, target classification, and target positioning in one neural network; the YOLO network structure consists of 24 convolutional layers and 2 fully connected layers. The network entry is 448x448. The image is resized before entering the network. The output result of the network is a tensor with the dimension: S*S*(B*5+C) Among them, S is the number of divided grids, B is the number of targets each grid is responsible for, and C is the number of categories; Step 3: Track the target pedestrian using the traditional single hypothesis tracking method of recursive Kalman filtering and frame-by-frame data association; Step 4: Get the detected pedestrian being followed, get the pedestrian outline, extract the pedestrian feature points from the two images at the same time, match the binocular feature points, and use similar triangles to calculate the depth information of the feature points on the image; Step 5: Calculate the speed and direction of the pedestrian through the depth information and the robot encoder information, and predict the pedestrian's movement direction trajectory at the next moment; The traditional single hypothesis tracking method of recursive Kalman filtering and frame-by-frame data association in step 3 specifically includes: (1) Define 8-dimensional state space Where (u, v) is the center coordinate of the detected pedestrian outline, γ is the aspect ratio, h is the height, and the remaining four variables represent the speed information in the image coordinate system; then the extended Kalman filter is used for state estimation; (2) Use the squared Mahalanobis distance to measure the distance between the Kalman state of the predicted track and the newly arrived detection. The metric prediction formula is: (3) Use the cosine distance to measure the distance between the appearance feature and the detection feature of each track to accurately predict the ID. The measurement prediction formula is: (4) Two binary functions are introduced to restrict the assignment matrix. The squared Mahalanobis distance and cosine distance are compared with the threshold to make a judgment. The two functions are combined to restrict the matrix. The restriction formula is: (5) Combined distance is used as the cost matrix to measure the distance between each track and detection. In this paper, only cosine distance is used for measurement, and Mahalanobis distance is used to exclude impossible situations, that is, the possible object positions inferred by the Kalman filter are ignored. The cosine distance measurement formula is: c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j) (6) After an object is blocked for a period of time, the uncertainty of the Kalman filter prediction increases greatly and the observability in the state space becomes very low, and the Mahalanobis distance tends to favor tracks with greater uncertainty. Cascade matching is introduced to prioritize matching detections with the most recently appeared tracks. (7) At the end of the algorithm, the IOU distance in SORT is used to solve the problem of local occlusion by calculating the IOU distance of unmatched tracks and unmatched detection.

2. The method for following pedestrians by a binocular stereo vision robot according to claim 1, characterized in that: The meaning of the expression in step 2 is: (1) Each small grid corresponds to B bounding boxes. The width and height of the bounding box are the entire image, indicating the position of the bounding box of the object centered on the small grid; (2) Each bounding box corresponds to a score, indicating whether there is an object and the positioning accuracy: (3) Each small grid corresponds to C probability values. Find the category P (Class|object) corresponding to the maximum probability, and consider that the small grid contains the object or part of the object.

Images