Indoor guide vehicle for the blind and its guiding method

Abstract

The present invention provides an indoor guide vehicle for the blind and a guide method thereof, wherein the indoor guide vehicle for the blind is driven by a driving motor to realize folding of the upper arm, the lower arm and the handle, thereby solving the problem of space occupation in the standby state; the indoor guide method of the present invention realizes high-precision self-positioning and path tracking through the combination of a depth camera, a monocular camera and IMU data, thereby solving the navigation problem in a complex indoor environment and improving the ability to reproduce a predetermined path. Environmental information is obtained based on the fusion of monocular and binocular vision, thereby improving the speed and accuracy of obstacle detection, calculating obstacle trends based on inter-frame changes, and realizing predictive path planning.

Description

Technical Field

[0001] The invention belongs to the technical field related to intelligent robots, and in particular relates to an indoor blind-guiding vehicle and a blind-guiding method thereof. Background Art

[0002] Blind people have special inconveniences in their daily life. In order to solve the obstacles in their daily life, people have trained guide dogs to assist them in walking. Although guide dogs bring great convenience to the blind, indoor scenes are complex and diverse. When blind people move around at home, they often do not use guide dogs or other guide devices because the disadvantage is that the traction method is not enough to make the traction object completely avoid spatial obstacles and cannot accurately reach the exact location the user wants to reach. Therefore, there are no robots on the market that can help the blind and other special groups to assist in navigation and walking in indoor scenes. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an indoor blind guide vehicle and a blind guide method thereof to solve the technical problems mentioned in the background technology.

[0004] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is: an indoor guide vehicle for the blind comprises an upper arm, a lower arm and a manned platform, a wheel mechanism for driving the indoor guide vehicle for the blind is arranged at the bottom of the manned platform, the lower arm and the manned platform as well as the lower arm and the upper arm are connected by a rotating shaft, and the lower arm and the upper arm are driven to rotate by a driving motor, and a handle is arranged at the free end of the upper arm;

[0005] A depth camera, an infrared light transmitter and a monocular camera are arranged on the outer side of the top of the upper arm, and a controller and an integrated module for receiving instructions are also integrated in the indoor guide car. The integrated module, the depth camera, the infrared light transmitter and the monocular camera are all connected to the input end of the controller, and the output end of the controller is respectively connected to the driving motor and the wheel mechanism.

[0006] Furthermore, a storage groove matching the lower arm is provided on the upper end surface of the manned platform.

[0007] Furthermore, the bottom end of the lower arm is a square structure. When the lower arm is rotated upward to a certain angle, the square structure abuts against the storage groove to achieve positioning of the lower wall.

[0008] Furthermore, a mechanical limiting structure that cooperates with each other is arranged between the lower arm and the upper arm. When the lower arm is parallel to the upper arm, the upper arm is limited by the mechanical limiting structure.

[0009] Furthermore, a variable-direction ratchet paddle is provided at the connection between the lower arm and the upper arm, and the variable-direction ratchet paddle is used to limit the rotation direction of the upper arm.

[0010] Furthermore, the integrated module includes a voice module, a Bluetooth module and an infrared receiver.

[0011] An indoor blind guiding method of the present invention comprises the following steps:

[0012] S1. Preparatory work, including:

[0013] S11, Visual SLAM Mapping: Map collection is completed by manual traction, and the three-dimensional map is converted into a two-dimensional grid map;

[0014] S12, deep learning training: objects with length, width and height greater than the threshold are identified as fixed objects, and the fixed objects are jointly trained with their names to associate the object information with the grid map position;

[0015] S13, monocular and depth camera calibration: use the calibration board to obtain the intrinsic and extrinsic parameters of the monocular and binocular cameras, and transform the depth map data from the depth camera coordinate system to the world coordinate system according to the intrinsic and extrinsic parameters, and then transform from the world coordinate system to the monocular camera coordinate system to complete the registration of the depth map and the color image;

[0016] S2, call start: the pager emits electromagnetic waves, and the car starts after receiving them;

[0017] S3, Positioning: After the car starts, it rotates once to collect image information, obtains its own position based on fixed objects, and determines the position of the pager in the map based on the call signal;

[0018] S4, Road Planning: Use the current point as the starting point and the target point as the end point to generate a static path using the D* algorithm;

[0019] S5, walking: The translation position is obtained by complementing the visual feature points between frames, and the rotation position is obtained by pre-integration of the IMU. The two are tightly coupled to obtain the robot's moving direction and distance;

[0020] S6, obstacle avoidance: During walking, the object is determined to be static or dynamic based on its own changes and changes in abnormal feature points. The outline size and distance of the object in the area in front of the camera are obtained through depth stratification, and then fed back to the map part, and obstacle avoidance is performed in real time according to the obstacle avoidance strategy.

[0021] Furthermore, the process of obtaining the self-positioning according to the fixed object in step S3 includes the following steps:

[0022] S31, after obtaining the grid map, assigning label attributes to large objects, and projecting the edges of the objects according to the maximum edges of the objects;

[0023] S32. Obtain the category information of large objects based on YOLOv5 training; large objects are required to be fixed and not move. Input the pictures of large objects into the YOLOv5s network and add respective labels to each object.

[0024] S33, rotating to obtain the position of the object; automatically rotating when powered on, using deep learning to identify object information, and obtaining synchronized depth information based on the pixels of the color image;

[0025] S34, obtaining the position of the object in the map by using the triangle theorem according to the distance information of two or more objects and their position information in the map;

[0026] S35, the standby state after completing the work, stays at the designated location and records the coordinates for next use.

[0027] Furthermore, step S5 specifically includes the following steps:

[0028] S51, color image feature point recognition: using the ORB algorithm to obtain the corner points of the image as image feature points; selecting the image frames whose feature point numbers exceed a specific threshold as key frames;

[0029] S52, obtaining depth information of feature points: obtaining distance information corresponding to feature points according to the monocular and depth registration matrix;

[0030] S53, feature point matching: obtaining image information of different key frames, and completing image matching based on the similarity of feature points;

[0031] S54, triangulation: The camera and the object at different times form a triangular relationship, and the body posture change is calculated according to the angle change and depth change of the feature points;

[0032] S55, obtaining IMU measurement data between two key frames, and performing pre-integration to obtain the body posture;

[0033] S56, sending the pose estimated by the camera and the pose pre-integrated by the IMU into the factor graph for graph optimization to obtain an accurate real-time pose.

[0034] Furthermore, step S6 obtains the size and distance of the object outline in the area in front of the camera through depth layering, and feeds back to the map part, and the process specifically includes:

[0035] a. Obtain image feature points through ORB algorithm;

[0036] b. Set the pixel spacing threshold k, traverse all feature points, and classify the points with a distance less than k into the same class. When the number of feature points in a class is greater than the threshold, it is classified as an obstacle.

[0037] c. Set a depth threshold h0, obtain the depth values of the feature points recorded as obstacles, traverse all points, calculate the depth difference h between two points, and consider the depths of two points to be continuous when h < h0; store the points that meet the depth continuity in sets.

[0038] d. Traverse each obstacle set, take its edge values to construct a three-dimensional contour, and combine the edge contour with the depth distance of the body to calculate its position in the map coordinate system, and add it to the temporary obstacle set of the map.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] 1. Foldable grip design: The upper arm, lower arm, and grip are folded by driving a motor, solving the problem of space occupation in the standby state.

[0041] 2. Autonomous positioning and multi-sensor fusion pose estimation: By combining the depth camera, monocular camera, and IMU data, high-precision self-positioning and path tracking are achieved, solving the navigation problem in complex indoor environments and improving the ability to reproduce the预定 path.

[0042] 3. Target detection and dynamic obstacle avoidance: According to the fusion of monocular and binocular vision to obtain environmental information, the obstacle detection speed and accuracy are improved, and the obstacle movement direction is calculated by the inter-frame change to achieve predictive path planning. Brief Description of the Drawings

[0043] Figure 1 is a schematic structural diagram of the indoor guide dog vehicle of the present invention;

[0044] Figure 2 is a schematic diagram of the limit structure;

[0045] Figure 3 is a schematic flow diagram of the indoor guide method of the present invention;

[0046] Figure 4 is a schematic flow diagram of the walking pose calculation based on the coupling of monocular, depth camera, and IMU provided in the embodiment;

[0047] Figure 5 is a schematic flow diagram specifically including the process of obtaining the object contour size and distance in the front area of the camera through depth stratification and feeding it back to the map part provided in the embodiment;

[0048] Figure 6 is a schematic diagram of the dynamic and static detection and obstacle avoidance logic provided in the embodiment;

[0049] Markings in the figure: 1. Grip, 2. Depth camera, 3. Infrared light emitter, 4. Monocular camera, 5. Upper arm, 6. Lower arm, 7. Passenger platform, 8. Wheel mechanism, 9. Mechanical limit structure, 10. Variable-direction ratchet paddle. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] The indoor guide vehicle of the present invention is as follows: Figure 1 As shown, it includes an upper arm 5, a lower arm 6 and a manned platform 7. A wheel mechanism 8 for driving the indoor guide vehicle for the blind is provided at the bottom of the manned platform 7. The lower arm 6 and the manned platform 7 as well as the lower arm 6 and the upper arm 5 are connected by a rotating shaft, and the lower arm and the upper arm are driven to rotate by a driving motor. A handle 1 is provided at the free end of the upper arm 5.

[0052] The top outer side of the upper arm 5 is provided with a depth camera 2, an infrared light transmitter 3 and a monocular camera 4, and the indoor guide car also integrates a controller and an integrated module for receiving instructions, the integrated module includes a voice module, a Bluetooth module and an infrared receiver, and of course, a battery module can also be integrated. The integrated module, the depth camera 2, the infrared light transmitter 3 and the monocular camera 4 are all connected to the input end of the controller, and the output end of the controller is respectively connected to the driving motor and the wheel mechanism 8. The present invention realizes the folding of the upper arm, the lower arm and the handle by driving the motor, thereby solving the problem of space occupation in the standby state.

[0053] In order to achieve the best use effect, a storage slot matching the lower arm is provided on the upper end surface of the manned platform 7. The bottom end of the lower arm 6 is a square structure, and when the lower arm 6 rotates upward to a certain angle, the square structure abuts against the storage slot to achieve the limit of the lower wall.

[0054] Further, such as Figure 2As shown, a mechanical limiting structure 9 cooperating with each other is provided between the lower arm 6 and the upper arm 5. When the lower arm is parallel to the upper arm, the upper arm is limited by the mechanical limiting structure 9. Furthermore, a variable direction ratchet paddle 10 is also provided at the connection between the lower arm 6 and the upper arm 5, and the rotation direction of the upper arm is limited by the variable direction ratchet paddle. It should be noted here that the setting of the variable direction ratchet paddle is a relatively conventional technical solution in the field. This embodiment only provides one of the setting modes to illustrate the technical solution of the present invention: In this embodiment, the gear inside the upper arm and the iron block, spring and cam below constitute a ratchet mechanism. Its operation mode is: when the cam pushes the right iron block away, the iron block compresses the spring, leaving space for the gear to move, and the left iron block relies on the spring to resist the gear. When the gear moves clockwise, after the gear teeth contact the iron block, the direction of its force is the same as the direction of the spring contraction. The spring contracts and the gear rotates smoothly. When the gear rotates counterclockwise, after the gear teeth contact the iron block, the direction of its force is perpendicular to the spring, and it is stuck by the iron block and cannot move counterclockwise. When the cam pushes the left iron block away, the gear can move in the opposite state to the above. The cam is driven by a small motor to achieve the upper arm's limit and normal movement. The outer paddle is for manual folding in case of an accident.

[0055] The present invention provides an indoor blind guiding method, such as Figure 3 As shown, the following steps are included:

[0056] 1. Preparatory work:

[0057] (1) Visual SLAM mapping, which involves manual map collection and conversion of 3D maps into 2D grid maps;

[0058] (2) Deep learning training: objects whose length, width and height are greater than a threshold are identified as fixed objects, and the fixed objects are trained with their names to associate object information with the grid map location.

[0059] (3) Monocular and depth camera calibration: Use the calibration board to obtain the intrinsic and extrinsic parameters of the monocular and binocular cameras. According to the intrinsic and extrinsic parameters, the depth map data is converted from the depth camera coordinate system to the world coordinate system, and then from the world coordinate system to the monocular camera coordinate system to complete the registration of the depth map and the color image.

[0060] 2. Call start: The pager emits electromagnetic waves, and the car starts after receiving them.

[0061] 3. Positioning: After the car starts, it rotates once to collect image information, obtains its own position based on fixed objects, and determines the position of the pager in the map based on the call signal;

[0062] The process of self-positioning based on monocular and depth camera joint self-positioning includes the following steps:

[0063] (1) After obtaining the grid map, label attributes are assigned to large objects, and the object edges are projected according to the maximum edge of the object;

[0064] (2) Based on YOLOv5 training, we obtain the category information of large objects. Large objects are required to be fixed and not move. The pictures of large objects are input into the YOLOv5s network, and each object is given its own label. This network is the smallest and has the fastest response speed. It has good performance when identifying large targets.

[0065] (3) Rotate to obtain the object position: The device automatically rotates when powered on, uses deep learning to identify object information, and obtains synchronized depth information based on the pixels of the color image;

[0066] (4) Based on the distance information of two or more objects and their location information on the map, use the triangle theorem to obtain its own location on the map;

[0067] (5) Save: After completing the work, the machine will enter the standby state, stay at the specified location and record the coordinates for next use.

[0068] 4. Route planning: Use the D* algorithm to generate a static path with the current point as the starting point and the target point as the end point.

[0069] 5. Walking: The translation position is obtained through the complementarity of visual feature points between frames, and the rotation position is obtained through the pre-integration of IMU. The two are tightly coupled to obtain the robot's moving direction and distance;

[0070] The walking posture calculation process based on the monocular, depth camera, and IMU coupling in this embodiment is as follows: Figure 4 As shown in the figure, walking posture can be divided into translational motion and rotational motion. The walking posture calculation process specifically includes the following steps:

[0071] (1) Color image feature point recognition: Use the ORB algorithm to obtain the corner points of the image as image feature points; select the image frames whose number of feature points exceeds a certain threshold as key frames;

[0072] (2) Obtaining depth information of feature points: Obtain the distance information corresponding to the feature points based on the monocular and depth registration matrix.

[0073] (3) Feature point matching: Obtain image information of different key frames and complete image matching based on the similarity of feature points.

[0074] (4) Triangulation: The camera and the object at different times form a triangular relationship, and the body posture change is calculated based on the angle change and depth change of the feature points.

[0075] (5) Obtain the IMU measurement data between two key frames and perform pre-integration to obtain the body posture.

[0076] (6) Feed the pose estimated by the camera and the pose of IMU pre-integration into the factor graph for graph optimization to obtain an accurate real-time pose.

[0077] 6. Obstacle avoidance: During walking, determine whether an object is static or dynamic based on its own changes and the changes of abnormal feature points. Obtain the size and distance of the object contour in the area in front of the camera through depth stratification, feedback it to the map part, and perform real-time obstacle avoidance according to the obstacle avoidance strategy.

[0078] The schematic diagram of the process of obtaining the size and distance of the object contour in the area in front of the camera through depth stratification and feeding it back to the map part provided in this embodiment is as Figure 5 shown, and specifically includes the following steps:

[0079] a. Obtain image feature points through the ORB algorithm;

[0080] b. Set the pixel point spacing threshold k, traverse all feature points, and classify the points with a distance less than k between two points into the same category. When the number of feature points in the category is greater than the threshold, classify them as obstacles;

[0081] c. Set the depth threshold h0, obtain the depth values of the feature points marked as obstacles, traverse all points, calculate the depth difference h between two points, and consider the depths of two points to be continuous when h < h0; Store the points that meet the depth continuity in a set;

[0082] d. Traverse each obstacle set, take its edge values to construct a three-dimensional contour, combine the edge contour with the depth distance of the body to calculate its position in the map coordinate system, and add it to the temporary obstacle set of the map.

[0083] The dynamic and static detection and obstacle avoidance logic provided in this embodiment is as Figure 6 shown.

[0084] The indoor blind guiding method of the present invention combines depth cameras, monocular cameras and IMU data to achieve high-precision self-positioning and path tracking, solve the navigation problem in complex indoor environments, and improve the ability to reproduce a predetermined path. And obtain environmental information according to the fusion of monocular and binocular vision, improve the obstacle detection speed and accuracy, calculate the obstacle movement trend through inter-frame changes, and realize predictive path planning.

[0085] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An indoor blind guide method based on an indoor blind guide vehicle, wherein the indoor blind guide vehicle comprises an upper arm, a lower arm and a manned platform, and a wheel mechanism for driving the indoor blind guide vehicle to travel is arranged at the bottom of the manned platform, characterized in that: The lower arm and the manned platform as well as the lower arm and the upper arm are connected via a rotating shaft, and both the lower arm and the upper arm are driven to rotate by a driving motor, and a handle is provided at the free end of the upper arm; A depth camera, an infrared light transmitter and a monocular camera are arranged on the outer side of the top of the upper arm, and a controller and an integrated module for receiving instructions are also integrated in the indoor guide car, the integrated module, the depth camera, the infrared light transmitter and the monocular camera are all connected to the input end of the controller, and the output end of the controller is respectively connected to the driving motor and the wheel mechanism; The indoor blind guiding method comprises the following steps: S1. Preparatory work, including: S11, Visual SLAM Mapping: Map collection is completed by manual traction, and the three-dimensional map is converted into a two-dimensional grid map; S12, deep learning training: objects with length, width and height greater than the threshold are identified as fixed objects, and the fixed objects are jointly trained with their names to associate the object information with the grid map position; S13, monocular and depth camera calibration: use the calibration board to obtain the intrinsic and extrinsic parameters of the monocular and binocular cameras, and transform the depth map data from the depth camera coordinate system to the world coordinate system according to the intrinsic and extrinsic parameters, and then transform from the world coordinate system to the monocular camera coordinate system to complete the registration of the depth map and the color image; S2, call start: the pager emits electromagnetic waves, and the car starts after receiving them; S3, Positioning: After the car starts, it rotates once to collect image information, obtains its own position based on fixed objects, and determines the position of the pager in the map based on the call signal; S4, path planning: using the current point as the starting point and the target point as the end point, use the D* algorithm to generate a static path; S5, walking: The translation position is obtained by complementing the visual feature points between frames, and the rotation position is obtained by pre-integration of the IMU. The two are tightly coupled to obtain the robot's moving direction and distance; S6, obstacle avoidance: During walking, the object is determined to be static or dynamic based on its own changes and changes in abnormal feature points. The outline size and distance of the object in the area in front of the camera are obtained through depth stratification, and fed back to the map part. Real-time obstacle avoidance is performed according to the obstacle avoidance strategy. The process of obtaining self-positioning based on fixed objects in step S3 includes the following steps: S31, after obtaining the grid map, assigning label attributes to large objects, and projecting the edges of the objects according to the maximum edges of the objects; S32. Obtain the category information of large objects based on YOLOv5 training; large objects are required to be fixed and not move. Input the pictures of large objects into the YOLOv5s network and add respective labels to each object. S33, rotating to obtain the position of the object; automatically rotating when powered on, using deep learning to identify object information, and obtaining synchronized depth information based on the pixels of the color image; S34, obtaining the position of the object in the map by using the triangle theorem according to the distance information of two or more objects and their position information in the map; S35, the standby state after completing the work, stays at the designated location and records the coordinates for next use.

2. The indoor blind guiding method according to claim 1, characterized in that: A storage groove matching the lower arm is provided on the upper end surface of the manned platform.

3. The indoor blind guiding method according to claim 2, characterized in that: The bottom end of the lower arm is a square structure. When the lower arm is rotated upward to a certain angle, the square structure abuts against the storage groove to achieve the limiting of the lower wall.

4. The indoor blind guiding method according to claim 1, characterized in that: A mechanical limit structure that cooperates with each other is provided between the lower arm and the upper arm. When the lower arm and the upper arm are parallel, the upper arm is limited by the mechanical limit structure.

5. The indoor blind guiding method according to claim 4, characterized in that: A variable-direction ratchet paddle is also provided at the connection between the lower arm and the upper arm to limit the rotation direction of the upper arm through the variable-direction ratchet paddle.

6. The indoor blind guiding method according to any one of claims 1 to 5, characterized in that: The integrated module includes a voice module, a Bluetooth module, and an infrared receiver.

7. The indoor blind guiding method according to claim 1, characterized in that: Step S5 specifically includes the following steps: S51. Color image feature point recognition: Use the ORB algorithm to obtain the corner points of the image as image feature points; select the image frames with the number of feature points in the image exceeding a specific threshold as key frames; S52. Obtain the depth information of the feature points: According to the registration matrix of the monocular and depth, obtain the distance information corresponding to the feature points; S53. Feature point matching: Obtain the picture information of different key frames and complete image matching according to the similarity of the feature points; S54. Triangulation: The camera and the object at different times form a triangular relationship, and calculate the body pose transformation amount according to the angle change and depth change of the feature points; S55. Obtain the IMU measurement data between two key frames and perform pre-integration to obtain the body pose; S56. Send the pose calculated by the camera and the pose pre-integrated by the IMU into the factor graph for graph optimization to obtain an accurate real-time pose.

8. The indoor blind guiding method according to claim 1, characterized in that: The process of obtaining the object contour size and distance in the area in front of the camera through depth stratification in step S6 and feeding it back to the map part specifically includes: a. Obtain image feature points through the ORB algorithm; b. Set the pixel point spacing threshold k, traverse all feature points, and classify the points with a distance less than k between two points into the same class. When the number of feature points in the class is greater than the threshold, classify them as obstacles; c. Set the depth threshold h0, obtain the depth value of the feature points recorded as obstacles, traverse all points, calculate the depth difference h between two points, and consider the two points to have continuous depth when h < h0; Store the points that meet the continuous depth in a set; d. Traverse each obstacle set, take its edge value to construct a three-dimensional contour, and combine the edge contour with the depth distance of the body to calculate its position in the map coordinate system and add it to the temporary obstacle set of the map.

Images