Obstacle avoidance navigation system based on binocular vision

Abstract

The invention discloses an obstacle avoidance navigation system based on binocular vision. The obstacle avoidance navigation system comprises a binocular camera and a GPS module. A visual processing and depth estimation algorithm module; a map generation module; and a path planning and dynamic obstacle avoidance module. The binocular camera and the GPS module are combined, so that visual-geographic information fusion is realized, and the environmental perception precision is improved; the three-dimensional point cloud generated by depth calculation provides three-dimensional space data support for obstacle avoidance, and is safer and more reliable than traditional two-dimensional recognition. The navigation map constructed by the semantic segmentation and target detection technology can dynamically distinguish complex obstacle types; the path combination strategy of the shortest path algorithm and the map API not only ensures the real-time performance of local obstacle avoidance, but also gives consideration to the global path optimality, so that the navigation efficiency of the system in complex terrains is higher.

Description

Technical Field

[0001] The present invention relates to an obstacle avoidance navigation method, and more particularly to an obstacle avoidance navigation system based on binocular vision. Background Art

[0002] With the rapid development of artificial intelligence, computer vision, and automation technologies, intelligent navigation systems have been widely applied in fields such as blind assistance devices, autonomous driving, and robot navigation. However, existing navigation technologies rely on traditional sensors such as lidar (LiDAR), ultrasonic sensors, and inertial measurement units (IMU). Although these sensors can provide accurate distance measurements and environmental perception, their high cost, large volume, and high power consumption limit their widespread use in low-cost, portable devices. In addition, these sensors are easily interfered with in harsh environments (such as rain, snow, and dust coverage), resulting in a decline in the reliability and applicability of the navigation system.

[0003] In recent years, pure vision algorithms represented by Tesla have brought a new direction for autonomous driving. By capturing environmental texture and color information through cameras, high-precision obstacle detection and path planning can be achieved. However, the adaptability of such technologies in specific scenarios is still insufficient. For example, in blind assistance and indoor navigation, vision algorithms face the problem of insufficient flexibility in temporary path planning in dynamic environments. In addition, due to hardware resource limitations, low-cost devices are difficult to bear the high computational load of pure vision algorithms, making them face technical bottlenecks in widespread applications. Generally speaking, there is room for optimization in the cost, applicability, and environmental robustness of existing navigation technologies. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology and overcoming the limitations of traditional sensors and pure vision algorithms, the purpose of the present invention is to provide an obstacle avoidance navigation system based on binocular vision, aiming to achieve real-time path planning and high-precision obstacle detection in specific scenarios through low-cost hardware and efficient algorithms, and provide a reliable navigation solution for blind navigation, embodied robots, inspection robots, etc.

[0005] To achieve the above purpose, the present invention provides the following technical solution: An obstacle avoidance navigation system based on binocular vision, comprising:

[0006] A binocular camera and a GPS module, which are used to synchronously collect images through the binocular camera to generate stereoscopic vision information, and provide real-time position information through the GPS module to determine the geographical location information of the objects in the images and the camera.

[0007] The visual processing and depth estimation algorithm module, communicatively connected to the binocular camera and the GPS module, receives the stereoscopic vision information and the real-time position information, and provides environmental stereoscopic information for path planning through image calibration, depth calculation, and 3D point cloud generation;

[0008] The map generation module, communicatively connected to the visual processing and depth estimation algorithm module, combines semantic segmentation and object detection technologies, extracts the walkable area and obstacle information in the image, and projects the 3D point cloud onto a 2D plane to generate an environmental map;

[0009] The path planning and dynamic obstacle avoidance module, communicatively connected to the visual processing and depth estimation algorithm module and the map generation module, generates a temporary path planning area by setting the depth of the path planning area, calculates the shortest path from the starting point to the candidate temporary destination based on the shortest path algorithm, then obtains the path from the candidate temporary destination to the end point by combining the map API, and finally selects the candidate temporary destination passed by the path with the shortest total distance as the temporary destination, and finally arrives at the end position by continuously updating and navigating to the temporary destination.

[0010] As a further improvement of the present invention, the specific steps for the visual processing and depth estimation algorithm module to perform visual processing are as follows:

[0011] Step 1, eliminate camera distortion through stereo calibration, and perform epipolar constraint on the image to ensure pixel alignment;

[0012] Step 2, perform stereo matching through the SGBM algorithm to generate a disparity map, and then use the depth estimation algorithm to convert the disparity value into actual depth information to generate a 3D point cloud.

[0013] As a further improvement of the present invention, the specific method for stereo calibration in Step 1 is as follows: First, correct the distortion to eliminate barrel or pincushion distortion, ensure that the fields of view of the left and right images are consistent, and then perform epipolar constraint. Through binocular geometric correction, correct the two images so that their corresponding pixels are aligned horizontally.

[0014] As a further improvement of the present invention, the specific method for performing stereo matching through the SGBM algorithm in Step 2 to generate a disparity map, and then using the depth estimation algorithm to convert the disparity value into actual depth information to generate a 3D point cloud is as follows:

[0015] Step 2-1, use the SGBM algorithm to generate a disparity map in the way of pixel block matching and deduce the depth information. Specifically, first input the left and right calibrated images, and then output the disparity value d of each pixel, which represents the difference in the positions of the same object captured by the two cameras in the image, usually in pixels. Finally, perform parameter optimization to adjust the window size and penalty coefficient to balance accuracy and computational efficiency;

[0016] Step 22: For each pixel, calculate the depth D based on the disparity d, the camera baseline distance B, and the focal length f:

[0017]

[0018] In the formula, D: the depth of the pixel, f: the focal length of the camera, usually in millimeters or meters, B: the baseline distance of the binocular camera, that is, the physical distance between the two cameras, d: the disparity;

[0019] Step 23: According to the depth map and the pixel position, convert each pixel to the actual three-dimensional space coordinates, specifically:

[0020] Based on the pinhole camera model, assuming that the depth value of a given pixel (x, y) is D (x,y) , then the three-dimensional coordinates (X, Y, Z) corresponding to this pixel can be calculated through the following camera projection model:

[0021]

[0022] In the formula, x, y: the position of the pixel in the image, representing the horizontal and vertical coordinates in the image coordinate system, C x , C y : the coordinates of the image optical center, which is the position of the image optical center in the image coordinate system. This is usually obtained during camera calibration and represents the position of the camera center. f: is the focal length of the camera, usually in pixels. The focal length determines the size of the field of view and the way an object is mapped in the image, D (x,y) : the depth value of the pixel (x, y) in the depth map, representing the actual distance from this pixel to the camera, (X, Y, Z): are the coordinates of this pixel in three-dimensional space, in meters.

[0023] As a further improvement of the present invention, the specific steps for the map generation module to extract the walkable area and obstacle information in the image by combining semantic segmentation and object detection technologies are as follows:

[0024] Step 1: Obstacle recognition and walkable area extraction based on semantic segmentation and object detection. By parallelly using semantic segmentation and object detection models and combining the post-processing stage of image processing, extract the obstacle information and walkable area in the image;

[0025] Step 2: Project the three-dimensional point cloud onto a two-dimensional plane, and then generate a grid map.

[0026] As a further improvement of the present invention, the specific steps for extracting the obstacle information and walkable area in the image in Step 1 are as follows:

[0027] Step 11: Input the image into two models, namely the semantic segmentation model and the object detection model, simultaneously to process the image in parallel. The semantic segmentation model gives the class label of each pixel, while the object detection model identifies the specific object positions in the image;

[0028] Step 12: Use an advanced semantic segmentation model to perform semantic segmentation on the input image, generating the class information of each pixel point in the image. Through semantic segmentation, perform per-pixel classification on the scene to obtain the segmentation mask of the walkable area and the obstacle area, and obtain the following two areas:

[0029] Walkable area: The pixels in the image are labeled as "0";

[0030] Obstacle area: The pixels in the image are labeled as "1";

[0031] Step 13: Use the object detection model to detect the specific positions of the obstacles, obtaining the bounding boxes and class labels of each obstacle;

[0032] Step 14: After the semantic segmentation and object detection in the above steps obtain their respective results, perform effective fusion on the two through a post-processing stage to ensure the accuracy of obstacle recognition and the integrity of the walkable area;

[0033] Step 15: After that, the model parameters of semantic segmentation and object detection need to be tuned through experiments to obtain the best accuracy and efficiency in different scenarios.

[0034] As a further improvement of the present invention, the specific steps for effective fusion in Step 14 are as follows:

[0035] Step 141: Based on the mask obtained by semantic segmentation, use the obstacle bounding box information detected by the object detection model to enhance or correct the obstacle area;

[0036] Step 142: Perform obstacle filtering. Consider the areas where the obstacles detected by object detection overlap or are close to the obstacles in semantic segmentation as obstacles, and handle them as obstacle avoidance areas in subsequent path planning;

[0037] Step 143: Based on the bounding boxes obtained by object detection, use the semantic segmentation results to further correct the obstacle bounding boxes.

[0038] As a further improvement of the present invention, the specific steps for projecting the three-dimensional point cloud onto a two-dimensional plane and then generating a grid map in Step 2 are as follows:

[0039] Step 21: Project the 3D point cloud onto a 2D plane. Specifically, only retain the X and Y information and ignore Z to generate a plane map. Then, in the 2D plane, each point is marked as a walkable area or an obstacle according to the depth value and the semantic segmentation result.

[0040] Step 22: Convert the point projection result on the 2D plane into a grid map as follows:

[0041] First, perform grid division. Divide the 2D plane into grids at a certain resolution. The actual size of the plane represented by each grid can be set. Then, perform obstacle classification. For each grid cell, check whether there is any obstacle overlapping with it. If there is, the grid is marked as an "unwalkable area" and labeled 1. If not, it is marked as a "walkable area" and labeled 0. After that, calculate the longitude and latitude of the center of the grid cell. Use the depth map and the internal and external parameters of the camera to calculate the position of each pixel point in the world coordinate system. Then, according to the world coordinates of each pixel, use the GPS position of the camera and the coordinate conversion method to calculate the longitude and latitude of this point. Finally, calculate the longitude and latitude of the center point of each grid cell based on the grid size.

[0042] As a further improvement of the present invention, the path planning and dynamic obstacle avoidance module generates a temporary path planning area by setting the depth of the path planning area, and calculates the shortest path from the starting point to the temporary destination based on the shortest path algorithm. The specific steps are as follows:

[0043] Step 3: Set the depth of M meters in front of the camera as the single-path planning depth. Then, through a preset ratio, the pixel length m corresponding to M meters in the grid map can be obtained. The area with a vertical distance from the horizontal line where the camera is located less than m pixels is constructed into a temporary path planning area.

[0044] Step 4: Consider the grid cells of the feasible area as nodes, build edges for the grids with common edges, and denote it as graph G(E, V), where E represents the edge set and V represents the node set. Then, set the walkable nodes on the edge line as candidate temporary destinations.

[0045] Step 5: Denote the side length as unit 1, and obtain the shortest path S min After that, multiply by k to obtain the actual pixel distance, and then multiply by the length M / m represented by each pixel to obtain the actual length value, where the distance from S to v i is denoted as dis(S, v i ) and is obtained through the shortest path algorithm.

[0046] Step 6: Set the feasible walking nodes on the edge line as candidate temporary destinations, obtain the set TP of candidate temporary destinations, traverse TP, add the distance from the candidate temporary destination temp (temp ∈ TP) to the starting point S to the distance from temp to the ending point T to get the shortest distance from S to T passing through temp, and select the point with the shortest total route distance as the temporary target point P for navigation;

[0047] Step 7: By calculating the direction vectors of two consecutive points in the path, use the arctangent function to find the azimuth angle and correct it to the range of 0° - 360°, and update the angle to guide the walking direction in real time to ensure the accuracy and dynamic adjustment ability of path planning;

[0048] Step 8: Compare the similarity of continuously collected pictures through a visual large model. If the similarity is lower than the preset threshold, recalculate and plan the path to achieve the final navigation from the starting point to the ending point. As a further improvement of the present invention, the determination and update process of the temporary target point in Step 6 can be achieved through the following formula:

[0049] if(S min >dis(S,temp)+DIS(temp,T)){

[0050] S min =dis(S,temp)+DIS(temp,T), P=temp

[0051] }

[0052] Record the temporary destination when S min is the minimum value as P. First, use P as the temporary target point for navigation, and update the temporary target point by continuously collecting environmental information and path calculation to finally achieve full-path navigation. The beneficial effects of the present invention:

[0053] · Reduce equipment costs: Traditional navigation systems usually rely on high-cost hardware such as lidar and ultrasonic sensors. The present invention significantly reduces the hardware investment by using low-cost devices such as binocular cameras and GPS, making the system suitable for wide applications in low-cost scenarios.

[0054] · Improve environmental adaptability: The performance of traditional sensors is limited in the detection of bad weather or specific object types. The present invention adopts pure vision technology, combined with deep learning algorithms, and realizes high adaptability to complex environments through dynamic image processing and environmental feature extraction, including rain, snow, haze conditions and diverse obstacles.

[0055] · Solve the problems of sensor size and weight: Devices such as lidar are not suitable for miniaturized devices due to their large volume and weight. This invention is designed based on a lightweight vision sensor, supports the deployment of compact terminals (such as smart glasses and portable inspection devices), and meets the design requirements of small devices.

[0056] · Optimize energy consumption management: Sensors such as lidar have high power consumption. This invention reduces power consumption through a pure vision solution and optimized algorithms, enabling the device to operate for a long time under battery power, and is suitable for scenarios such as construction site inspections and computer room inspections that require long battery life.

[0057] · Simplify the complexity of data processing: The data processing of traditional navigation systems is complex and lacks real-time performance. This invention simplifies the multi-sensor fusion steps by integrating visual perception and path planning, improves the system response speed, and meets the real-time requirements.

[0058] · Make up for the lack of visual information: Lidar lacks object texture and color information. This invention obtains rich environmental detail information through vision technology, further enhancing the path planning and obstacle recognition capabilities, especially suitable for blind navigation and inspection scenarios.

[0059] · Improve the flexibility and scalability of the system: Traditional navigation systems have limited environmental adaptability and high maintenance costs. This invention expands the scene adaptation ability through software algorithm updates and vision technology, and can be quickly deployed in multi-field scenarios such as blind navigation, construction site inspections, and computer room inspections.

[0060] · Meet the requirements of dynamic path planning: For the adjustment of temporary destinations, this invention supports a flexible path re-planning function, which is closer to user needs, especially suitable for blind navigation and inspection tasks in complex environments.

[0061] · Strengthen user safety guarantee: The system focuses on path planning and obstacle detection, optimizes the risk control ability, and provides users with highly reliable and secure navigation services. Description of the Drawings

[0062] Figure 1 It is a schematic diagram of the working process of the obstacle avoidance navigation system based on binocular vision of the present invention;

[0063] Figure 2 It is a planar map for projecting three-dimensional point clouds onto a two-dimensional plane;

[0064] Figure 3 It is a grid map;

[0065] Figure 4 It is a schematic diagram of the temporary path planning area;

[0066] Figure 5 It is a schematic diagram of an undirected map and candidate temporary destinations;

[0067] Figure 6 It is a schematic diagram of the temporary optimal navigation path. Detailed implementation manners

[0068] The present invention will be further described in detail below in conjunction with the embodiments given in the accompanying drawings.

[0069] Referring to Figure 1 As shown, a binocular vision-based obstacle avoidance and navigation system according to this embodiment includes the following modules:

[0070] (1) Binocular camera + GPS module

[0071] This module synchronously collects images through a binocular vision camera to generate stereoscopic vision information, and uses a depth estimation algorithm to calculate the depth information of each pixel. At the same time, the GPS module provides real-time position information to help determine the geographical location between the object in the image and the camera. The binocular camera needs to have high resolution, low latency, and wide-angle vision to ensure high-precision image acquisition and the accuracy of real-time navigation.

[0072] (2) Visual processing and depth estimation algorithm module

[0073] This module is responsible for image calibration, depth calculation, and three-dimensional point cloud generation. First, camera distortion is eliminated through stereo calibration, and epipolar constraints are applied to the image to ensure pixel alignment. Then, stereo matching is performed through the SGBM algorithm to generate a disparity map, and then the depth estimation algorithm is used to convert the disparity value into actual depth information to generate a three-dimensional point cloud, finally providing depth information support for the environment for path planning.

[0074] (3) Map generation module

[0075] This module combines semantic segmentation and object detection technologies to extract the walkable area and obstacle information in the image. Each pixel's category is marked through semantic segmentation, and the precise position of the obstacle is obtained using object detection. Then, the three-dimensional point cloud in the image is projected onto a two-dimensional plane to generate a grid map. On this basis, the obstacle area is separated from the walkable area to provide map support for subsequent path planning.

[0076] (4) Path planning and dynamic obstacle avoidance module

[0077] This module is mainly responsible for path planning and dynamic obstacle avoidance. First, a temporary path planning area is generated by setting the depth of the path planning area, and the shortest path from the starting point to the temporary destination is calculated based on the shortest path algorithms such as A*. Subsequently, the path from the temporary destination to the end point is obtained by combining with the map API, and finally the path with the shortest total distance is selected as the navigation target. By calculating the direction vector of two consecutive points in the path, the azimuth angle is obtained using the arctangent function and corrected to the range of 0° - 360°, and the angle is updated in real time to guide the walking direction, ensuring the accuracy and dynamic adjustment ability of path planning. At the same time, the system compares the similarity of the collected images in real time to dynamically adjust the path, ensuring the accuracy and safety of navigation.

[0078] Furthermore, in this embodiment, the following binocular camera + GPS module is provided for collecting external environment images and providing real-time position information at the same time:

[0079] The binocular vision camera system consists of two high-resolution, low-latency wide-angle cameras, which are installed on specific devices (such as robotic dogs, blind glasses, blind canes, etc.). The baseline distance between the two cameras remains constant, usually about 5 - 10 cm. The system forms a stereo vision input by synchronously collecting images from the left and right perspectives, and uses a depth estimation algorithm to deduce the depth information of each pixel. The GPS can obtain the longitude and latitude information of the current camera point and can be used for subsequent calculation of the longitude and latitude of the walkable area.

[0080] Camera hardware requirements:

[0081] High resolution: At least support 720p or higher resolution.

[0082] Low latency: Ensure the real-time nature of the images and avoid latency affecting the navigation accuracy.

[0083] Wide-angle field of view: Provide a field of view of 120° or larger, while taking into account the image quality in indoor and outdoor environments. The GPS module is the commonly used GPS module in the prior art.

[0084] Furthermore, in this embodiment, the following visual processing and depth estimation algorithm module is provided for image calibration, depth calculation, and generation of three-dimensional point clouds for stereo vision information and real-time position information. The visual processing and depth estimation algorithm module is implemented through the following steps:

[0085] Stereo calibration

[0086] To ensure the alignment of the images generated by the binocular camera system and eliminate distortion. Use a checkerboard pattern for camera calibration to obtain the internal parameter matrix (focal length, optical center coordinates, etc.) and the external parameter matrix (rotation, translation matrix).

[0087] Rectify distortion: Eliminate barrel or pincushion distortion to ensure consistent fields of view for the left and right images.

[0088] Epipolar constraint: Through binocular geometric rectification, correct the two images so that their corresponding pixels are aligned horizontally. Specifically, the pixel rows in the rectified images will be parallel, ensuring that each pixel in a row has the same depth information during stereo matching, facilitating the calculation of the depth map through disparity.

[0089] Stereo matching

[0090] Use the SGBM algorithm (Semi - Global Block Matching) to generate a disparity map in the way of pixel - block matching and infer depth information.

[0091] Input: Left and right rectified images.

[0092] Output: The disparity value d for each pixel, representing the difference in the positions of the same object captured by the two cameras in the image, usually in pixels.

[0093] Parameter optimization: Adjust the window size and penalty coefficient to balance accuracy and computational efficiency.

[0094] Depth estimation

[0095] The depth here refers to the distance from the object surface to the camera, usually the distance along the optical axis of the camera. For each pixel point, calculate the depth D based on the disparity value d, the camera baseline distance B, and the focal length f:

[0096]

[0097] D: The depth of the pixel point (unit: meter).

[0098] f: The focal length of the camera, usually in millimeters or meters.

[0099] B: The baseline distance of the binocular camera, that is, the physical distance between the two cameras.

[0100] d: The disparity value.

[0101] Through this formula, we can convert the disparity value of each pixel in the 2D image into the actual depth information in the scene, thereby generating a depth map representing the real - world distance corresponding to each pixel point.

[0102] 3D point cloud generation

[0103] Convert each pixel to its actual three-dimensional spatial coordinates based on the depth map and pixel position. Specifically, the generation of the three-dimensional point cloud requires converting the depth value (the depth information corresponding to each pixel in the depth map) and the two-dimensional position of the pixel into points in the three-dimensional space through a projection model.

[0104] Based on the pinhole camera model, assume that the depth value of a given pixel point (x, y) is D (x,y) , then the three-dimensional coordinates (X, Y, Z) corresponding to this pixel point can be calculated through the following camera projection model:

[0105]

[0106] x, y: The position of the pixel point in the image, representing the horizontal and vertical coordinates in the image coordinate system.

[0107] C x , C y : The image optical center coordinates, which are the position of the image optical center in the image coordinate system. This is usually obtained during camera calibration and represents the position of the camera center.

[0108] f: The focal length of the camera, usually in pixels. The focal length determines the size of the field of view and the way an object is mapped in the image.

[0109] D (x,y) : The depth value of the pixel point (x, y) in the depth map, representing the actual distance from this pixel point to the camera (unit: meter).

[0110] (X, Y, Z): The coordinates of this pixel in the three-dimensional space, in meters.

[0111] Through the above formula, the depth information D of each pixel (x,y) and the corresponding image coordinates (x, y) can be converted into three-dimensional coordinates (X, Y, Z). For each pixel point in the image, the corresponding three-dimensional point can be obtained. The set of these three-dimensional points constitutes the three-dimensional point cloud of the entire image.

[0112] Furthermore, in this embodiment, the following map generation module is provided to extract the walkable area and obstacle information in the image, which is specifically implemented by the following steps:

[0113] Method for obstacle recognition and walkable area extraction based on semantic segmentation and object detection

[0114] This method extracts obstacle information and traversable areas in an image by parallelly using semantic segmentation and object detection models and combining the post-processing stage of image processing. The specific steps are as follows:

[0115] ① Image input: The input image is simultaneously fed into two models - a semantic segmentation model and an object detection model - to process the image in parallel. The semantic segmentation model gives the class label of each pixel, while the object detection model identifies the specific object positions in the image.

[0116] ② Semantic segmentation: Use an advanced semantic segmentation model (such as DeepLabV3+ or FCN) to perform semantic segmentation on the input image, generating the class information of each pixel point in the image. The goal of semantic segmentation is to classify the scene pixel by pixel, thereby obtaining a segmentation mask for the traversable area and the obstacle area.

[0117] Traversable area: The pixels in the image are labeled as "0".

[0118] Obstacle area: The pixels in the image are labeled as "1", such as buildings, obstacles on the road surface, etc.

[0119] ③ Object detection: Use an object detection model (such as YOLOv5) to detect the specific positions of obstacles, obtaining the bounding boxes and class labels (such as pedestrians, vehicles, etc.) of each obstacle.

[0120] ④ Post-processing and fusion: After semantic segmentation and object detection obtain their respective results, they are effectively fused through the post-processing stage to ensure the accuracy of obstacle recognition and the integrity of the traversable area, as follows:

[0121] Merge the segmentation mask and the object detection results: Based on the mask obtained from semantic segmentation, the obstacle bounding box information detected by the object detection model is used to enhance or correct the obstacle area. Specifically, if an object detection identifies an obstacle, the pixel points within the bounding box area of the obstacle will be regarded as obstacles, and even if semantic segmentation does not fully label this area as an obstacle, it will be corrected through this bounding box.

[0122] Obstacle filtering: If the obstacles detected by object detection overlap or are close to the obstacles in semantic segmentation, these areas are considered as obstacles and are treated as obstacle avoidance areas in subsequent path planning.

[0123] Corrected Bounding Box: Based on the bounding boxes obtained from object detection, the semantic segmentation results are used to further correct the bounding boxes of obstacles. The specific method is that if there is a walkable area outside the bounding box and an unlabeled obstacle area inside, the mask of semantic segmentation can be used to expand or adjust the bounding box to ensure that the object detection box accurately covers the entire area of the obstacle and avoid missed detections.

[0124] ⑤ Parameter Tuning: The model parameters of semantic segmentation and object detection (such as learning rate, training dataset size, network architecture, etc.) need to be tuned through experiments to obtain the best accuracy and efficiency in different scenarios.

[0125] 3D to 2D Projection

[0126] ① Project the 3D point cloud onto a 2D plane (bird's-eye view)

[0127] Only retain the X, Y information and ignore Z (height) to generate a planar map.

[0128] In the 2D plane, each point is labeled as a walkable area or an obstacle according to the depth value and the semantic segmentation results, and a map is obtained as Figure 2 shown.

[0129] ② Generate a grid map

[0130] Convert the projection results of the points on the 2D plane into a grid map (2D matrix) as follows:

[0131] Grid Division: Divide the 2D plane into grids at a certain resolution (e.g., k pixels). Here, "k" represents the size of the grid cell, that is, the actual spatial area covered by each grid (e.g.: 1 grid = 0.5m x 0.5m, and this value can be set dynamically). By controlling the size of the grid (k pixels), the balance between accuracy and computational efficiency can be achieved.

[0132] Obstacle Classification: For each grid cell, check if there are any obstacles (such as buildings, vehicles, etc.) overlapping with it. If there are, the grid is marked as "non-walkable area" and labeled as 1. If not, it is marked as "walkable area" and labeled as 0. Each grid cell records the corresponding area type and obstacle-related information. Based on this information, the distance between the area and the obstacle can be obtained.

[0133] Calculation of the longitude and latitude of the center of the grid cell: Use the depth map and the internal and external parameters of the camera to calculate the position of each pixel point in the world coordinate system. Then, based on the world coordinates (X, Y, Z) of each pixel, use the GPS position of the camera and the coordinate transformation method to calculate the longitude and latitude of this point. Finally, according to the grid size (e.g., k pixels), the longitude and latitude of the center point of each grid cell can be deduced to obtain a grid map as shown in Figure 3 as shown.

[0134] Furthermore, this embodiment provides the following path planning and dynamic obstacle avoidance module, which specifically realizes path planning and dynamic obstacle avoidance through the following steps:

[0135] Set the path planning area

[0136] Set the depth of M meters in front of the camera as the single-path planning depth. M is recommended to be 3 - 6 meters, which can be obtained based on the results of statistical analysis and can be set according to needs through the background. Through the preset ratio, the pixel length m corresponding to M meters in the grid map can be obtained. The area with a vertical distance from the horizontal line where the camera is located less than m pixels is constructed into a temporary path planning area, as shown in Figure 4 as shown.

[0137] Construct an undirected graph of the walkable area and set candidate temporary destinations

[0138] Regard the grid of the feasible area as nodes and build edges for the grids with common edges. Denote it as graph G(E, V), where E represents the set of edges and V represents the set of nodes. The following graph structure can be obtained. The walkable nodes on the edge line are set as candidate temporary destinations, as shown by the dark nodes in the following figure (b), denoted as set TP. In this example, TP = {v1, v2, v3, v4, v5, v6, v7, v 14 , v 15 , v 25 , v 26 , v 27 , v 28 , v 29 , v 30 , v 31 , v 32 , v 33}, as shown in Figure 5 as shown. Calculate the shortest path from S to the candidate temporary destination

[0139] Since the scaling ratio is the same, the side length can be denoted as unit 1. After obtaining the shortest path S min , multiply it by k to obtain the actual pixel distance, and then multiply it by the length M / m represented by each pixel to obtain the actual length value. The distance from S to v i is denoted as dis(S, v i) can be obtained through shortest path algorithms (such as A* algorithm, Spfa, Dijkstra algorithm, etc.).

[0140] Calculate the shortest path from the candidate temporary destination to the actual destination T

[0141] Based on the longitude and latitude of the candidate temporary destination and the longitude and latitude of the actual destination T, obtain the shortest distance from each candidate temporary destination to the target location T through the map API. v i The distance to T is denoted as DIS(v i , T). Calculate the temporary destination and the optimal temporary navigation path

[0142] Traverse TP, add the distance from the candidate temporary destination temp (temp ∈ TP) to the starting point S to the distance from temp to the ending point T, and obtain the shortest distance from S to T passing through temp. Select the point with the shortest total route distance as the temporary target point P. The determination and update process of the temporary target point can be achieved through the following formula:

[0143]

[0144] Record the temporary destination when S min is the minimum value as P. Assume the node v 28 is the temporary destination, then S->v4->v 10 ->v 19 ->v 18 ->v 17 ->v 28 and S->v4->v 10 ->v 19 ->v 18 ->v 29 ->v 28 are both optional paths, and either one can be chosen as the final temporary navigation path, as Figure 6 shown.

[0145] Direction calculation

[0146] By calculating the direction vectors of two consecutive points in the path, use the arctangent function to find the azimuth angle and correct it to the range of 0° - 360°, and update the angle to guide the walking direction in real time to ensure the accuracy and dynamic adjustment ability of the path planning. In the blind device, the blind can adjust the direction through voice broadcast or vibration reminder. In devices such as robotic dogs, embodied robots, and inspection robots, the walking direction of the device can be directly controlled.

[0147] Dynamic obstacle avoidance path planning

[0148] The similarity of continuously collected images is compared through a visual large model. If the similarity is lower than a certain threshold, such as 95%, the calculation is restarted and the path is planned to achieve the final navigation from the starting point to the ending point.

[0149] In addition, the large model can be combined to detect obstacles during movement and predict their trajectories, so as to achieve more accurate and perfect obstacle avoidance navigation.

[0150] Thus, a binocular vision-based obstacle avoidance navigation system is jointly composed of a binocular camera, a GPS module, a visual processing and depth estimation algorithm module, a map generation module, and a path planning and dynamic obstacle avoidance module. This system:

[0151] (1) Precise positioning and depth perception technology integrating binocular vision and GPS

[0152] Stereo images are synchronously collected by the binocular camera. Combining with the depth estimation algorithm, the depth information of each pixel is accurately measured. At the same time, the real-time geographical location data provided by the GPS is fused to achieve high-precision spatial positioning of the object position. This technology greatly improves the accuracy of environmental perception and meets the requirements of high-precision navigation in complex scenarios.

[0153] (2) Three-dimensional point cloud generation and depth calculation optimization technology based on stereo vision

[0154] The present invention proposes an efficient stereo matching technology combining the SGBM (Semi-Global Block Matching) algorithm, which can quickly generate a disparity map and accurately estimate the depth information of the three-dimensional space. Through stereo calibration and epipolar constraint optimization, the reliability and accuracy of the depth calculation results are ensured, providing technical support for three-dimensional point cloud generation.

[0155] (3) Intelligent grid map generation technology based on parallel fusion calculation of semantic segmentation and object detection

[0156] The parallel fusion technology based on semantic segmentation and object detection improves the efficiency and accuracy of grid map generation, comprehensively extracts the walkable area and obstacle information, and generates a grid map through the projection of the three-dimensional point cloud to the two-dimensional plane, adapting to dynamic scene changes, and providing accurate and real-time data support for path planning and dynamic obstacle avoidance.

[0157] (4) Temporary area dynamic path planning and real-time obstacle avoidance technology

[0158] Based on the greedy thought, the system innovatively combines the shortest path algorithm with the map API. Each time, it only conducts precise navigation for a specific area ahead, plans the path in stages, from the starting point to a temporary destination on the edge of the planned area grid and then to the final destination, so as to achieve the global optimization of the path. At the same time, through the real-time dynamic similarity analysis of images, the path is adjusted to cope with environmental changes, ensuring the efficiency and safety of navigation.

[0159] (5) Low-cost intelligent navigation hardware design

[0160] Provide a hardware solution with binocular vision as the core to reduce the overall cost of the navigation system.

[0161] The obstacle avoidance navigation system based on binocular vision in this embodiment can be applied to blind glasses, blind canes and other blind navigation devices, as well as various intelligent devices such as robotic dogs, embodied robots, construction site inspections, and computer room inspections.

[0162] The above are only the preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as within the protection scope of the present invention.

Claims

1. A binocular vision-based obstacle avoidance and navigation system, characterized in that: Including: A binocular camera and a GPS module, which are used to synchronously collect images through the binocular camera to generate stereoscopic vision information, and provide real-time position information through the GPS module to determine the geographical location information of the objects and the camera in the images; A vision processing and depth estimation algorithm module, which is communicatively connected to the binocular camera and the GPS module, receives the stereoscopic vision information and the real-time position information, and provides environmental stereoscopic information for path planning through image calibration, depth calculation, and three-dimensional point cloud generation; A map generation module, which is communicatively connected to the vision processing and depth estimation algorithm module, combines semantic segmentation and object detection technologies, extracts the walkable areas and obstacle information in the images, and projects the three-dimensional point cloud onto a two-dimensional plane to generate an environmental map; A path planning and dynamic obstacle avoidance module, which is communicatively connected to the vision processing and depth estimation algorithm module and the map generation module, generates a temporary path planning area by setting the depth of the path planning area, calculates the shortest path from the starting point to the candidate temporary destination based on the shortest path algorithm, then combines with the map API to obtain the path from the candidate temporary destination to the end point, and finally selects the candidate temporary destination passed by the path with the shortest total distance as the temporary destination, and finally reaches the end position by continuously updating and navigating to the temporary destination; 2. The binocular vision-based obstacle avoidance and navigation system according to claim 1, characterized in that: The specific steps for the vision processing and depth estimation algorithm module to perform vision processing are as follows: Step 1, eliminate camera distortion through stereo calibration and perform epipolar constraint on the images to ensure pixel alignment; Step 2, perform stereo matching through the SGBM algorithm to generate a disparity map, and then use the depth estimation algorithm to convert the disparity value into actual depth information to generate a three-dimensional point cloud; 3. The binocular vision-based obstacle avoidance and navigation system according to claim 2, wherein: The specific method of stereo calibration in Step 1 is as follows: First, correct the distortion to eliminate barrel or pincushion distortion and ensure that the fields of view of the left and right images are consistent, and then perform epipolar constraint. Through binocular geometric correction, the two images are corrected so that their corresponding pixels are aligned horizontally; 4. The binocular vision-based obstacle avoidance and navigation system according to claim 3, characterized in that: The specific method of performing stereo matching through the SGBM algorithm in Step 2 to generate a disparity map, and then using the depth estimation algorithm to convert the disparity value into actual depth information to generate a three-dimensional point cloud is as follows: Step 2-1, use the SGBM algorithm to generate a disparity map in the way of pixel block matching and calculate the depth information. Specifically, first input the left and right calibrated images, and then output the disparity value d of each pixel, which represents the difference in the positions of the same object captured by the two cameras in the images, usually in pixels. Finally, perform parameter optimization to adjust the window size and penalty coefficient to balance accuracy and calculation efficiency; Step 2-2, for each pixel point, calculate the depth D based on the disparity value d, the camera baseline distance B, and the focal length f: In the formula, D: the depth of the pixel point, f: the focal length of the camera, usually in millimeters or meters, B: the baseline distance of the binocular camera, that is, the physical distance between the two cameras, d: the disparity value; Step 2-3, according to the depth map and pixel positions, convert each pixel point into the actual three-dimensional space coordinates. Specifically: Based on the pinhole camera model, assuming that the depth value of a given pixel point (x, y) is D (x,y) , then the three-dimensional coordinates (X, Y, Z) corresponding to this pixel point can be calculated through the following camera projection model: Z = D (x,y) Where x and y are the positions of the pixel in the image, representing the horizontal and vertical coordinates in the image coordinate system, C x , C y : are the coordinates of the optical center of the image, which is the position of the optical center of the image in the image coordinate system, usually obtained during camera calibration, representing the position of the camera center, f: is the focal length of the camera, usually in pixels. The focal length determines the size of the field of view and the way an object is mapped in the image, D (x,y) : is the depth value of the pixel (x, y) in the depth map, representing the actual distance from the pixel to the camera, (X, Y, Z): are the coordinates of the pixel in three-dimensional space, in meters.

5. The binocular vision-based obstacle avoidance and navigation system according to any one of claims 1 to 3, characterized in that: The specific steps for the map generation module to extract the walkable area and obstacle information in the image by combining semantic segmentation and object detection technologies are as follows: Step 1, Obstacle recognition and walkable area extraction based on semantic segmentation and object detection. By using semantic segmentation and object detection models in parallel and combining the post-processing stage of image processing, the obstacle information and walkable area in the image are extracted; Step 2, Project the 3D point cloud onto a 2D plane and then generate a grid map.

6. The binocular vision-based obstacle avoidance and navigation system according to claim 5, characterized in that: The specific steps for extracting the obstacle information and walkable area in the image in Step 1 are as follows: Step 11, Input the image into two models simultaneously - a semantic segmentation model and an object detection model, and process the image in parallel. The semantic segmentation model gives the class label of each pixel, while the object detection model identifies the specific object positions in the image; Step 12, Use an advanced semantic segmentation model to perform semantic segmentation on the input image, generate the class information of each pixel point in the image, and perform pixel-by-pixel classification on the scene through semantic segmentation to obtain the segmentation mask of the walkable area and the obstacle area, and obtain the following two areas: Walkable area: The pixels in the image are marked as "0"; Obstacle area: The pixels in the image are marked as "1"; Step 13, Use the object detection model to detect the specific positions of the obstacles, and obtain the bounding box and class label of each obstacle; Step 14, After the semantic segmentation and object detection in the above steps obtain their respective results, effectively fuse the two through the post-processing stage to ensure the accuracy of obstacle recognition and the integrity of the walkable area; Step 15, Then the model parameters of semantic segmentation and object detection need to be tuned through experiments to obtain the best accuracy and efficiency in different scenarios.

7. The binocular vision-based obstacle avoidance navigation system according to claim 6, wherein: The specific steps for effective fusion in Step 14 are as follows: Step 141, Based on the mask obtained by semantic segmentation, enhance or correct the obstacle area according to the obstacle bounding box information detected by the object detection model; Step 142, Perform obstacle filtering. Overlap or proximity of the obstacles detected by object detection and the obstacles in semantic segmentation are considered as obstacles, and these areas are treated as obstacle avoidance areas in subsequent path planning; Step 143, Based on the bounding box obtained by object detection, further correct the obstacle bounding box using the semantic segmentation result.

8. The binocular vision-based obstacle avoidance and navigation system according to claim 5, wherein: The specific steps for projecting the 3D point cloud onto a 2D plane and then generating a grid map in Step 2 are as follows: Step 21, Project the 3D point cloud onto a 2D plane. Specifically, only retain the X and Y information, ignore Z to generate a plane map, and then in the 2D plane, each point is marked as a walkable area or an obstacle according to the depth value and the semantic segmentation result; Step 22, Convert the projection result of the points on the 2D plane into a grid map, specifically as follows: First, perform grid division. Divide the two-dimensional plane into grids at a certain resolution. The actual size of the plane represented by each grid can be set. Then, conduct obstacle classification. For each grid cell, check whether there is any obstacle overlapping with it. If there is, the grid is marked as an "unwalkable area" and labeled 1. If not, it is marked as a "walkable area" and labeled 0. After that, calculate the longitude and latitude of the center of each grid cell. Use the depth map and the internal and external parameters of the camera to calculate the position of each pixel point in the world coordinate system. Then, based on the world coordinates of each pixel, use the GPS position of the camera and the coordinate conversion method to calculate the longitude and latitude of this point. Finally, calculate the longitude and latitude of the center point of each grid cell based on the grid size.

9. The binocular vision-based obstacle avoidance and navigation system according to any one of claims 1 to 3, characterized in that: The path planning and dynamic obstacle avoidance module generates a temporary path planning area by setting the depth of the path planning area, and the specific steps for calculating the shortest path from the starting point to the temporary destination based on the shortest path algorithm are as follows: Step 3: Set the depth of M meters in front of the camera as the single-path planning depth. Then, through a preset ratio, the pixel length m corresponding to M meters in the grid map can be obtained. The area with a vertical distance from the horizontal line where the camera is located less than m pixels is constructed into a temporary path planning area. Step 4: Regard the grid cells in the feasible area as nodes, build edges for the grids with common edges, and denote it as graph G(E, V). E represents the edge set, and V represents the node set. Then, set the walkable nodes on the edge line as candidate temporary destinations. Step 5, record the side length as unit 1, and obtain the shortest path S min After that, multiply it by k to obtain the actual pixel distance, and then multiply it by the length M / m represented by each pixel to obtain the actual length value, where the distance from S to v i is denoted as dis(S, v i ), which is obtained by the shortest path algorithm; Step 6: Set the walkable nodes on the edge line as candidate temporary destinations to obtain the candidate temporary destination set TP. Traverse TP, add the distance from the candidate temporary destination temp (temp ∈ TP) to the starting point S and the distance from temp to the ending point T, to get the shortest distance from S to T passing through temp. Select the point with the shortest total route distance as the temporary target point P for navigation. Step 7: By calculating the direction vector of two consecutive points in the path, use the arctangent function to find the azimuth angle and correct it to the range of 0° - 360° to update the angle to guide the walking direction in real time, ensuring the accuracy and dynamic adjustment ability of path planning. Step 8: Compare the similarity of continuously collected pictures through a visual large model. If the similarity is lower than the preset threshold, recalculate and plan the path to achieve the final navigation from the starting point to the ending point.

10. The binocular vision-based obstacle avoidance navigation system according to claim 9, characterized in that: The determination and update process of the temporary target point in Step 6 can be achieved through the following formula: if(S min >dis(S,temp)+DIS(temp,T)){ S min = dis(S, temp) + DIS(temp, T), P = temp } Record S min When the temporary destination is P at the minimum value, first use P as the temporary target point for navigation, and update the temporary target point by continuously collecting environmental information and path calculation, and finally achieve full-path navigation.

Images