Visual impairment aided navigation method and device based on multiple sensors and computer equipment

Abstract

The invention relates to the technical field of intelligent navigation, and discloses a visual impairment aided navigation method based on multiple sensors, which comprises the following steps: acquiring environmental data by using a laser radar and a visual sensor, and constructing a two-dimensional grid map by using an SLAM (Simultaneous Localization and Mapping) algorithm; adopting a preset ant colony optimization algorithm to plan an initial path plan of the movement of the visually impaired person; detecting obstacles in the moving process of the visually impaired person in real time so as to give out voice prompt; constructing an obstacle environment map, and planning an obstacle avoidance path for the visually impaired person to obstacle crossing until the visually impaired person returns to the initial path; and carrying out environment obstacle monitoring on the residual path in real time, and updating the residual initial path. According to the invention, by fusing the laser radar, the visual sensor and the SLAM technology, high-precision positioning, dynamic obstacle avoidance and path planning in a complex environment are realized, and the core problems of incomplete environment modeling, lagging obstacle detection, poor path planning reliability and the like of traditional visual impairment auxiliary equipment in a dynamic complex scene are solved. And safer and more intelligent navigation assistance is provided for visually impaired people.

Description

Technical Field

[0001] The present invention relates to the field of intelligent navigation technology, and in particular to a multi-sensor based visually impaired assisted navigation method, device and computer equipment. Background Art

[0002] In the field of assisted navigation for the visually impaired, helping the visually impaired achieve safe and autonomous movement in complex environments has put forward an urgent demand for high-precision environmental perception, real-time obstacle detection, and reliable path planning. Traditional visually impaired assistance devices mostly rely on single sensors (such as ultrasound and simple cameras), which have problems such as incomplete environmental information acquisition, insufficient positioning accuracy, and weak dynamic obstacle recognition capabilities. It is difficult to build a three-dimensional spatial model relying solely on ultrasonic sensors, and it is easy to miss high-altitude obstacles or low obstacles in complex terrain. Single visual sensors are limited by lighting conditions and viewing angles, and have low recognition accuracy for special obstacles such as glass curtain walls and transparent objects. Independently operating SLAM (Simultaneous Localization and Mapping) technology is prone to accumulation of positioning deviations due to mismatching of environmental features in scenes with dynamic crowds and complex light and shadow changes, and cannot provide real-time and reliable obstacle avoidance guidance and path planning for visually impaired users.

[0003] Therefore, there is an urgent need for a visually impaired assisted navigation method that integrates the advantages of multimodal sensors to provide safer and more intelligent navigation assistance for the visually impaired. Summary of the Invention

[0004] The present invention provides a multi-sensor based visually impaired assisted navigation method and device. By integrating laser radar (LiDAR), visual sensors (cameras) and SLAM (simultaneous localization and mapping) technology, it achieves high-precision positioning, dynamic obstacle avoidance and path planning in complex environments. It solves the core problems of traditional visually impaired assistance equipment in dynamic and complex scenes, such as incomplete environmental modeling, delayed obstacle detection, and poor path planning reliability, and provides safer and more intelligent navigation assistance for the visually impaired.

[0005] The present invention provides a multi-sensor based visually impaired assisted navigation method, comprising:

[0006] Use lidar and visual sensors to obtain the geometric distance data and optical image data of the environment, and use the SLAM algorithm to build a two-dimensional grid map;

[0007] Performing initial path planning using a preset ant colony optimization algorithm in the two-dimensional grid map so that the visually impaired person moves along the initial path;

[0008] Using visual sensors to detect obstacles in the movement of the visually impaired person in real time, and issuing voice reminders to the visually impaired person based on the obstacles;

[0009] Using a Kinect sensor and a laser radar to construct an obstacle environment map, and performing obstacle avoidance path planning based on the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path;

[0010] The laser radar and visual sensor are used to monitor the environmental obstacles of the remaining path in real time, and the remaining initial path is updated according to the environmental obstacles.

[0011] Furthermore, the step of using a preset ant colony optimization algorithm to perform initial path planning in the two-dimensional grid map so that the visually impaired person moves along the initial path includes:

[0012] In the two-dimensional grid map, determine the end point of the visually impaired person's movement, and initialize the ant system in the ant colony algorithm, setting the number of ants to n and the maximum number of iterations to N. max ;

[0013] Initialize k=1; where k is the ant number, and k=1, 2, ..., n;

[0014] Place ant k on the current node and update the traversed nodes in the tabu table;

[0015] When ant k is in a deadlock state, it is determined according to the taboo table that ant k is in a deadlock state, a preset deadlock handling mechanism is used to handle the deadlock state, and the process returns to the step of placing ant k on the current node and updating the traversed nodes in the taboo table;

[0016] Calculate the state transition probability of ant k and use the roulette wheel method to select the next feasible node. If the next feasible node has not reached the end point, return to the step of placing ant k on the current node and updating the traversed nodes in the tabu table.

[0017] When the next feasible node reaches the end point, it is determined that k is less than n, k=k+1 is calculated, and the process returns to the step of placing ant k on the current node and updating the traversed nodes in the tabu table;

[0018] When k is equal to n, the preset pheromone update method is used to update the pheromone, and it is determined that the current number of iterations N is less than the maximum number of iterations N. max When , calculate N = N + 1 and return to the step of initializing k = 1;

[0019] Determine whether the current number of iterations N is equal to the maximum number of iterations N max When the optimal path is output as the initial path, the visually impaired person moves along the initial path.

[0020] Furthermore, in the step of determining, according to the taboo table, when ant k falls into a deadlock state, using a preset deadlock handling mechanism to handle the situation, using the preset deadlock handling mechanism includes:

[0021] When a node has only one feasible adjacent node, it is marked as an obstacle node and withdrawn to prevent the ants from falling into a U-shaped trap. In addition, when the ants have no feasible nodes during the pathfinding process, they are killed directly.

[0022] In the step of calculating the state transition probability of ant k, the calculation formula is:

[0023]

[0024] in, is the state transition probability of ant k crawling from grid point i to grid point j; α and β are the influence factors of pheromone and heuristic information on the state transition probability, respectively, allow k Represents the set of optional paths for the node in the next stage; τ ij is the pheromone concentration value of path i to j; η ij is the heuristic information of path i to j, and the calculation formula is:

[0025]

[0026] C(n)=phi1×bend+phi2×angle

[0027]

[0028] Where Q is a constant greater than 1; C(n) is the bending suppression factor; bend is the number of bends in the path from the previous node to the next node; angle is the angle of the corner; phi1 and phi2 are the conversion coefficients of the number of turns and the degree of the turn, respectively; g(n) is the cost of node n from the starting point; S (i) represents the moving distance from the parent node of node i to i-1; h(n) is the estimated cost of node n from the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) are the target node coordinates.

[0029] Furthermore, when k is equal to n, in the step of updating the pheromone using a preset pheromone updating method, the preset pheromone updating method includes:

[0030] Set the initial pheromone concentration to:

[0031]

[0032] Among them, τ ij (0) is the initial pheromone concentration, c is a constant, d(j,l) is the distance between the adjacent grid j and the line l connecting the target point and the starting point. The closer the grid point is to the line l, the higher the initial pheromone concentration is, and the more the ant colony tends to choose it at the beginning;

[0033] Set the pheromone update rule as follows:

[0034]

[0035] Among them, Q1 and Q2 are constants greater than 1; L best represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the number of turns on the optimal path; x and y are the weighted coefficients of the turning angle and the number of turns.

[0036] Furthermore, the step of using a visual sensor to detect obstacles in the movement of the visually impaired person in real time and issuing a voice reminder to the visually impaired person according to the obstacles includes:

[0037] Using a visual sensor to detect obstacles in real time during the movement of the visually impaired person, and when an obstacle is detected, calculating the straight-line distance to the obstacle; wherein the straight-line distance is the shortest distance to the outline of the obstacle;

[0038] When the straight-line distance is greater than the threshold, no voice reminder is issued, the visually impaired person continues to move along the initial path, and returns to continue using the visual sensor to detect obstacles in the visually impaired person's movement in real time;

[0039] When the straight-line distance is less than or equal to a threshold, a voice reminder to stop is issued to the visually impaired person, so that the visually impaired person stops moving and waits for a new reminder;

[0040] The movement attribute of the obstacle is determined according to the time when the obstacle is detected and the position of the obstacle, so as to determine the final position of the obstacle according to the movement attribute of the obstacle.

[0041] Furthermore, the step of determining the movement attribute of the obstacle based on the time when the obstacle is detected and the position of the obstacle, and determining the final position of the obstacle based on the movement attribute of the obstacle, includes:

[0042] When an obstacle is detected, the visual sensor captures an initial environment image including the obstacle, and when the straight-line distance is less than or equal to a threshold, the visual sensor captures a target environment image including the obstacle;

[0043] Comparing the center point positions of the obstacle in the initial environment image and the target environment image, and determining whether the difference between the center point positions exceeds a set value;

[0044] If the difference in the center point position does not exceed the set value, the obstacle's mobility attribute is determined to be a fixed obstacle, and its position is used as the final position of the obstacle;

[0045] If the difference in the center point positions exceeds a set value, the movement attribute of the obstacle is determined to be a moving obstacle, and the time required for the visually impaired person to move to the obstacle is calculated as a time period based on the straight-line distance;

[0046] Calculating the average speed of the obstacle during the time period in which the obstacle appears, so as to predict the final position of the obstacle according to the average speed; specifically comprising:

[0047] The obstacle appearance time is the time period from the initial environment image to the target environment image. The average speed per unit time is calculated within the appearance time with the time period as the unit time, and then the average speed per unit time is calculated to obtain the average speed within the appearance time.

[0048] The crossing time of the obstacle is estimated according to the moving speed of the visually impaired person, and the moving distance of the obstacle is obtained by multiplying the average speed by the sum of the time period and the crossing time, and finally the final position of the obstacle is obtained.

[0049] Furthermore, the step of using a Kinect sensor and a lidar to construct an obstacle environment map, and performing obstacle avoidance path planning based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, wherein the step of using the lidar to construct a local map includes:

[0050] Using a Kinect sensor to collect depth information of the environment currently containing the obstacle and to create a global map based on the environment; wherein the outline of the obstacle is the union of the outline of the obstacle in the target environment image and the outline of the obstacle at the final position;

[0051] Taking the center emission point of the LiDAR as the coordinate origin, the LiDAR is used to simultaneously scan the environment and the data points of the global map. The Cartesian formula is used to convert the coordinates of the data points into global coordinates in the Cartesian system, so as to connect the data points into a segment structure that can be segmented.

[0052] The coordinates of the data points transformed by the Cartesian formula are grouped together by linear functions to form vector lines that describe the linear regions of the environment. Multiple vector lines divide the environment into multiple linear regions.

[0053] Taking the linear area as the unit, the dynamic threshold method is used to construct a local map of the environment. The expression of the vertical axis coordinate calculation point is:

[0054]

[0055] Among them, ε i represents the dynamic threshold constant; y0 represents the local map construction parameter; y i+1 Represents the local map construction error; the global map established by the Kinect sensor and the local map established by the lidar are overlapped using the covariance function to refine the virtual map established based on the environment. The horizontal coordinate of the local map is:

[0056]

[0057] in, represents the covariance constant; S represents the overlap coefficient between the global map and the local map; α r Represents the weight of the virtual map.

[0058] Furthermore, the step of constructing an obstacle environment map using a Kinect sensor and a lidar, and performing obstacle avoidance path planning based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, and performing obstacle avoidance path planning based on the obstacle environment map includes:

[0059] Calculating a map complexity coefficient C according to the obstacle environment map, and calculating a step size S according to the map complexity coefficient;

[0060]

[0061] Among them, A obstacle Indicates the area of ​​the obstacle; A map Indicates the map area; D obstacle Indicates the distribution of obstacles;

[0062] The point on the obstacle closest to the initial path is taken as the target point, and the point at which the target point is extended in the opposite direction of the obstacle on the initial path by a set distance is taken as the end point N 终 ;

[0063] Based on the RRT algorithm, set the random tree from the starting point N 起 Start growing, let the starting point N 起 As the root node of the random tree, a sampling point set N is randomly generated in the map 采 = {N1, N2, N3}, select the sampling point closest to the end point as the final sampling point N 采 , traverse the nodes in the random tree and find the distance N from the sampling point 采The nearest node N 近 ; Set node N 近 Along sampling point N 采 When the direction growth step is S and there is no collision with obstacles, a new node N is obtained. 新 , node N 新 Add random tree and define node N 新 The parent node N 近 ;

[0064] According to the node N 新 , the search space is divided into two areas, where the end point is defined as the valid area, and the other area is defined as the invalid area. 新 As the starting point, in the next random sampling, sampling is performed in the valid area to obtain a new node N again. 新2 ;

[0065] When the random tree still cannot continue to grow after the set number of iterations at the current new node, the new node is deemed invalid and deleted from the random tree. The effective sampling area of ​​the random tree is set to the effective area obtained by segmenting the previous new node. The random tree restarts to grow in the updated effective sampling area until the distance between the generated new node and the end point is less than the step size S, and the line connecting the two points does not collide with any obstacles. The end point is then determined to have been reached.

[0066] The end point is added to the random tree, and all nodes with parent nodes in the random tree are traced back to form a node set, and the preset selection connection is performed according to the node order to obtain the obstacle avoidance path.

[0067] Furthermore, in the step of using a laser radar and a visual sensor to monitor environmental obstacles in the remaining path in real time and updating the remaining initial path according to the environmental obstacles, updating the remaining initial path according to the environmental obstacles includes:

[0068] monitoring the position of the environmental obstacle in real time, and obtaining the outline of the environmental obstacle when the environmental obstacle covers the remaining path;

[0069] The environmental obstacle contour divides the remaining path into an initial section, a covered section, and an end section, and obtains a starting intersection point and an end intersection point of the environmental obstacle contour and the remaining path;

[0070] When the starting intersection point and the ending intersection point are the points on the outline of the environmental obstacle closest to the starting road section and the ending road section respectively, the starting intersection point and the ending intersection point are selected as the starting target point and the ending target point;

[0071] When the starting intersection point and the ending intersection point are not the points on the environmental obstacle contour closest to the starting section and the ending section, the points on the environmental obstacle contour closest to the starting section and the ending section are obtained as reference points, a straight line perpendicular to the starting section and the ending section is drawn through the reference point, and the intersections of the straight line with the starting section and the ending section are used as the starting target point and the ending target point, and the starting intersection point and / or the ending intersection point closest to the target point are discarded;

[0072] Use the distance between the starting target point and the ending target point as the diameter and the center of the line segment between the starting target point and the ending target point as the center of the circle;

[0073] When the circle does not intersect the environmental obstacle, draw the chord closest to the environmental obstacle, and use the path of the shortest semicircle + chord on both sides of the remaining path as the updated path of the covered road segment;

[0074] When the circle intersects the environmental obstacle, a semicircle is drawn with the distance between the point on the environmental obstacle farthest from the covered section and the center of the circle as the radius, and the outline of the semicircle is used as the updated path of the covered section.

[0075] The present invention also provides a multi-sensor based visually impaired navigation aid device, comprising:

[0076] A construction module is used to acquire geometric distance data and optical image data of the environment using lidar and vision sensors, and to construct a two-dimensional grid map using the SLAM algorithm;

[0077] A first planning module is configured to perform initial path planning in the two-dimensional grid map using a preset ant colony optimization algorithm so that the visually impaired person moves along the initial path;

[0078] a prompt module, configured to detect obstacles in the movement of the visually impaired person in real time using a visual sensor, and to issue a voice reminder to the visually impaired person based on the obstacles;

[0079] a second planning module, configured to construct an obstacle environment map using a Kinect sensor and a lidar, and perform obstacle avoidance path planning based on the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path;

[0080] The monitoring module is used to monitor the environmental obstacles of the remaining path in real time using a laser radar and a visual sensor, and to update the remaining initial path according to the environmental obstacles.

[0081] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0082] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0083] The beneficial effects of the present invention are:

[0084] The present invention uses laser radar and visual sensors to acquire environmental data, and then uses the SLAM algorithm to construct a two-dimensional grid map. The initial path for the visually impaired person is planned based on the two-dimensional grid map, and obstacles are detected in real time during the movement process to issue voice reminders to the visually impaired person. When obstacles are encountered, an obstacle environment map is constructed to plan an obstacle avoidance path, allowing the visually impaired person to overcome the obstacles and return to the initial path. Finally, the laser radar and visual sensors are used to detect environmental obstacles along the remaining path in real time to update the initial path. This method provides better navigation assistance for the visually impaired, achieving high-precision positioning, dynamic obstacle avoidance, and path planning in complex environments. It addresses the core issues of traditional visually impaired assistance devices in dynamic and complex scenes, such as incomplete environmental modeling, delayed obstacle detection, and poor path planning reliability, providing safer and more intelligent navigation assistance for the visually impaired. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 Schematic diagram of a method flow according to an embodiment of the present invention.

[0086] Figure 2 FIG. 1 is a schematic diagram of the device structure according to an embodiment of the present invention.

[0087] Figure 3 Schematic diagram of the internal structure of a computer device according to an embodiment of the present invention.

[0088] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0089] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0090] like Figure 1 As shown, the present invention provides a multi-sensor based visually impaired assisted navigation method, comprising:

[0091] S1, using laser radar and visual sensors to obtain the geometric distance data and optical image data of the environment, and using the SLAM algorithm to build a two-dimensional grid map;

[0092] S2. performing initial path planning in the two-dimensional grid map using a preset ant colony optimization algorithm so that the visually impaired person moves along the initial path;

[0093] S3, using a visual sensor to detect obstacles in the movement of the visually impaired person in real time, and issuing a voice reminder to the visually impaired person based on the obstacles;

[0094] S4. Using a Kinect sensor and a laser radar to construct an obstacle environment map, and performing obstacle avoidance path planning based on the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path;

[0095] S5. Use laser radar and visual sensors to monitor the environmental obstacles of the remaining path in real time, and update the remaining initial path according to the environmental obstacles.

[0096] As described in steps S1-S5 above, the LiDAR can collect geometric distance data of the environment, and the visual sensor (camera) can collect optical image data of the environment. The geometric distance data and optical image data of the environment are used to construct a two-dimensional grid map using the SLAM algorithm. The core concept of SLAM (Simultaneous Localization and Mapping) is: when moving in an unknown environment, the environment is observed in real time by sensors (LiDAR, camera), the user's position and posture are simultaneously estimated, and a map of the environment is constructed. In the present invention, SLAM is used to construct a two-dimensional grid map to facilitate initial path planning using a preset ant colony optimization algorithm. The visually impaired person can move along the initial path and use the visual sensor to detect obstacles in real time during the movement. Then, an obstacle environment map is constructed to determine the location of obstacles and plan an obstacle avoidance path so that the visually impaired person can cross the obstacles and return to the initial path. At the same time, the LiDAR and visual sensor also monitor obstacles remaining in the initial path so that the initial path can be updated according to the obstacles that appear.

[0097] In one embodiment, the step of using a preset ant colony optimization algorithm to perform initial path planning in the two-dimensional grid map so that the visually impaired person moves along the initial path includes:

[0098] S201: In the two-dimensional grid map, determine the end point of the visually impaired person's movement, and initialize the ant system in the ant colony algorithm, setting the number of ants to n and the maximum number of iterations to N. max ;

[0099] S202, initialize k=1; where k is the ant number, and k=1, 2, ..., n;

[0100] S203: Place ant k on the current node and update the traversed nodes in the tabu table;

[0101] S204: When ant k is in a deadlock state, it is determined according to the taboo table that ant k is in a deadlock state, and a preset deadlock handling mechanism is used for processing, and the process returns to step S203;

[0102] S205: Calculate the state transition probability of ant k and use the roulette wheel method to select the next feasible node. If the next feasible node has not reached the end point, return to step S203;

[0103] S206. When the next feasible node reaches the end point, if k is less than n, calculate k=k+1 and return to step S203.

[0104] S207: When k is equal to n, the pheromone update method is used to update the pheromone, and it is determined that the current number of iterations N is less than the maximum number of iterations N. max When N=N+1, the process returns to step S202;

[0105] S208: Determine whether the current number of iterations N is equal to the maximum number of iterations N. max When the optimal path is output as the initial path, the visually impaired person moves along the initial path.

[0106] As described in steps S201-S208 above, the improved ant colony star optimization algorithm combines the ant colony algorithm with A * The algorithm is integrated to find the optimal path as the initial path. The ant colony algorithm is a bionic algorithm inspired by ant foraging. It establishes the relationship between the amount of pheromone released by ants and the length of the path. The cumulative concentration of pheromones on a shorter path is proportional to the time, which attracts more ants to choose the path, thereby realizing the discovery of the optimal foraging path. * In order to avoid blind search, the algorithm uses its heuristic cost to update the heuristic information in the ant colony algorithm using the estimated function equation. * The algorithm has excellent optimization ability and search efficiency. It incorporates its inspired intelligent search ideas into the heuristic function of the ant colony algorithm to search for the optimal path. It uses the estimated movement cost to guide ants to reach the target point faster. At the same time, adding a bend suppression operator to the heuristic can reduce the number of bends and the cumulative turning angle.

[0107] In one embodiment, in step S204, adopting a preset deadlock handling mechanism includes:

[0108] When a node has only one feasible adjacent node, it is marked as an obstacle node and withdrawn to prevent the ants from falling into a U-shaped trap. In addition, when the ants have no feasible nodes during the pathfinding process, they are killed directly.

[0109] When the movement environment becomes more complex, due to the presence of taboo tables, ants may run out of nodes to choose from next, leading to deadlock. To address this deadlock issue, the present invention directly marks a node as an obstacle and withdraws it when it has only one feasible neighbor. This method effectively prevents ants from falling into U-shaped traps. Furthermore, during pathfinding, when an ant runs out of feasible nodes, it is killed, ensuring that the algorithm does not stagnate and does not increase the computational workload.

[0110] In step S205, in the step of calculating the state transition probability of ant k, the calculation formula is:

[0111]

[0112] in, is the state transition probability of ant k crawling from grid point i to grid point j; α and β are the influence factors of pheromone and heuristic information on the state transition probability, respectively, allow k Represents the set of optional paths for the node in the next stage; τ ij is the pheromone concentration value of path i to j; η ij is the heuristic information of path i to j, and the calculation formula is:

[0113]

[0114] C(n)=phi1×bend+phi2×angle

[0115]

[0116] Where Q is a constant greater than 1; C(n) is the bending suppression factor; bend is the number of bends in the path from the previous node to the next node; angle is the angle of the corner; phi1 and phi2 are the conversion coefficients of the number of turns and the degree of the turn, respectively; g(n) is the cost of node n from the starting point; S (i) represents the moving distance from the parent node of node i to i-1; h(n) is the estimated cost of node n from the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) are the target node coordinates.

[0117] As described in the above steps, the heuristic information η of path i to j ij Adopted A * The heuristic cost of the algorithm uses an estimation function, which avoids blind search and has higher search efficiency.

[0118] In one embodiment, in step S207, the preset pheromone updating method includes:

[0119] Set the initial pheromone concentration to:

[0120]

[0121] Among them, τ ij (0) is the initial pheromone concentration, c is a constant, d(j,l) is the distance between the adjacent grid j and the line l connecting the target point and the starting point. The closer the grid point is to the line l, the higher the initial pheromone concentration is, and the more the ant colony tends to choose it at the beginning;

[0122] Set the pheromone update rule as follows:

[0123]

[0124] Among them, Q1 and Q2 are constants greater than 1; L best represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the number of turns on the optimal path; x and y are the weighted coefficients of the turning angle and the number of turns.

[0125] As described in the above steps, the initial pheromone concentration of the traditional ant colony algorithm is the same everywhere, resulting in the initial ants foraging without direction, and thus a slow convergence rate. Therefore, the present invention sets a new initial pheromone concentration distribution method to guide the ants to take the optimal path; because the optimal path is more likely to be distributed in the vicinity of the line connecting the starting point and the target point, when setting the initial pheromone concentration, the pheromone concentration within this range is increased to guide the ants to find the optimal path. In addition, the traditional ant colony algorithm updates the pheromones of all ants in each iteration, resulting in a slow convergence rate. At the same time, the traditional ant colony algorithm only considers the path length factor when updating pheromones, which has the problem of considering too few factors. By comprehensively considering multiple factors such as path length, number of turns, and turning angles, the rationality of path selection is improved and unnecessary turns in the path are reduced. At the same time, the idea of ​​updating the optimal path pheromone by the maximum and minimum ant system is introduced, and an elite ant pheromone update rule for the optimal path is proposed based on multiple factors to determine the optimal path.

[0126] In one embodiment, the step of using a visual sensor to detect obstacles in the movement of the visually impaired person in real time and issuing a voice reminder to the visually impaired person based on the obstacles includes:

[0127] S301, using a visual sensor to detect obstacles in real time during the movement of the visually impaired person, and when an obstacle is detected, calculating the straight-line distance to the obstacle; wherein the straight-line distance is the shortest distance to the outline of the obstacle;

[0128] S302: When the straight-line distance is greater than a threshold, no voice reminder is issued, the visually impaired person continues to move along the initial path, and the visual sensor is returned to continuously detect obstacles in the visually impaired person's movement in real time;

[0129] S303: When the straight-line distance is less than or equal to a threshold, issuing a voice reminder to stop to the visually impaired person, so that the visually impaired person stops moving and waits for a new reminder;

[0130] S304: Determine a movement attribute of the obstacle according to the time when the obstacle is detected and the position of the obstacle, so as to determine a final position of the obstacle according to the movement attribute of the obstacle.

[0131] As described in steps S301-S304 above, the visual sensor detects obstacles in real time during the movement of the visually impaired person. When an obstacle is detected, the detection data is uploaded to calculate the straight-line distance between the visually impaired person and the obstacle (the shortest distance between the visually impaired person and the obstacle outline is used as the straight-line distance to avoid collision with the obstacle during the movement of the visually impaired person). When the straight-line distance is greater than a threshold value (set according to specific circumstances and not limited here), it indicates that the visually impaired person is not at risk of collision with the obstacle, so no voice reminder is issued and obstacle monitoring continues; when the straight-line distance is less than the threshold value, the visually impaired person may be about to collide with the obstacle, so a voice reminder is issued to the visually impaired person to stop moving. Then, based on the time when the obstacle is detected and the position of the obstacle, it can be determined whether the obstacle is fixed or mobile, and its final position can be estimated based on whether it is a fixed obstacle or mobile obstacle, in preparation for planning an obstacle avoidance path.

[0132] In one embodiment, the step of determining the movement attribute of the obstacle based on the time when the obstacle is detected and the position of the obstacle, and determining the final position of the obstacle based on the movement attribute of the obstacle, includes:

[0133] S3041. When an obstacle is detected, the visual sensor captures an initial environment image including the obstacle, and when the straight-line distance is less than or equal to a threshold, the visual sensor captures a target environment image including the obstacle.

[0134] S3042: Compare the center point positions of the obstacle in the initial environment image and the target environment image, and determine whether the difference between the center point positions exceeds a set value;

[0135] S3043: If the difference in the center point position does not exceed the set value, determine the movement attribute of the obstacle as a fixed obstacle, and use its position as the final position of the obstacle;

[0136] S3044: If the difference in the center point positions exceeds a set value, determining the movement attribute of the obstacle as a moving obstacle, and calculating the time required for the visually impaired person to move to the obstacle based on the straight-line distance as a time period;

[0137] S3045: Calculate the average speed of the obstacle during the time period, and predict the final position of the obstacle based on the average speed. This includes:

[0138] 1) The obstacle appearance time is the time period from the initial environment image to the target environment image. The average speed per unit time is calculated within the appearance time with the time period as the unit time, and then the average speed per unit time is calculated for multiple units of time to obtain the average speed within the appearance time;

[0139] 2) Estimate the time required to cross the obstacle based on the visually impaired person's moving speed, and multiply the average speed by the sum of the time period and the crossing time to obtain the distance the obstacle has moved, and ultimately obtain the final position of the obstacle.

[0140] As described in the above steps S3041-3045, when an obstacle is detected, an image is taken as the initial environment image, and when the visually impaired person is ordered to stop moving, another image is taken as the target environment image. Based on the specific position of the obstacle in the initial environment image and the target environment image, it can be determined whether the obstacle is a fixed obstacle or a mobile obstacle. The position of a fixed obstacle is fixed and the obstacle avoidance path can be planned directly based on its position and outline and the subsequently constructed obstacle environment map. The moving obstacle moves as the visually impaired person moves. Therefore, the time period for the visually impaired person to move the straight-line distance from the stop instruction position can be obtained through the visually impaired person's moving speed. During this time period, the obstacle is also moving, but the speed of the obstacle may be different. Therefore, taking the time period as the unit time, the average moving speed of the obstacle from the appearance of the obstacle to the time when the visually impaired person is about to collide with the obstacle is calculated (calculating the average speed of multiple time periods and finally calculating the total average speed can remove invalid data and make the calculation result more accurate). After obtaining the obstacle moving speed, the final position of the obstacle after the visually impaired person crosses the obstacle can be estimated, so as to prevent the visually impaired person from colliding with the obstacle during the obstacle crossing process.

[0141] In one embodiment, the step of using a Kinect sensor and a lidar to construct an obstacle environment map, and performing obstacle avoidance path planning based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, constructing a local map using the lidar includes:

[0142] S401, using a Kinect sensor to collect depth information of the environment currently containing the obstacle, and establishing a global map based on the environment; wherein the outline of the obstacle is the union of the obstacle outline in the target environment image and the obstacle outline at the final position;

[0143] S402. Using the laser radar center emission point as the coordinate origin, use the laser radar to simultaneously scan the environment and the data points of the global map. By comparing the distribution of the two data points, identify the missing data of the global map. All data points use the laser radar center emission point as the coordinate origin. This way, the coordinates of each data point obtained are clear, reducing the possibility of map creation errors. In order to connect the data points into a segment structure that can be split, use the Cartesian formula to convert the data point coordinates into global coordinates in the Cartesian system to connect the data points into a segment structure that can be split.

[0144] S403. The coordinates of the data points converted by the Cartesian formula are grouped together using a linear function to connect adjacent coordinates into vector lines that describe the linear region of the environment. The multiple vector lines divide the environment into multiple linear regions. The linear function is expressed as:

[0145]

[0146] Where N is the linear constant; y n Represents the adjacent coordinate clustering coefficient; m represents the vector line connection error; θ represents the coordinate weight of the unit data point; multiple vector lines divide the environment into multiple linear regions.

[0147] S404: The environment segmented by line segments is transformed from the original large area into small linear areas segmented by vector lines. Using the linear areas as units, a local map of the environment is constructed using the dynamic threshold method. The expression for calculating the vertical axis coordinate is:

[0148]

[0149] Among them, ε i represents the dynamic threshold constant; y0 represents the local map construction parameter; y i+1 Represents the local map construction error; the global map established by the Kinect sensor and the local map established by the lidar are overlapped using the covariance function to refine the virtual map established based on the environment. The horizontal coordinate of the local map is:

[0150]

[0151] in, represents the covariance constant; S represents the overlap coefficient between the global map and the local map; α r Represents the weight of the virtual map.

[0152] As described in steps S401-S404 above, the construction of the obstacle environment map is the basis for planning the obstacle avoidance path. In order to further enhance the integrity of the obstacle avoidance path, the construction of the obstacle environment map is divided into two levels: global and local. The global map uses depth information acquisition equipment to obtain data and provide environmental information from an overall perspective. However, more detailed data is required in the obstacle avoidance process. Therefore, the lidar, combined with laser aggregation and segmentation methods, constructs the area between any line segment and the endpoint, and constructs a local map with more detailed information based on the global map. In this way, a more accurate and complete obstacle environment map is provided, thereby better planning the obstacle avoidance path.

[0153] In one embodiment, the step of constructing an obstacle environment map using a Kinect sensor and a lidar, and performing obstacle avoidance path planning based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, performing obstacle avoidance path planning based on the obstacle environment map includes:

[0154] S411. Calculating a map complexity coefficient C according to the obstacle environment map, and obtaining a step length S according to the map complexity coefficient.

[0155]

[0156] Among them, A obstacle Indicates the area of ​​the obstacle; A map Indicates the map area; D obstacle Indicates the distribution of obstacles;

[0157] S412: The point on the obstacle closest to the initial path is taken as the target point, and the point on the initial path that is extended by a set distance from the target point in the opposite direction of the obstacle is taken as the end point N. 终 ;

[0158] S413, based on the RRT algorithm, set the random tree from the starting point N 起 Start growing, let the starting point N 起 As the root node of the random tree, a sampling point set N is randomly generated in the map 采 = {N1, N2, N3}, select the sampling point closest to the end point as the final sampling point N 采 , traverse the nodes in the random tree and find the distance N from the sampling point 采 The nearest node N 近 ; Set node N 近 Along sampling point N 采 When the direction growth step is S and there is no collision with obstacles, a new node N is obtained. 新 , node N新 Add random tree and define node N 新 The parent node N 近 ;

[0159] S414, according to the node N 新 , the search space is divided into two areas, where the end point is defined as the valid area, and the other area is defined as the invalid area. 新 As the starting point, in the next random sampling, sampling is performed in the valid area to obtain a new node N again. 新2 ;

[0160] S415. When the random tree still cannot continue to grow after the set number of iterations at the current new node, the new node is deemed invalid and deleted from the random tree. The effective sampling area of ​​the random tree is set to the effective area obtained by segmenting the previous new node. The random tree restarts growing within the updated effective sampling area until the distance between the generated new node and the end point is less than the step size S, and the line connecting the two points does not collide with any obstacles. The end point is determined to have been reached.

[0161] S416: Add the end point to the random tree, and trace back all nodes with parent nodes in the random tree to form a node set, and perform preset selection and connection according to the node order to obtain an obstacle avoidance path.

[0162] As described in the above steps S411-S416, the obstacle avoidance path is planned based on the rapidly-exploring random tree (RRT) algorithm. However, the traditional RRT algorithm has problems such as poor environmental adaptability, slow convergence speed and poor planned path quality. Therefore, the present invention improves the traditional RRT algorithm and introduces a map complexity evaluation strategy into the traditional RRT algorithm to calculate the step size that is most suitable for the corresponding map; utilizes the dynamic update strategy of the sampling area and the sampling point optimization strategy to improve the effectiveness and quality of the sampling points, so as to obtain the asymptotically optimal sampling points while retaining the randomness of the traditional RRT algorithm, thereby ensuring that the random tree grows positively toward the target point; based on the node reconnection strategy, collision detection is performed after reconnecting the initial path nodes to delete redundant nodes, so that the obstacle avoidance path is more optimized.

[0163] In one embodiment, the step of using a laser radar and a visual sensor to monitor environmental obstacles in real time on the remaining path and updating the remaining initial path according to the environmental obstacles includes:

[0164] S501: monitoring the position of the environmental obstacle in real time, and obtaining the outline of the environmental obstacle when the environmental obstacle covers the remaining path;

[0165] S502: The environmental obstacle contour divides the remaining path into an initial segment, a covered segment, and an end segment, and obtains a starting intersection point and an end intersection point between the environmental obstacle contour and the remaining path;

[0166] S503: When the starting intersection point and the ending intersection point are the points on the environmental obstacle contour closest to the starting road section and the ending road section, respectively, the starting intersection point and the ending intersection point are selected as the starting target point and the ending target point;

[0167] S504: When the starting intersection point and the ending intersection point are not the points on the environmental obstacle contour closest to the starting section and the ending section, obtain the points on the environmental obstacle contour closest to the starting section and the ending section as reference points, draw a straight line perpendicular to the starting section and the ending section through the reference points, and use the intersections of the straight line with the starting section and the ending section as the starting target point and the ending target point, discarding the starting intersection point and / or the ending intersection point closest to the target point.

[0168] S505: Draw a circle with the distance between the starting target point and the ending target point as the diameter and the center of the line segment between the starting target point and the ending target point as the center;

[0169] S506: When the circle and the environmental obstacle do not intersect, draw a chord closest to the environmental obstacle, and use the path consisting of the shortest semicircle on both sides of the remaining path plus the chord as the updated path for the covered road segment;

[0170] S507: When the circle intersects the environmental obstacle, draw a semicircle with the distance between the point on the environmental obstacle farthest from the covered road segment and the center of the circle as the radius, and use the outline of the semicircle as the updated path of the covered road segment.

[0171] As described in the above steps S501-S507, the environment of the remaining path is often accompanied by changes in the object environment (such as temporary parking, temporary operations, etc.), which will affect the remaining path in real time. Therefore, the position of the environmental obstacle is monitored in real time, and when it covers the remaining path, its contour is obtained to update the path; the environmental obstacle contour divides the remaining path into an initial section, a covered section and an end section, and a starting intersection and an ending intersection; when the obstacle contour is irregular, the starting intersection and the ending intersection may not represent the two ends of the obstacle. At this time, the starting intersection and the ending intersection cannot be directly used as the starting target point and the ending target point. It is necessary to obtain the point on the environmental obstacle contour closest to the starting section and the ending section as a reference point, and make a straight line perpendicular to the starting section and the ending section through the reference point, and use the intersection of the straight line with the starting section and the ending section as the starting target point and the ending target point, and discard the starting intersection and / or the ending intersection closest to the target point; the obstacle finally obtained The combination of the two ends of the object contour may be any two combinations of the starting intersection point, the ending intersection point, the starting target point, and the ending target point (depending on the actual situation). Finally, the distance between the starting target point and the ending target point is used as the diameter, and the center of the line segment between the starting target point and the ending target point is used as the center of the circle. When the circle does not have an intersection with the environmental obstacle, the chord closest to the environmental obstacle is drawn so that the changed chord can be used as part of the updated path to shorten the distance of the updated path, and the path of the shortest part of the semicircle + chord on both sides of the remaining path is used as the updated path of the covered section; when the circle has an intersection with the environmental obstacle, the distance between the point on the environmental obstacle farthest from the covered section and the center of the circle is used as the radius to draw a semicircle (applicable to environmental obstacles with narrow ends in the direction parallel to the path and long ends perpendicular to the path) so that the obstacle can be completely bypassed according to the contour of the semicircle, and the semicircles on the left and right sides of the remaining path are of different sizes. The contour of the small semicircle is selected as the updated path of the covered section.

[0172] The present invention uses laser radar and visual sensors to acquire environmental data, and then uses the SLAM algorithm to construct a two-dimensional grid map. The initial path for the visually impaired person is planned based on the two-dimensional grid map, and obstacles are detected in real time during the movement process to issue voice reminders to the visually impaired person. When obstacles are encountered, an obstacle environment map is constructed to plan an obstacle avoidance path, allowing the visually impaired person to overcome the obstacles and return to the initial path. Finally, the laser radar and visual sensors are used to detect environmental obstacles along the remaining path in real time to update the initial path. This method provides better navigation assistance for the visually impaired, achieving high-precision positioning, dynamic obstacle avoidance, and path planning in complex environments. It addresses the core issues of traditional visually impaired assistance devices in dynamic and complex scenes, such as incomplete environmental modeling, delayed obstacle detection, and poor path planning reliability, providing safer and more intelligent navigation assistance for the visually impaired.

[0173] like Figure 2As shown, the present invention also provides a multi-sensor based visually impaired navigation auxiliary device, comprising:

[0174] Construction module 1 is used to use lidar and visual sensors to obtain geometric distance data and optical image data of the environment, and use SLAM algorithm to build a two-dimensional grid map;

[0175] A first planning module 2 is configured to perform initial path planning in the two-dimensional grid map using a preset ant colony optimization algorithm, so that the visually impaired person moves along the initial path;

[0176] Prompt module 3, used for using visual sensors to detect obstacles in the movement of the visually impaired person in real time, and issuing voice reminders to the visually impaired person according to the obstacles;

[0177] The second planning module 4 is configured to construct an obstacle environment map using a Kinect sensor and a lidar, and perform obstacle avoidance path planning based on the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path;

[0178] The monitoring module 5 is used to monitor the environmental obstacles of the remaining path in real time using a laser radar and a visual sensor, and to update the remaining initial path according to the environmental obstacles.

[0179] In one embodiment, the first planning module 2 includes:

[0180] A determination unit is used to determine the end point of the visually impaired person's movement in the two-dimensional grid map, and initialize the ant system in the ant colony algorithm, setting the number of ants to n and the maximum number of iterations to N. max ;

[0181] An initialization unit, used to initialize k=1; wherein k is the ant sequence number, and k=1, 2, ..., n;

[0182] The update unit is used to place ant k on the current node and update the traversed nodes in the tabu table;

[0183] a processing unit configured to determine, based on the taboo table, when ant k is in a deadlock state, adopt a preset deadlock processing mechanism to process the ant, and return to the step of placing ant k on the current node and updating the traversed nodes in the taboo table;

[0184] a calculation unit, configured to calculate the state transition probability of ant k and select the next feasible node using a roulette wheel method, and when the next feasible node has not reached the end point, return to the step of placing ant k on the current node and updating the traversed nodes in the tabu table;

[0185] A judgment unit, configured to, when the next feasible node reaches the end point, determine that k is less than n, calculate k=k+1, and return to the step of placing ant k on the current node and updating the traversed nodes in the tabu table;

[0186] The pheromone update unit is used to update the pheromone using the preset pheromone update method when k is equal to n, and to determine whether the current number of iterations N is less than the maximum number of iterations N. max When , calculate N = N + 1 and return to the step of initializing k = 1;

[0187] Output unit, used to determine whether the current number of iterations N is equal to the maximum number of iterations N max When the optimal path is output as the initial path, the visually impaired person moves along the initial path.

[0188] In one embodiment, in the processing unit, using a preset deadlock handling mechanism includes:

[0189] When a node has only one feasible adjacent node, it is marked as an obstacle node and withdrawn to prevent the ants from falling into a U-shaped trap. In addition, when the ants have no feasible nodes during the pathfinding process, they are killed directly.

[0190] In the calculation unit, the calculation formula is:

[0191]

[0192] in, is the state transition probability of ant k crawling from grid point i to grid point j; α and β are the influence factors of pheromone and heuristic information on the state transition probability, respectively, allow k Represents the set of optional paths for the node in the next stage; τ ij is the pheromone concentration value of path i to j; η ij is the heuristic information of path i to j, and the calculation formula is:

[0193]

[0194] C(n)=phi1×bend+phi2×angle

[0195]

[0196] Where Q is a constant greater than 1; C(n) is the bending suppression factor; bend is the number of bends in the path from the previous node to the next node; angle is the angle of the corner; phi1 and phi2 are the conversion coefficients of the number of turns and the degree of the turn, respectively; g(n) is the cost of node n from the starting point; S (i)represents the moving distance from the parent node of node i to i-1; h(n) is the estimated cost of node n from the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) are the target node coordinates.

[0197] In one embodiment, in the pheromone updating unit, the preset pheromone updating method includes:

[0198] Set the initial pheromone concentration to:

[0199]

[0200] Among them, τ ij (0) is the initial pheromone concentration, c is a constant, d(j,l) is the distance between the adjacent grid j and the line l connecting the target point and the starting point. The closer the grid point is to the line l, the higher the initial pheromone concentration is, and the more the ant colony tends to choose it at the beginning;

[0201] Set the pheromone update rule as follows:

[0202]

[0203] Among them, Q1 and Q2 are constants greater than 1; L best represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the number of turns on the optimal path; x and y are the weighted coefficients of the turning angle and the number of turns.

[0204] In one embodiment, the prompt module 3 includes:

[0205] a detection unit, configured to detect obstacles in the movement of the visually impaired person in real time using a visual sensor, and when an obstacle is detected, calculate a straight-line distance to the obstacle; wherein the straight-line distance is the shortest distance to the outline of the obstacle;

[0206] a returning unit, configured to, when the straight-line distance is greater than a threshold, not issue a voice reminder, allowing the visually impaired person to continue moving along the initial path, and return to continue using a visual sensor to detect obstacles in the visually impaired person's movement in real time;

[0207] a stopping unit, configured to issue a voice reminder to stop to the visually impaired person when the straight-line distance is less than or equal to a threshold, so that the visually impaired person stops moving and waits for a new reminder;

[0208] The position determination unit is configured to determine a movement attribute of the obstacle according to a time when the obstacle is detected and a position of the obstacle, so as to determine a final position of the obstacle according to the movement attribute of the obstacle.

[0209] In one embodiment, the position determination unit includes:

[0210] a photographing subunit, configured to, when an obstacle is detected, cause the visual sensor to photograph an initial environment image containing the obstacle, and, when the straight-line distance is less than or equal to a threshold, cause the visual sensor to photograph a target environment image containing the obstacle;

[0211] a comparison subunit, configured to compare the center point positions of the obstacles in the initial environment image and the target environment image, and determine whether the difference between the center point positions exceeds a set value;

[0212] a fixed position determination subunit, configured to determine, when the difference in the center point position does not exceed a set value, that the movement attribute of the obstacle is a fixed obstacle, and use the position of the obstacle as the final position of the obstacle;

[0213] a time period calculation subunit, configured to determine, when the difference in the center point positions exceeds a set value, that the movement attribute of the obstacle is a moving obstacle, and calculate, based on the straight-line distance, the time required for the visually impaired person to move to the obstacle as a time period;

[0214] The average speed calculation subunit is configured to calculate the average speed during the obstacle appearance period according to the time period, so as to predict the final position of the obstacle according to the average speed; specifically comprising:

[0215] The obstacle appearance time is the time period from the initial environment image to the target environment image. The average speed per unit time is calculated within the appearance time with the time period as the unit time, and then the average speed per unit time is calculated to obtain the average speed within the appearance time.

[0216] The crossing time of the obstacle is estimated according to the moving speed of the visually impaired person, and the moving distance of the obstacle is obtained by multiplying the average speed by the sum of the time period and the crossing time, and finally the final position of the obstacle is obtained.

[0217] In one embodiment, in the second planning module 4, constructing a local map using the laser radar includes:

[0218] An establishment unit is configured to use a Kinect sensor to collect depth information of the environment currently containing the obstacle and establish a global map based on the environment; wherein the outline of the obstacle is the union of the outline of the obstacle in the target environment image and the outline of the obstacle at the final position;

[0219] A conversion unit is used to use the laser radar center emission point as the coordinate origin, use the laser radar to simultaneously scan the environment and the data points of the global map, and use the Cartesian formula to convert the coordinates of the data points into global coordinates in the Cartesian system, so as to connect the data points into a segment structure that can be segmented;

[0220] The aggregation unit is used to aggregate adjacent coordinates of the data point coordinates transformed by the Cartesian formula through a linear function, and connect them together to form vector lines describing the linear area of ​​the environment. Multiple vector lines divide the environment into multiple linear areas;

[0221] The coordinate calculation unit is used to construct a local map of the environment using the dynamic threshold method in linear regions. The expression for the vertical axis coordinate calculation point is:

[0222]

[0223] Among them, ε i represents the dynamic threshold constant; y0 represents the local map construction parameter; y i+1 Represents the local map construction error; the global map established by the Kinect sensor and the local map established by the lidar are overlapped using the covariance function to refine the virtual map established based on the environment. The horizontal coordinate of the local map is:

[0224]

[0225] in, represents the covariance constant; S represents the overlap coefficient between the global map and the local map; α r Represents the weight of the virtual map.

[0226] In one embodiment, in the second planning module 4, performing obstacle avoidance path planning according to the obstacle environment map includes:

[0227] a step length calculation unit, configured to calculate a map complexity coefficient C according to the obstacle environment map, so as to calculate a step length S according to the map complexity coefficient;

[0228]

[0229] Among them, A obstacle Indicates the area of ​​the obstacle; A map Indicates the map area; D obstacle Indicates the distribution of obstacles;

[0230] An extension unit is used to take the point on the obstacle closest to the initial path as the target point, and to extend the target point in the opposite direction of the obstacle on the initial path by a set distance as the end point N 终 ;

[0231] New node unit, used to set the random tree from the starting point N based on the RRT algorithm 起 Start growing, let the starting point N 起 As the root node of the random tree, a sampling point set N is randomly generated in the map 采 = {N1, N2, N3}, select the sampling point closest to the end point as the final sampling point N 采 , traverse the nodes in the random tree and find the distance N from the sampling point 采 The nearest node N 近 ; Set node N 近 Along sampling point N 采 When the direction growth step is S and there is no collision with obstacles, a new node N is obtained. 新 , node N 新 Add random tree and define node N 新 The parent node N 近 ;

[0232] The area definition unit is used to define the node N according to the 新 , the search space is divided into two areas, where the end point is defined as the valid area, and the other area is defined as the invalid area. 新 As the starting point, in the next random sampling, sampling is performed in the valid area to obtain a new node N again. 新2 ;

[0233] The node updating unit is used to determine that when the random tree cannot continue to grow at the current new node after a set number of iterations, the new node is invalid and deleted from the random tree, and the effective sampling area of ​​the random tree is set to the effective area obtained by segmenting the previous new node. The random tree restarts to grow in the updated effective sampling area until the distance between the generated new node and the end point is less than the step size S, and the line connecting the two points does not collide with any obstacles, and the end point is determined to have been reached;

[0234] The connection unit is used to add the end point to the random tree, and to form a node set by tracing back all nodes with parent nodes in the random tree, and to perform preset selective connections according to the node order to obtain an obstacle avoidance path.

[0235] In one embodiment, in the monitoring module 5, updating the remaining initial path according to the environmental obstacles includes:

[0236] a contour acquisition unit, configured to monitor the position of the environmental obstacle in real time and acquire the contour of the environmental obstacle when the environmental obstacle covers the remaining path;

[0237] a segment division unit, configured to divide the remaining path into an initial segment, a covered segment, and an end segment based on the environmental obstacle contour, and obtain a starting intersection point and an end intersection point between the environmental obstacle contour and the remaining path;

[0238] A first target point selection unit is configured to select the starting intersection point and the ending intersection point as the starting target point and the ending target point when the starting intersection point and the ending intersection point are the points on the environmental obstacle contour closest to the starting road section and the ending road section respectively;

[0239] a second target point selection unit, configured to, when the starting intersection point and the ending intersection point are not the points on the environmental obstacle contour closest to the starting section and the ending section, obtain the points on the environmental obstacle contour closest to the starting section and the ending section as reference points, draw a straight line perpendicular to the starting section and the ending section through the reference point, and use the intersection of the straight line with the starting section and the ending section as the starting target point and the ending target point, and discard the starting intersection point and / or the ending intersection point closest to the target point;

[0240] A circle drawing unit is used to draw a circle with the distance between the starting target point and the ending target point as the diameter and the center of the line segment between the starting target point and the ending target point as the center;

[0241] a first updating unit, configured to, when the circle does not intersect the environmental obstacle, draw a chord closest to the environmental obstacle, and use a path consisting of the shortest portions of the semicircle on both sides of the remaining path plus the chord as an updated path for the covered road segment;

[0242] The second updating unit is used to draw a semicircle with the distance between the point on the environmental obstacle farthest from the covered road section and the center of the circle as the radius when the circle has an intersection with the environmental obstacle, and use the outline of the semicircle as the update path of the covered road section.

[0243] The above modules, units and sub-units are used to execute the corresponding steps in the above multi-sensor based visually impaired assisted navigation method. The specific implementation method thereof is described in the above method embodiment and will not be repeated here.

[0244] like Figure 3 As shown, the present invention also provides a computer device, which can be a server, and its internal structure can be as follows Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store all data required for the process of the multi-sensor based visually impaired assisted navigation method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the multi-sensor based visually impaired assisted navigation method is implemented.

[0245] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0246] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, any one of the above-mentioned multi-sensor-based visually impaired assisted navigation methods is implemented.

[0247] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0248] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0249] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-sensor based visually impaired assisted navigation method, characterized in that: include: Use lidar and visual sensors to obtain the geometric distance data and optical image data of the environment, and use the SLAM algorithm to build a two-dimensional grid map; Performing initial path planning using a preset ant colony optimization algorithm in the two-dimensional grid map so that the visually impaired person moves along the initial path; Using visual sensors to detect obstacles in the movement of the visually impaired person in real time, and issuing voice reminders to the visually impaired person based on the obstacles; Using a Kinect sensor and a laser radar to construct an obstacle environment map, and performing obstacle avoidance path planning based on the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path; The laser radar and visual sensor are used to monitor the environmental obstacles of the remaining path in real time, and the remaining initial path is updated according to the environmental obstacles.

2. The multi-sensor based visually impaired assisted navigation method according to claim 1, characterized in that: The step of using a preset ant colony optimization algorithm to perform initial path planning in the two-dimensional grid map so that the visually impaired person moves along the initial path includes: In the two-dimensional grid map, determine the end point of the visually impaired person's movement, and initialize the ant system in the ant colony algorithm, setting the number of ants to n and the maximum number of iterations to N. max ; Initialize k=1; where k is the ant number, and k=1, 2, ..., n; Place ant k on the current node and update the traversed nodes in the tabu table; When ant k is in a deadlock state, it is determined according to the taboo table that ant k is in a deadlock state, a preset deadlock handling mechanism is used to handle the deadlock state, and the process returns to the step of placing ant k on the current node and updating the traversed nodes in the taboo table; Calculate the state transition probability of ant k and use the roulette wheel method to select the next feasible node. If the next feasible node has not reached the end point, return to the step of placing ant k on the current node and updating the traversed nodes in the tabu table. When the next feasible node reaches the end point, it is determined that k is less than n, k=k+1 is calculated, and the process returns to the step of placing ant k on the current node and updating the traversed nodes in the tabu table; When k is equal to n, the preset pheromone update method is used to update the pheromone, and it is determined that the current number of iterations N is less than the maximum number of iterations N. max When , calculate N = N + 1 and return to the step of initializing k = 1; Determine whether the current number of iterations N is equal to the maximum number of iterations N max When the optimal path is output as the initial path, the visually impaired person moves along the initial path.

3. The multi-sensor based visually impaired assisted navigation method according to claim 2, characterized in that: In the step of determining, according to the taboo table, when ant k falls into a deadlock state, using a preset deadlock handling mechanism to handle the situation, using the preset deadlock handling mechanism includes: When a node has only one feasible adjacent node, it is marked as an obstacle node and withdrawn to prevent the ants from falling into a U-shaped trap. In addition, when the ants have no feasible nodes during the pathfinding process, they are killed directly. In the step of calculating the state transition probability of ant k, the calculation formula is: in, is the state transition probability of ant k crawling from grid point i to grid point j; α and β are the influence factors of pheromone and heuristic information on the state transition probability, respectively, allow k Represents the set of optional paths for the node in the next stage; τ ij is the pheromone concentration value of path i to j; η ij is the heuristic information of path i to j, and the calculation formula is: C(n)=phi1×bend+phi2×angle Where Q is a constant greater than 1; C(n) is the bending suppression factor; bend is the number of bends in the path from the previous node to the next node; angle is the angle of the corner; phi1 and phi2 are the conversion coefficients of the number of turns and the degree of the turn, respectively; g(n) is the cost of node n from the starting point; S (i) represents the moving distance from the parent node of node i to i-1; h(n) is the estimated cost of node n from the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) are the target node coordinates.

4. The multi-sensor based visually impaired assisted navigation method according to claim 2, characterized in that: When k is equal to n, in the step of updating pheromones using a preset pheromone updating method, the preset pheromone updating method includes: Set the initial pheromone concentration to: Among them, τ ij (0) is the initial pheromone concentration, c is a constant, d(j,l) is the distance between the adjacent grid j and the line l connecting the target point and the starting point. The closer the grid point is to the line l, the higher the initial pheromone concentration is, and the more the ant colony tends to choose it at the beginning; Set the pheromone update rule as follows: Among them, Q1 and Q2 are constants greater than 1; L best represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the number of turns on the optimal path; x and y are the weighted coefficients of the turning angle and the number of turns.

5. The multi-sensor based visually impaired assisted navigation method according to claim 1, characterized in that: The step of using a visual sensor to detect obstacles in the movement of the visually impaired person in real time and issuing a voice reminder to the visually impaired person according to the obstacles includes: Using a visual sensor to detect obstacles in real time during the movement of the visually impaired person, and when an obstacle is detected, calculating the straight-line distance to the obstacle; wherein the straight-line distance is the shortest distance to the outline of the obstacle; When the straight-line distance is greater than the threshold, no voice reminder is issued, the visually impaired person continues to move along the initial path, and returns to continue using the visual sensor to detect obstacles in the visually impaired person's movement in real time; When the straight-line distance is less than or equal to a threshold, a voice reminder to stop is issued to the visually impaired person, so that the visually impaired person stops moving and waits for a new reminder; The movement attribute of the obstacle is determined according to the time when the obstacle is detected and the position of the obstacle, so as to determine the final position of the obstacle according to the movement attribute of the obstacle.

6. The multi-sensor based visually impaired assisted navigation method according to claim 5, characterized in that: The step of determining the movement attribute of the obstacle according to the time when the obstacle is detected and the position of the obstacle, and determining the final position of the obstacle according to the movement attribute of the obstacle, includes: When an obstacle is detected, the visual sensor captures an initial environment image including the obstacle, and when the straight-line distance is less than or equal to a threshold, the visual sensor captures a target environment image including the obstacle; Comparing the center point positions of the obstacle in the initial environment image and the target environment image, and determining whether the difference between the center point positions exceeds a set value; If the difference in the center point position does not exceed the set value, the obstacle's mobility attribute is determined to be a fixed obstacle, and its position is used as the final position of the obstacle; If the difference in the center point positions exceeds a set value, the movement attribute of the obstacle is determined to be a moving obstacle, and the time required for the visually impaired person to move to the obstacle is calculated as a time period based on the straight-line distance; Calculating the average speed of the obstacle during the time period in which the obstacle appears, so as to predict the final position of the obstacle according to the average speed; specifically comprising: The obstacle appearance time is the time period from the initial environment image to the target environment image. The average speed per unit time is calculated within the appearance time with the time period as the unit time, and then the average speed per unit time is calculated to obtain the average speed within the appearance time. The crossing time of the obstacle is estimated according to the moving speed of the visually impaired person, and the moving distance of the obstacle is obtained by multiplying the average speed by the sum of the time period and the crossing time, and finally the final position of the obstacle is obtained.

7. The multi-sensor based visually impaired assisted navigation method according to claim 6, characterized in that: The step of using a Kinect sensor and a laser radar to construct an obstacle environment map, and performing obstacle avoidance path planning based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, constructing a local map using the laser radar includes: Using a Kinect sensor to collect depth information of the environment currently containing the obstacle and to create a global map based on the environment; wherein the outline of the obstacle is the union of the outline of the obstacle in the target environment image and the outline of the obstacle at the final position; Taking the center emission point of the LiDAR as the coordinate origin, the LiDAR is used to simultaneously scan the environment and the data points of the global map. The Cartesian formula is used to convert the coordinates of the data points into global coordinates in the Cartesian system, so as to connect the data points into a segment structure that can be segmented. The coordinates of the data points transformed by the Cartesian formula are grouped together by linear functions to form vector lines that describe the linear regions of the environment. Multiple vector lines divide the environment into multiple linear regions. Taking the linear area as the unit, the dynamic threshold method is used to construct a local map of the environment. The expression of the vertical axis coordinate calculation point is: Among them, ε i represents the dynamic threshold constant; y0 represents the local map construction parameter; y i+1 Represents the local map construction error; the global map established by the Kinect sensor and the local map established by the lidar are overlapped using the covariance function to refine the virtual map established based on the environment. The horizontal coordinate of the local map is: in, represents the covariance constant; S represents the overlap coefficient between the global map and the local map; α r Represents the weight of the virtual map.

8. The multi-sensor based visually impaired assisted navigation method according to claim 1, characterized in that: The step of constructing an obstacle environment map using a Kinect sensor and a laser radar, and performing obstacle avoidance path planning based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, performing obstacle avoidance path planning based on the obstacle environment map includes: Calculating a map complexity coefficient C according to the obstacle environment map, and calculating a step size S according to the map complexity coefficient; Among them, A obstacle Indicates the area of ​​the obstacle; A map Indicates the map area; D obstacle Indicates the distribution of obstacles; The point on the obstacle closest to the initial path is taken as the target point, and the point extending the target point in the opposite direction of the obstacle on the initial path by a set distance is taken as the end point N 终 ; Based on the RRT algorithm, set the random tree from the starting point N 起 Start growing, let the starting point N 起 As the root node of the random tree, a sampling point set N is randomly generated in the map 采 = {N1, N2, N3}, select the sampling point closest to the end point as the final sampling point N 采 , traverse the nodes in the random tree and find the distance N from the sampling point 采 The nearest node N 近 ; Set node N 近 Along sampling point N 采 When the direction growth step is S and there is no collision with obstacles, a new node N is obtained. 新 , node N 新 Add random tree and define node N 新 The parent node N 近 ; According to the node N 新 , the search space is divided into two areas, where the end point is defined as the valid area, and the other area is defined as the invalid area. 新 As the starting point, in the next random sampling, sampling is performed in the valid area to obtain a new node N again. 新2 ; When the random tree still cannot continue to grow after the set number of iterations at the current new node, the new node is deemed invalid and deleted from the random tree. The effective sampling area of ​​the random tree is set to the effective area obtained by segmenting the previous new node. The random tree restarts to grow in the updated effective sampling area until the distance between the generated new node and the end point is less than the step size S, and the line connecting the two points does not collide with any obstacles. The end point is then determined to have been reached. The end point is added to the random tree, and all nodes with parent nodes in the random tree are traced back to form a node set, and the preset selection connection is performed according to the node order to obtain the obstacle avoidance path.

9. The multi-sensor based visually impaired assisted navigation method according to claim 1, characterized in that: The step of using a laser radar and a visual sensor to monitor environmental obstacles in the remaining path in real time and updating the remaining initial path according to the environmental obstacles includes: monitoring the position of the environmental obstacle in real time, and obtaining the outline of the environmental obstacle when the environmental obstacle covers the remaining path; The environmental obstacle contour divides the remaining path into an initial section, a covered section, and an end section, and obtains a starting intersection point and an end intersection point of the environmental obstacle contour and the remaining path; When the starting intersection point and the ending intersection point are the points on the outline of the environmental obstacle closest to the starting road section and the ending road section respectively, the starting intersection point and the ending intersection point are selected as the starting target point and the ending target point; When the starting intersection point and the ending intersection point are not the points on the environmental obstacle contour closest to the starting section and the ending section, the points on the environmental obstacle contour closest to the starting section and the ending section are obtained as reference points, a straight line perpendicular to the starting section and the ending section is drawn through the reference point, and the intersections of the straight line with the starting section and the ending section are used as the starting target point and the ending target point, and the starting intersection point and / or the ending intersection point closest to the target point are discarded; Use the distance between the starting target point and the ending target point as the diameter and the center of the line segment between the starting target point and the ending target point as the center of the circle; When the circle does not intersect the environmental obstacle, draw the chord closest to the environmental obstacle, and use the path of the shortest semicircle + chord on both sides of the remaining path as the updated path of the covered road segment; When the circle intersects the environmental obstacle, a semicircle is drawn with the distance between the point on the environmental obstacle farthest from the covered section and the center of the circle as the radius, and the outline of the semicircle is used as the updated path of the covered section.

10. A multi-sensor based navigation device for the visually impaired, characterized in that: include: A construction module is used to acquire geometric distance data and optical image data of the environment using lidar and vision sensors, and to construct a two-dimensional grid map using the SLAM algorithm; A first planning module is configured to perform initial path planning in the two-dimensional grid map using a preset ant colony optimization algorithm so that the visually impaired person moves along the initial path; a prompt module, configured to detect obstacles in the movement of the visually impaired person in real time using a visual sensor, and to issue a voice reminder to the visually impaired person based on the obstacles; a second planning module, configured to construct an obstacle environment map using a Kinect sensor and a lidar, and perform obstacle avoidance path planning based on the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path; The monitoring module is used to monitor the environmental obstacles of the remaining path in real time using a laser radar and a visual sensor, and to update the remaining initial path according to the environmental obstacles.

Images