Multi-sensor-based visual impairment assistance navigation method, device and computer equipment

Abstract

The application relates to the technical field of intelligent navigation, and discloses a visual impairment auxiliary navigation method based on multiple sensors, which comprises the following steps: acquiring environment data by using a laser radar and a visual sensor, and constructing a two-dimensional grid map by using a SLAM algorithm; planning an initial path for movement of a visual impairment person by using a preset ant colony star optimization algorithm; detecting obstacles in the movement process of the visual impairment person in real time to issue voice reminders; constructing an obstacle environment map and planning an obstacle avoidance path for obstacle crossing of the visual impairment person until returning to the initial path; monitoring environment obstacles of a remaining path in real time, and updating the remaining initial path; by fusing the laser radar, the visual sensor and the SLAM technology, high-precision positioning, dynamic obstacle avoidance and path planning in a complex environment are realized, and core problems such as incomplete environment modeling, obstacle detection lag and poor path planning reliability of traditional visual impairment auxiliary equipment in a dynamic complex scene are solved, so that safer and more intelligent navigation assistance is provided for visual impairment persons.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent navigation, and in particular to a visual impairment auxiliary navigation method and device based on multiple sensors and a computer device. BACKGROUND

[0002] In the field of visual impairment auxiliary navigation, helping the visually impaired to achieve safe and autonomous movement in complex environments poses an urgent need for high-precision environmental perception, real-time obstacle detection and reliable path planning. Traditional visual impairment auxiliary devices rely on a single sensor (such as ultrasonic waves, simple cameras), which has problems such as incomplete acquisition of environmental information, insufficient positioning accuracy, and weak dynamic obstacle recognition capability: it is difficult to construct a three-dimensional space model relying solely on ultrasonic sensors, and high-altitude obstacles or low obstacles may be missed in complex terrain; single visual sensors are limited by lighting conditions and viewing angle range, and have low recognition accuracy for special obstacles such as glass curtain walls and transparent objects; and independently running SLAM (simultaneous localization and mapping) technology is prone to positioning deviation accumulation due to environmental feature matching errors in dynamic human flow and complex light and shadow changes, and cannot provide real-time and reliable obstacle avoidance guidance and path planning for the visually impaired.

[0003] Therefore, there is an urgent need for a visual impairment auxiliary navigation method that integrates the advantages of multiple modal sensors to provide safer and more intelligent navigation assistance for the visually impaired. SUMMARY

[0004] The present application provides a visual impairment auxiliary navigation method and device based on multiple sensors, which realizes high-precision positioning, dynamic obstacle avoidance and path planning in complex environments by integrating laser radar (LiDAR), visual sensors (cameras) and SLAM (simultaneous localization and mapping) technology, and solves the core problems of incomplete environmental modeling, obstacle detection lag and poor path planning reliability of traditional visual impairment auxiliary devices in dynamic and complex scenes, providing safer and more intelligent navigation assistance for the visually impaired.

[0005] The present application provides a visual impairment auxiliary navigation method based on multiple sensors, comprising:

[0006] Laser radar and visual sensors are used to obtain geometric distance data and optical image data of the environment, and a SLAM algorithm is used to construct a two-dimensional grid map;

[0007] A preset ant colony star optimization algorithm is used in the two-dimensional grid map for initial path planning, so that the visually impaired person moves according to the initial path;

[0008] A visual sensor is used to detect obstacles in the movement of the visually impaired person in real time, and a voice reminder is given to the visually impaired person according to the obstacles;

[0009] The obstacle environment map is constructed by using a Kinect sensor and a laser radar, and an obstacle avoidance path is planned according to the obstacle environment map, so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path.

[0010] The laser radar and the visual sensor are used to monitor the environmental obstacles of the remaining path in real time, and the initial path is updated according to the environmental obstacles.

[0011] Further, the step of planning an initial path in the two-dimensional grid map by using a preset ant colony star optimization algorithm so that the visually impaired person moves along the initial path comprises:

[0012] In the two-dimensional grid map, the end point of the movement of the visually impaired person is determined, and an ant system in the ant colony algorithm is initialized, with the number of ants being n and the maximum number of iterations being N max ;

[0013] k is initialized to 1, wherein k is the ant number, and k = 1, 2, …, n;

[0014] The ant k is placed on the current node, and the nodes that have been traversed are updated in the taboo table;

[0015] When the ant k is in a deadlock state, a preset deadlock handling mechanism is used for handling according to the taboo table, and the step of placing the ant k on the current node and updating the nodes that have been traversed in the taboo table is returned;

[0016] The state transition probability of the ant k is calculated, and the next feasible node is selected by using a roulette method, and when the next feasible node does not reach the end point, the step of placing the ant k on the current node and updating the nodes that have been traversed in the taboo table is returned;

[0017] When the next feasible node reaches the end point, it is judged whether k is less than n, k = k + 1 is calculated, and the step of placing the ant k on the current node and updating the nodes that have been traversed in the taboo table is returned;

[0018] When k is equal to n, the pheromone is updated by using a preset pheromone updating method, and it is judged whether the current iteration number N is less than the maximum iteration number N max , N = N + 1 is calculated, and the step of initializing k = 1 is returned;

[0019] When the current iteration number N is equal to the maximum iteration number N max , the optimal path is output as the initial path so that the visually impaired person moves along the initial path.

[0020] Further, in the step of judging whether the ant k is in a deadlock state according to the tabu table and processing the ant k by using a preset deadlock processing mechanism, the preset deadlock processing mechanism includes:

[0021] When the node has only one feasible adjacent node, the node is directly marked as an obstacle node and is withdrawn to avoid the ant from falling into a U-shaped trap; and when the ant has no feasible node in the ant pathfinding process, the ant is directly killed;

[0022] In the step of calculating the state transition probability of the ant k, the calculation formula is:

[0023]

[0024] wherein, is the state transition probability of the ant k from the grid point i to the grid point j; α and β are respectively the influence factors of pheromone and heuristic information on the state transition probability; allow k represents the set of selectable paths of the node in the next stage; τ ij is the pheromone concentration value of the path i to j; η ij is the heuristic information of the path i to j, and the calculation formula is:

[0025]

[0026] C(n)=phi1×bend+phi2×angle

[0027]

[0028] wherein, Q is a constant greater than 1; C(n) is a bending inhibition factor; bend is the number of path bends from the previous node to the next node; angle is the included angle of the corner; phi1 and phi2 are respectively the conversion coefficients of the number of turns and the degree of turning; g(n) is the cost of the node n from the starting point; S (i) represents the moving distance of the parent node of the node i to i-1; h(n) is the estimated cost of the node n from the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) is the coordinate of the target node.

[0029] Further, in the step of updating the pheromone by using a preset pheromone updating method when k is equal to n, the preset pheromone updating method includes:

[0030] The initial pheromone concentration is set as:

[0031]

[0032] wherein, τ ij (0) is the initial pheromone concentration, c is a constant, d(j, l) is the distance between adjacent grid j and the connecting line l between the target point and the starting point, the closer to the connecting line l, the higher the initial pheromone concentration, and the more the ant colony tends to choose in the initial stage;

[0033] The pheromone update rule is set as:

[0034]

[0035] wherein, Q1, Q2 are constants greater than 1; L best represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the number of turns on the optimal path; x, y are weight proportion coefficients of the turning angle and the number of turns.

[0036] Further, the step of detecting obstacles in the movement of the visually impaired person in real time by using the visual sensor and issuing a voice reminder to the visually impaired person according to the obstacles comprises:

[0037] The visual sensor is used to detect obstacles in the movement of the visually impaired person in real time, and when an obstacle is detected, a straight-line distance from the obstacle is calculated; wherein the straight-line distance is the shortest distance from the outline of the obstacle;

[0038] When the straight-line distance is greater than a threshold value, no voice reminder is issued, the visually impaired person continues to move along the initial path, and the visual sensor is used to continue to detect obstacles in the movement of the visually impaired person in real time;

[0039] When the straight-line distance is less than or equal to the threshold value, a voice reminder is issued to the visually impaired person to stop, so that the visually impaired person stops moving and waits for a new reminder;

[0040] The movement attribute of the obstacle is determined according to the time when the obstacle is detected and the position of the obstacle, so as to determine the final position of the obstacle according to the movement attribute of the obstacle.

[0041] Further, the step of determining the movement attribute of the obstacle according to the time when the obstacle is detected and the position of the obstacle, so as to determine the final position of the obstacle according to the movement attribute of the obstacle, comprises:

[0042] When the obstacle is detected, the visual sensor captures an initial environment image with the obstacle, and when the straight-line distance is less than or equal to the threshold value, the visual sensor captures a target environment image with the obstacle;

[0043] comparing the center point positions of the obstacle in the initial environment image and the target environment image, and determining whether the difference of the center point positions exceeds a set value;

[0044] if the difference of the center point positions does not exceed the set value, determining that the moving attribute of the obstacle is a fixed obstacle, and taking the position as the final position of the obstacle;

[0045] if the difference of the center point positions exceeds the set value, determining that the moving attribute of the obstacle is a moving obstacle, and calculating the required time for the visually impaired person to move to the obstacle as a time period according to the straight line distance;

[0046] calculating the average speed in the appearance time of the obstacle according to the time period, so as to predict the final position of the obstacle according to the average speed; specifically comprising:

[0047] the appearance time of the obstacle is the time period from the initial environment image to the target environment image, the average speed of each unit time is calculated in the unit time within the appearance time, and the average speed in the appearance time is obtained by calculating the average speeds of multiple unit times;

[0048] the crossing time of the visually impaired person crossing the obstacle is estimated according to the moving speed of the visually impaired person, and the moving distance of the obstacle is obtained by multiplying the sum of the time period and the crossing time by the average speed, so as to obtain the final position of the obstacle.

[0049] Further, the obstacle environment map is constructed by using the Kinect sensor and the laser radar, and the obstacle avoidance path planning is performed according to the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path, and in the step of returning to the initial path, the local map is constructed by using the laser radar, comprising:

[0050] the depth information of the environment where the obstacle is located is collected by using the Kinect sensor, and a global map based on the environment is established; wherein the outline of the obstacle is the union of the obstacle outline in the target environment image and the obstacle outline in the final position;

[0051] the center of the laser radar emission point is taken as the coordinate origin, the data points of the environment and the global map are scanned by using the laser radar, the data point coordinates are converted into global coordinates in the Cartesian system by using the Cartesian formula, and the data points are connected into a line segment structure that can be segmented;

[0052] the data point coordinates converted by the Cartesian formula are gathered together by a linear function, and are connected together to form a vector line describing the linear region of the environment, and multiple vector lines divide the environment into multiple linear regions;

[0053] In linear region, the local map of the environment is constructed by using dynamic threshold method, and the expression of the longitudinal coordinate calculation point is:

[0054]

[0055] wherein, ε i represents the dynamic threshold constant; y0 represents the local map construction parameter; y i+1 represents the local map construction error; the global map established by the Kinect sensor and the local map established by the laser radar are overlapped by using the covariance function, so that the virtual map established according to the environment is refined, and the horizontal coordinate of the local map is:

[0056]

[0057] wherein, represents the covariance constant; S represents the overlapping coefficient of the global map and the local map; α r represents the weight of the virtual map.

[0058] Further, the step of constructing the obstacle environment map by using the Kinect sensor and the laser radar, and planning the obstacle avoidance path according to the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path until returning to the initial path, wherein planning the obstacle avoidance path according to the obstacle environment map comprises:

[0059] calculating the map complexity coefficient C according to the obstacle environment map, so as to calculate the step length S according to the map complexity coefficient;

[0060]

[0061] wherein, A obstacle represents the obstacle area; A map represents the map area; D obstacle represents the obstacle distribution;

[0062] the point on the obstacle closest to the initial path is taken as the target point, and the point on the initial path with a set distance in the opposite direction of the obstacle from the target point is taken as the end point N 终 ;

[0063] based on the RRT algorithm, a random tree is set to grow from the starting point N 起 , and the starting point N 起 is taken as the root node of the random tree, a set of sampling points N 采 ={N1,N2,N3} is randomly generated in the map, the sampling point closest to the end point is selected as the final sampling point N 采 , the nodes in the random tree are traversed, and the sampling point N 采The latest node N 近 ; the node N 近 is added to the random tree, and the node N 采 is defined as the parent node of the node N 新 ; 新 , the search space is divided into two regions according to the node N 新 , and the region where the end point is located is defined as the effective region, and the other region is defined as the invalid region, and the node N 新 is taken as the starting point to sample in the effective region in the next random sampling, so as to obtain a new node N 近 again; 近

[0064] According to the node N 新 , the search space is divided into two regions, and the region where the end point is located is defined as the effective region, and the other region is defined as the invalid region, and the node N 新 is taken as the starting point to sample in the effective region in the next random sampling, so as to obtain a new node N 新2 again; 新2

[0065] When the random tree cannot continue to grow after a certain number of iterations of the current new node, it is determined that the new node is invalid, and it is deleted from the random tree, and the effective sampling region of the random tree is set to the effective region obtained based on the last new node, and the random tree starts to grow again in the updated effective sampling region until the distance between the generated new node and the end point is less than the step length S, and the line connecting the two points does not collide with the obstacle, and it is determined that the end point is reached;

[0066] The end point is added to the random tree, and all nodes with parent nodes in the random tree are traced back to form a node set, and a preset selection connection is performed according to the node order to obtain an obstacle avoidance path.

[0067] Further, the step of monitoring the environment obstacle of the remaining path in real time by using the laser radar and the visual sensor, and updating the remaining initial path according to the environment obstacle, comprises:

[0068] The position of the environment obstacle is monitored in real time, and when the environment obstacle covers the remaining path, the profile of the environment obstacle is obtained;

[0069] The environment obstacle profile divides the remaining path into an initial section, a covered section and an end section, and obtains the starting intersection point and the ending intersection point of the environment obstacle profile and the remaining path;

[0070] When the starting intersection point and the ending intersection point are the points on the environment obstacle profile closest to the starting section and the ending section respectively, the starting intersection point and the ending intersection point are selected as the starting target point and the ending target point;

[0071] When the start intersection point and the end intersection point are not the points on the environmental obstacle profile closest to the start path segment and the end path segment, the points on the environmental obstacle profile closest to the start path segment and the end path segment are obtained as reference points, a straight line perpendicular to the start path segment and the end path segment is drawn through the reference points, and the intersection points of the straight line and the start path segment and the end path segment are taken as the start target point and the end target point, and the start intersection point and / or the end intersection point closest to the target point are discarded;

[0072] A circle is drawn with the distance between the start target point and the end target point as a diameter and the center of the line segment between the start target point and the end target point as a center;

[0073] When the circle does not have intersection points with the environmental obstacle, the shortest chord closest to the environmental obstacle is drawn, and the path of the shortest half circle + chord on both sides of the remaining path is taken as the updated path of the coverage path segment;

[0074] When the circle has intersection points with the environmental obstacle, a half circle is drawn with the distance between the point on the environmental obstacle farthest from the coverage path segment and the center of the circle as a radius, and the profile of the half circle is taken as the updated path of the coverage path segment.

[0075] The application also provides a visual impairment auxiliary navigation device based on multiple sensors, comprising:

[0076] A construction module is configured to acquire geometric distance data and optical image data of an environment by using a laser radar and a visual sensor, and construct a two-dimensional grid map by using a SLAM algorithm;

[0077] A first planning module is configured to plan an initial path in the two-dimensional grid map by using a preset ant colony star optimization algorithm, so that a visually impaired person moves according to the initial path;

[0078] A prompting module is configured to detect obstacles in the movement of the visually impaired person in real time by using a visual sensor, and give voice prompts to the visually impaired person according to the obstacles;

[0079] A second planning module is configured to construct an obstacle environment map by using a Kinect sensor and a laser radar, and plan an obstacle avoidance path according to the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path until returning to the initial path;

[0080] A monitoring module is configured to monitor environmental obstacles of a remaining path in real time by using a laser radar and a visual sensor, and update the remaining initial path according to the environmental obstacles.

[0081] The application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0082] The application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method when executed by a processor.

[0083] The application has the following beneficial effects:

[0084] The application acquires environmental data by using a laser radar and a visual sensor, constructs a two-dimensional grid map by using a SLAM algorithm, plans an initial path for a visually impaired person to move based on the two-dimensional grid map, detects obstacles in a moving process in real time, and issues a voice prompt to the visually impaired person; when there is an obstacle, an obstacle environmental map is constructed to plan an obstacle avoidance path, so that the visually impaired person can cross the obstacle to return to the initial path; finally, the laser radar and the visual sensor are used to detect environmental obstacles of a remaining path in real time to update the initial path. The application better provides navigation assistance for the visually impaired person, realizes high-precision positioning, dynamic obstacle avoidance and path planning in a complex environment, solves core problems such as incomplete environmental modeling, obstacle detection lag and poor path planning reliability of a traditional visually impaired assistance device in a dynamic complex scene, and provides safer and more intelligent navigation assistance for the visually impaired person. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 The figure is a method flowchart of an embodiment of the application.

[0086] Figure 2 The figure is a device structure schematic diagram of an embodiment of the application.

[0087] Figure 3 The figure is a computer device internal structure schematic diagram of an embodiment of the application.

[0088] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0089] It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0090] As shown in the figure, the application provides a visually impaired assistance navigation method based on multiple sensors, comprising: Figure 1

[0091] S1, acquiring geometric distance data and optical image data of an environment by using a laser radar and a visual sensor, and constructing a two-dimensional grid map by using a SLAM algorithm;

[0092] ​S2, initial path planning is performed in the two-dimensional grid map by using a preset ant colony star optimization algorithm, so that the visually impaired person moves according to the initial path;

[0093] S3, obstacles in the moving process of the visually impaired person are detected in real time by using a visual sensor, and voice prompts are given to the visually impaired person according to the obstacles;

[0094] S4, an obstacle environment map is constructed by using a Kinect sensor and a laser radar, and an obstacle avoidance path is planned according to the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path until returning to the initial path;

[0095] S5, the remaining path is monitored in real time by using a laser radar and a visual sensor, and the initial path is updated according to the environmental obstacles.

[0096] As described in the above steps S1-S5, the laser radar LiDAR can collect the geometric distance data of the environment, and the visual sensor (camera) can collect the optical image data of the environment. The geometric distance data and the optical image data of the environment are used to construct a two-dimensional grid map by using a SLAM algorithm. The core idea of SLAM (Simultaneous Localization and Mapping) is that when moving in an unknown environment, the environment is observed in real time by a sensor (laser radar, camera), the position and attitude of the sensor are estimated synchronously, and an environment map is constructed. In the present application, a two-dimensional grid map is constructed by using SLAM, so that the initial path planning is performed by using a preset ant colony star optimization algorithm. The visually impaired person can move according to the initial path, and the visual sensor is used to detect obstacles in real time during the movement, and then an obstacle environment map is constructed, so that the position of the obstacle can be determined, the obstacle avoidance path is planned, and the visually impaired person can pass through the obstacle to return to the initial path. Meanwhile, the laser radar and the visual sensor also monitor the obstacles of the remaining initial path, so as to update the initial path according to the obstacles.

[0097] In one embodiment, the step of performing initial path planning in the two-dimensional grid map by using a preset ant colony star optimization algorithm, so that the visually impaired person moves according to the initial path, comprises:

[0098] S201, in the two-dimensional grid map, the end point of the movement of the visually impaired person is determined, and an ant system in the ant colony algorithm is initialized, the number of ants is set to n, and the maximum number of iterations is set to N max ;

[0099] S202, initialize k=1; wherein k is the ant serial number, and k=1, 2, …, n;

[0100] S203, placing the ant k on the current node and updating the node that has been traversed in the tabu table;

[0101] S204, judging whether the ant k falls into a deadlock state according to the tabu table, adopting a preset deadlock processing mechanism to process, and returning to step S203;

[0102] S205, calculating the state transition probability of the ant k, and selecting the next feasible node using the roulette method, when the next feasible node does not reach the terminal point, returning to step S203;

[0103] S206, when the next feasible node reaches the terminal point, judging whether k is less than n, calculating k=k+1, and returning to step S203;

[0104] S207, when k is equal to n, adopting a preset pheromone updating method to update the pheromone, and judging whether the current iteration number N is less than the maximum iteration number N max , calculating N=N+1, and returning to step S202;

[0105] S208, judging whether the current iteration number N is equal to the maximum iteration number N max , outputting the optimal path as the initial path, so that the visually impaired person moves according to the initial path.

[0106] As described in steps S201-S208, the improved ant colony star optimization algorithm combines the ant colony algorithm and A * algorithm to find the optimal path as the initial path. The ant colony algorithm is a bionics algorithm inspired by the foraging of ants, which establishes the relationship between the amount of pheromone released by ants and the path length, so that the cumulative concentration of pheromone on the shorter path is proportional to time, attracting more ants to choose that path, so as to realize the discovery of the optimal foraging path. * The A * algorithm uses the heuristic cost estimation function equation to update the heuristic information in the ant colony algorithm to avoid blind search. The A * algorithm has excellent optimization ability and search efficiency, and its heuristic intelligent search idea is integrated into the heuristic function of the ant colony algorithm to search for the optimal path. With the help of the estimated moving cost, the ant can reach the target point faster, and the bending suppression operator added in the heuristic function can reduce the number of bends and the cumulative turning angle.

[0107] In one embodiment, in step S204, the preset deadlock processing mechanism includes:

[0108] When the node has only one feasible adjacent node, the node is directly marked as an obstacle node and withdrawn to avoid the ant falling into a U-shaped trap; and when the ant has no feasible node during the ant's path search, the ant is directly killed.

[0109] When the movement environment is more complex, the next step of the ant may have no node to select due to the taboo table, thereby falling into a deadlock. For the deadlock problem, the application sets, when a node has only one feasible adjacent node, the node is directly marked as an obstacle node and is withdrawn, which can effectively avoid the ant falling into a U-shaped trap. In addition, in the pathfinding process, when the ant has no feasible node, the ant is directly killed, so as to ensure that the algorithm does not stop and does not increase the calculation amount.

[0110] In step S205, the calculation formula in the step of calculating the state transition probability of the ant k is:

[0111]

[0112] wherein, is the state transition probability of the ant k from the grid point i to the grid point j; α and β are respectively the influence factors of pheromone and heuristic information on the state transition probability; allow k represents the set of selectable paths of the next stage of the node; τ ij is the pheromone concentration value of the path i to j; η ij is the heuristic information of the path i to j, and the calculation formula is:

[0113]

[0114] C(n) = phi1 x bend + phi2 x angle

[0115]

[0116] wherein, Q is a constant greater than 1; C(n) is a bending inhibition factor; bend is the number of path bends from the previous node to the next node; angle is the included angle of the corner; phi1 and phi2 are respectively the conversion coefficients of the number of turns and the degree of turning; g(n) is the cost of the node n from the starting point; S (i) represents the moving distance of the parent node of the node i to i-1; h(n) is the estimated cost of the node n from the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) is the coordinate of the target node.

[0117] As described in the above steps, the heuristic information η ij The heuristic cost of the A * algorithm uses an estimation function, which avoids blind search and has high search efficiency.

[0118] In one embodiment, in step S207, the preset pheromone updating method comprises:

[0119] The initial pheromone concentration is set as:

[0120]

[0121] Wherein, τ ij (0) is the initial pheromone concentration, c is a constant, and d(j, l) is the distance between the adjacent grid j and the connecting line l between the target point and the starting point. The closer the grid point is to the connecting line l, the higher the initial pheromone concentration is, and the more the ant colony tends to select in the initial stage;

[0122] The pheromone updating rule is set as:

[0123]

[0124] Wherein, Q1 and Q2 are constants greater than 1; L best represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the turning times on the optimal path; x and y are weight proportion coefficients of the turning angle and the turning time.

[0125] As described in the above steps, the initial pheromone concentration of the traditional ant colony algorithm is the same everywhere, which leads to the initial ants foraging without direction, thereby the convergence speed is slow. Therefore, the new initial pheromone concentration distribution method is set in the application to guide the ants to walk the optimal path. Since the optimal path is more likely to be distributed in the vicinity of the connecting line between the starting point and the target point, the pheromone concentration in this range is increased when setting the initial pheromone concentration to guide the ants to optimize. Moreover, the traditional ant colony algorithm updates the pheromones of all ants in each iteration, which is slow in convergence speed. Meanwhile, the traditional ant colony algorithm only considers the path length factor in pheromone updating, which has the problem of too single consideration factor. The multiple factors such as path length, turning times and turning angle are comprehensively considered to improve the rationality of path selection and reduce unnecessary turning of the path. Meanwhile, the maximum and minimum ant system updating optimal path pheromone idea is introduced, and the elite ant pheromone updating rule of the optimal path method with multiple factor judgment is proposed.

[0126] In one embodiment, the step of using a visual sensor to detect obstacles in the movement of the visually impaired person in real time and issuing a voice reminder to the visually impaired person according to the obstacles comprises:

[0127] S301, using a visual sensor to detect obstacles in the movement of the visually impaired person in real time, and calculating the straight-line distance from the obstacle when detecting the obstacle; wherein the straight-line distance is the shortest distance from the outline of the obstacle;

[0128] S302, when the straight line distance is greater than a threshold value, no voice prompt is given, the visually impaired person continues to move along the initial path, and returns to continue to use the visual sensor to detect obstacles in real time during the movement of the visually impaired person;

[0129] S303, when the straight line distance is less than or equal to a threshold value, a voice prompt is given to the visually impaired person to stop, so that the visually impaired person stops moving and waits for a new prompt;

[0130] S304, determining the movement attribute of the obstacle according to the time when the obstacle is detected and the position of the obstacle, so as to determine the final position of the obstacle according to the movement attribute of the obstacle.

[0131] As described in the above steps S301-S304, the visual sensor detects obstacles in real time during the movement of the visually impaired person, when the obstacle is detected, the detection data is uploaded to calculate the straight line distance between the visually impaired person and the obstacle (the shortest distance between the visually impaired person and the outline of the obstacle is taken as the straight line distance, which can avoid collision between the visually impaired person and the obstacle during movement), when the straight line distance is greater than a threshold value (which is set according to specific circumstances, which is not limited here), it indicates that there is no risk of collision between the visually impaired person and the obstacle, so no voice prompt is given and the obstacle is continuously monitored; when the straight line distance is less than the threshold value, the visually impaired person may collide with the obstacle, so a voice prompt is given to the visually impaired person to stop moving, and then according to the time when the obstacle is detected and the position of the obstacle, it can be determined whether the obstacle is a fixed obstacle or a moving obstacle, and the final position of the fixed obstacle or the moving obstacle can be estimated to prepare for planning an obstacle avoidance path.

[0132] In one embodiment, the step of determining the movement attribute of the obstacle according to the time when the obstacle is detected and the position of the obstacle, so as to determine the final position of the obstacle according to the movement attribute of the obstacle, comprises:

[0133] S3041, when the obstacle is detected, the visual sensor captures an initial environment image with the obstacle, and when the straight line distance is less than or equal to a threshold value, the visual sensor captures a target environment image with the obstacle;

[0134] S3042, comparing the center point positions of the obstacles in the initial environment image and the target environment image, and determining whether the difference of the center point positions exceeds a set value;

[0135] S3043, if the difference of the center point positions does not exceed the set value, it is determined that the movement attribute of the obstacle is a fixed obstacle, and the position thereof is taken as the final position of the obstacle;

[0136] S3044, if the difference of the center point positions exceeds a set value, determining that the moving attribute of the obstacle is a moving obstacle, and calculating the required time for the visually impaired person to move to the obstacle as a time period according to the straight line distance;

[0137] S3045, calculating the average speed in the appearance time of the obstacle according to the time period, to predict the final position of the obstacle according to the average speed; specifically including:

[0138] 1) The appearance time of the obstacle is the time period from the initial environment image to the target environment image, and the average speed per unit time is calculated in the time period, and then the average speed in the appearance time is obtained by calculating the average speed of multiple unit times;

[0139] 2) The crossing time of the obstacle is estimated according to the moving speed of the visually impaired person, and the moving distance of the obstacle is obtained by multiplying the sum of the time period and the crossing time by the average speed, and the final position of the obstacle is finally obtained.

[0140] As described in steps S3041-3045 above, when the obstacle is detected, an image is taken as an initial environment image, and when the visually impaired person is instructed to stop moving, another image is taken as a target environment image. According to the specific position of the obstacle in the initial environment image and the target environment image, it can be determined whether the obstacle is a fixed obstacle or a moving obstacle. The position of the fixed obstacle is fixed and unchangeable, and the obstacle avoidance path can be planned directly according to its position and contour and the subsequent constructed obstacle environment map. The moving obstacle moves with the visually impaired person, so the time period for the visually impaired person to move a straight line distance from the position of the stop instruction can be obtained by the moving speed of the visually impaired person. In this time period, the obstacle is also moving, but the speed of the obstacle may vary. Therefore, the time period is taken as a unit of time, and the average moving speed of the obstacle in the time period from the appearance of the obstacle to the collision of the visually impaired person with the obstacle is calculated (the average speed of multiple time periods is calculated, and finally the total average speed is calculated, which can remove invalid data and make the calculation result more accurate). After obtaining the moving speed of the obstacle, the final position of the obstacle after the visually impaired person crosses the obstacle can be estimated, so as to avoid the collision of the visually impaired person with the obstacle during the crossing process.

[0141] In one embodiment, the steps of constructing an obstacle environment map using a Kinect sensor and a laser radar, and planning an obstacle avoidance path according to the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path, until returning to the initial path, the local map constructed by the laser radar includes:

[0142] S401, depth information of an environment containing an obstacle is collected by using a Kinect sensor, and a global map based on the environment is established; wherein, the outline of the obstacle is the union of the outline of the obstacle in the target environment image and the outline of the obstacle in the final position;

[0143] S402, taking the center of the laser radar as the coordinate origin, the data points of the global map and the environment scanned by the laser radar are used to identify the missing data of the global map by comparing the distribution of the two data points. All data points are taken as the coordinate origin of the center of the laser radar, so that the coordinate of each data point obtained is clear, reducing the possibility of map creation error. In order to connect the data points into a segmented line segment structure, the Cartesian formula is used to convert the data point coordinates into global coordinates in Cartesian system, so as to connect the data points into a segmented line segment structure;

[0144] S403, the data point coordinates converted by the Cartesian formula are gathered together by a linear function to form a vector line describing the linear region of the environment, and a plurality of vector lines divide the environment into a plurality of linear regions; the expression of the linear function is:

[0145]

[0146] wherein, N represents a linear constant; y n represents the gathering coefficient of adjacent coordinates; m represents the connection error of the vector line; θ represents the coordinate weight of the unit data point; a plurality of vector lines divide the environment into a plurality of linear regions.

[0147] S404, the environment segmented by the line segment is converted from the original large area region into a small area linear region segmented by the vector line. Taking the linear region as a unit, a dynamic threshold method is used to construct a local map of the environment, and the expression of the longitudinal coordinate calculation point is:

[0148]

[0149] wherein, ε i represents a dynamic threshold constant; y0 represents a local map construction parameter; y i+1 represents a local map construction error; the global map established by the Kinect sensor and the local map established by the laser radar are overlapped by using the covariance function, so that the virtual map established according to the environment is refined, and the horizontal coordinate of the local map is:

[0150]

[0151] wherein, represents a covariance constant; S represents an overlapping coefficient of the global map and the local map; α r represents the weight of the virtual map.

[0152] As described in steps S401-S404 above, obstacle environment map construction is the foundation for planning obstacle avoidance paths. To further enhance the completeness of obstacle avoidance paths, obstacle environment map construction is divided into two levels: global and local. The global map uses depth information acquisition equipment to obtain data, providing overall environmental information. However, more detailed data is needed during obstacle avoidance. Therefore, LiDAR, combined with laser aggregation and segmentation methods, is used to construct the region between arbitrary line segments and endpoints, and a local map with more detailed information is constructed based on the global map. This approach provides a more accurate and complete obstacle environment map, thereby enabling better obstacle avoidance path planning.

[0153] In one embodiment, the step of constructing an obstacle environment map using Kinect sensors and LiDAR, and planning an obstacle avoidance path based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path, includes planning the obstacle avoidance path based on the obstacle environment map as follows:

[0154] S411. Calculate the map complexity coefficient C based on the obstacle environment map, and calculate the step size S based on the map complexity coefficient;

[0155]

[0156] Among them, A obstacle Indicates the area of ​​the obstacle; A map Indicates the area of ​​the map; D obstacle Indicates the distribution of obstacles;

[0157] S412. Take the point on the obstacle closest to the initial path as the target point, and take the point on the initial path that extends a predetermined distance in the opposite direction of the obstacle as the endpoint N. 终 ;

[0158] S413. Based on the RRT algorithm, the random tree starts from the starting point N. 起 Start growing, let the starting point N 起 As the root node of the random tree, a set N of sampling points is randomly generated in the map. 采 ={N1,N2,N3}, select the sampling point closest to the endpoint as the final sampling point N. 采 Traverse the nodes in the random tree to find the distance N from the sampling point. 采 The nearest node N 近 ; Node N 近 Along sampling point N 采 When the directional growth step size is S and there is no collision with obstacles, a new node N is obtained. 新 , will node N新 Add a random tree, and define a node N 新 , as the parent node N 近 ;

[0159] S414, according to the node N 新 , the search space is divided into two regions, and the region where the end point is located is defined as the effective region, and the other region is defined as the invalid region, and the node N 新 is taken as the starting point to sample in the effective region when the next random sampling is performed, so that a new node N 新2 is obtained again.

[0160] S415, when the random tree cannot continue to grow after the current new node is iterated for a set number of times, it is determined that the new node is invalid, and it is deleted from the random tree, and the effective sampling region of the random tree is set to the effective region obtained based on the last new node, and the random tree starts to grow again in the updated effective sampling region until the distance between the generated new node and the end point is less than the step length S, and the line connecting the two points does not collide with the obstacle, and it is determined that the end point is reached.

[0161] S416, the end point is added to the random tree, and all nodes with parent nodes in the random tree are traced back to form a node set, and a preset selection connection is performed according to the node order to obtain an obstacle avoidance path.

[0162] As described above in steps S411-S416, in planning the obstacle avoidance path, the rapidly-exploring random tree (RRT) algorithm is used as the basis, but the traditional RRT algorithm has problems such as poor environmental adaptability, slow convergence speed, and poor path planning quality, so the traditional RRT algorithm is improved in the present application, and a map complexity evaluation strategy is introduced into the traditional RRT algorithm to calculate the most suitable step length for the corresponding map; the effectiveness and quality of the sampling points are improved by using a sampling region dynamic updating strategy and a sampling point optimization strategy, so that the progressive optimal sampling points are obtained while the randomness of the traditional RRT algorithm is preserved, thereby ensuring the random tree to grow towards the target point; based on the node reconnection strategy, collision detection is performed after reconnecting the initial path nodes to delete redundant nodes, so that the obstacle avoidance path is more optimized.

[0163] In one embodiment, the step of using a laser radar and a visual sensor to monitor the environmental obstacles of the remaining path in real time and updating the remaining initial path according to the environmental obstacles, wherein the step of updating the remaining initial path according to the environmental obstacles, comprises:

[0164] S501, the position of the environmental obstacle is monitored in real time, and when the environmental obstacle covers the remaining path, the contour of the environmental obstacle is obtained.

[0165] S502, the environmental obstacle profile divides the remaining path into an initial section, a covering section and an ending section, and obtains a starting intersection point and an ending intersection point of the environmental obstacle profile and the remaining path;

[0166] S503, when the starting intersection point and the ending intersection point are respectively the points on the environmental obstacle profile closest to the initial section and the ending section, the starting intersection point and the ending intersection point are selected as a starting target point and an ending target point;

[0167] S504, when the starting intersection point and the ending intersection point are not the points on the environmental obstacle profile closest to the initial section and the ending section, a reference point on the environmental obstacle profile closest to the initial section and the ending section is obtained, a straight line perpendicular to the initial section and the ending section is drawn through the reference point, an intersection point of the straight line and the initial section and the ending section is taken as the starting target point and the ending target point, and the starting intersection point and / or the ending intersection point closest to the target point is discarded;

[0168] S505, a circle is drawn with a distance between the starting target point and the ending target point as a diameter and a line segment center between the starting target point and the ending target point as a center;

[0169] S506, when the circle does not have an intersection point with the environmental obstacle, a chord closest to the environmental obstacle is drawn, and a path of the shortest half circle and chord on both sides of the environmental obstacle is taken as an updated path of the covering section;

[0170] S507, when the circle has an intersection point with the environmental obstacle, a semicircle is drawn with a distance between a point on the environmental obstacle farthest from the covering section and the center of the circle as a radius, and a profile of the semicircle is taken as an updated path of the covering section.

[0171] As described in steps S501-S507 above, the remaining path environment is often accompanied by changes in the object environment (such as temporary parking, temporary work, etc.), which will affect the remaining path in real time. Therefore, the position of environmental obstacles is monitored in real time, and when they cover the remaining path, their outlines are obtained to update the path. The environmental obstacle outline divides the remaining path into the initial segment, the covered segment, and the ending segment, and identifies the starting and ending intersection points. When the obstacle outline is irregular, the starting and ending intersection points may not represent the two ends of the obstacle. In this case, the starting and ending intersection points cannot be directly used as the starting and ending target points. It is necessary to obtain the point on the environmental obstacle outline that is closest to the starting and ending segments as a reference point. A straight line perpendicular to the starting and ending segments is drawn through the reference point, and the intersection of this straight line with the starting and ending segments is used as the starting and ending target points. The starting and / or ending intersection points that are closest to the target points are discarded. The final obstacle is obtained. The combination of the two ends of the object's outline can be any two of the following: starting intersection point, ending intersection point, starting target point, and ending target point (depending on the actual situation). Finally, a circle is drawn with the distance between the starting target point and the ending target point as the diameter and the center of the line segment between the starting target point and the ending target point as the center. When the circle does not intersect with the environmental obstacle, the chord closest to the environmental obstacle is drawn so that the chord can be used as part of the update path, shortening the update path. The path of the shortest semicircle on both sides of the remaining path plus the chord is used as the update path of the covered road segment. When the circle intersects with the environmental obstacle, a semicircle is drawn with the distance between the point on the environmental obstacle farthest from the covered road segment and the center of the circle as the radius (applicable to environmental obstacles that are narrow at both ends in the direction parallel to the path and long at both ends perpendicular to the path), so that the semicircle outline can completely bypass the obstacle. If the semicircles on the left and right sides of the remaining path are of different sizes, the outline of the smaller semicircle is selected as the update path of the covered road segment.

[0172] This invention employs LiDAR and visual sensors to acquire environmental data, then uses SLAM algorithms to construct a two-dimensional grid map. Based on this map, an initial path is planned for the visually impaired user's movement. Obstacles are detected in real-time, and voice prompts are provided to the user. When obstacles are present, an obstacle environment map is constructed to plan obstacle avoidance paths, enabling the user to overcome obstacles and return to the initial path. Finally, LiDAR and visual sensors are used to detect environmental obstacles on the remaining path in real-time to update the initial path. This provides better navigation assistance for the visually impaired, achieving high-precision positioning, dynamic obstacle avoidance, and path planning in complex environments. It solves the core problems of traditional visually impaired assistive devices, such as incomplete environmental modeling, delayed obstacle detection, and poor path planning reliability in dynamic and complex scenes, providing safer and more intelligent navigation assistance for visually impaired individuals.

[0173] like Figure 2As shown, the application also provides a multi-sensor-based visual impairment auxiliary navigation device, comprising:

[0174] A construction module 1 is configured to acquire geometric distance data and optical image data of an environment by using a laser radar and a visual sensor, and construct a two-dimensional grid map by using a SLAM algorithm;

[0175] A first planning module 2 is configured to perform initial path planning in the two-dimensional grid map by using a preset ant colony star optimization algorithm, so that a visually impaired person moves according to the initial path;

[0176] A prompting module 3 is configured to detect obstacles in the movement of the visually impaired person in real time by using a visual sensor, and issue a voice prompt to the visually impaired person according to the obstacles;

[0177] A second planning module 4 is configured to construct an obstacle environment map by using a Kinect sensor and a laser radar, and perform obstacle avoidance path planning according to the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path until returning to the initial path;

[0178] A monitoring module 5 is configured to perform real-time environment obstacle monitoring of the remaining path by using a laser radar and a visual sensor, and update the remaining initial path according to the environment obstacles.

[0179] In one embodiment, the first planning module 2 comprises:

[0180] A determination unit is configured to determine an end point of the movement of the visually impaired person in the two-dimensional grid map, and initialize an ant system in the ant colony algorithm, set the number of ants to n, and the maximum number of iterations to N max ;

[0181] An initialization unit is configured to initialize k = 1; wherein k is an ant serial number, and k = 1, 2, …, n;

[0182] An update unit is configured to place ant k on a current node, and update the nodes that have been traversed in a taboo table;

[0183] A processing unit is configured to, when ant k is in a deadlock state, judge according to the taboo table, process by using a preset deadlock processing mechanism, and return to the step of placing ant k on the current node and updating the nodes that have been traversed in the taboo table;

[0184] A calculation unit is configured to calculate the state transition probability of ant k, and select the next feasible node by using a roulette method, and when the next feasible node does not reach the end point, return to the step of placing ant k on the current node and updating the nodes that have been traversed in the taboo table;

[0185] a judging unit configured to, when the next feasible node reaches the end point, judge that k is less than n, calculate k=k+1, and return to the step of placing the ant k on the current node and updating the nodes that have been traversed in the taboo list;

[0186] a pheromone updating unit configured to, when k is equal to n, update the pheromone by using a preset pheromone updating method, and judge that the current iteration number N is less than the maximum iteration number N max , calculate N=N+1, and return to the step of initializing k=1;

[0187] an output unit configured to, when the current iteration number N is equal to the maximum iteration number N max , output the optimal path as an initial path, so that the visually impaired person moves according to the initial path.

[0188] In one embodiment, the preset deadlock handling mechanism in the processing unit includes:

[0189] when a node has only one feasible adjacent node, directly marking the node as an obstacle node and withdrawing it to avoid the ant falling into a U-shaped trap; and when the ant has no feasible node in the ant pathfinding process, directly killing the ant;

[0190] In the calculation unit, the calculation formula is:

[0191]

[0192] wherein, is a state transition probability of the ant k from the grid point i to the grid point j; α and β are respectively influence factors of the pheromone and the heuristic information on the state transition probability; allow k represents a set of selectable paths of the node in the next stage; τ ij is a pheromone concentration value of the path i to j; η ij is a heuristic information of the path i to j, and the calculation formula is:

[0193]

[0194] C(n)=phi1×bend+phi2×angle

[0195]

[0196] wherein, Q is a constant greater than 1; C(n) is a bend suppression factor; bend is the number of path bends from the previous node to the next node; angle is the included angle of the corner; phi1 and phi2 are respectively conversion coefficients of the number of turns and the degree of turning; g(n) is the cost of the node n from the starting point; S (i)denotes the moving distance from the parent node of node i to i-1; h(n) is the estimated cost of node n to the end point; (n x ,n y ) is the coordinate of the current node; (f x ,f y ) is the coordinate of the target node.

[0197] In an embodiment, in the pheromone updating unit, the preset pheromone updating method comprises:

[0198] The initial pheromone concentration is set as:

[0199]

[0200] Wherein, τ ij (0) is the initial pheromone concentration, c is a constant, and d(j, l) is the distance between the adjacent grid j and the connecting line l between the target point and the starting point. The closer the grid point to the connecting line l, the higher the initial pheromone concentration, and the more the ant colony tends to select at the beginning stage;

[0201] The pheromone updating rule is set as:

[0202]

[0203] Wherein, Q1 and Q2 are constants greater than 1; L best denotes the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the turning times on the optimal path; x and y are weight proportion coefficients of the turning angle and the turning time.

[0204] In an embodiment, the prompting module 3 comprises:

[0205] The detection unit is configured to detect obstacles in the moving process of the visually impaired person in real time by using a visual sensor, and calculate a straight-line distance from the obstacles when the obstacles are detected; wherein the straight-line distance is the shortest distance from the outline of the obstacles;

[0206] The returning unit is configured to, when the straight-line distance is greater than a threshold value, not issue a voice prompt, and the visually impaired person continues to move along the initial path, and the visual sensor continues to detect obstacles in the moving process of the visually impaired person in real time;

[0207] The stopping unit is configured to, when the straight-line distance is less than or equal to the threshold value, issue a voice prompt to the visually impaired person to stop, so that the visually impaired person stops moving and waits for a new prompt;

[0208] The position determination unit is configured to determine the moving attribute of the obstacles according to the time when the obstacles are detected and the position of the obstacles, so as to determine the final position of the obstacles according to the moving attribute of the obstacles.

[0209] In one embodiment, the position determining unit comprises:

[0210] The shooting sub-unit is configured to cause the visual sensor to shoot an initial environment image with the obstacle when the obstacle is detected, and to shoot a target environment image with the obstacle when the straight-line distance is less than or equal to a threshold value;

[0211] The comparison sub-unit is configured to compare the positions of the center points of the obstacle in the initial environment image and the target environment image, and to determine whether the difference between the positions of the center points exceeds a set value;

[0212] The fixed position determining sub-unit is configured to determine that the moving attribute of the obstacle is a fixed obstacle when the difference between the positions of the center points does not exceed the set value, and to determine the position of the obstacle as the final position of the obstacle;

[0213] The time period calculating sub-unit is configured to determine that the moving attribute of the obstacle is a moving obstacle when the difference between the positions of the center points exceeds the set value, and to calculate the time required for the visually impaired person to move to the obstacle as a time period according to the straight-line distance;

[0214] The average speed calculating sub-unit is configured to calculate the average speed in the appearance time of the obstacle according to the time period, and to predict the final position of the obstacle according to the average speed; specifically comprising:

[0215] The appearance time of the obstacle is the time period from the initial environment image to the target environment image, the average speed in a unit time is calculated according to the time period in the unit time in the appearance time, and the average speed in the appearance time is obtained by calculating the average speeds in multiple unit times;

[0216] The crossing time of the visually impaired person crossing the obstacle is estimated according to the moving speed of the visually impaired person, and the moving distance of the obstacle is obtained by multiplying the average speed by the sum of the time period and the crossing time, and the final position of the obstacle is finally obtained.

[0217] In one embodiment, the second planning module 4, the local map is constructed by using the laser radar, comprising:

[0218] The establishing unit is configured to collect the depth information of the environment in which the obstacle is located by using the Kinect sensor, and to establish a global map based on the environment in which the obstacle is located; wherein the contour of the obstacle is the union of the contour of the obstacle in the target environment image and the contour of the obstacle in the final position;

[0219] A transformation unit is configured to take the laser radar center emitting point as the coordinate origin, use the laser radar to scan the data points of the environment and the global map simultaneously, convert the data point coordinates into global coordinates in the Cartesian system by using the Cartesian formula, and connect the data points into a line segment structure that can be segmented;

[0220] A gathering unit is configured to gather the adjacent coordinates together by using a linear function after the Cartesian formula conversion, and connect them together into a vector line that describes a linear region of the environment, and multiple vector lines segment the environment into multiple linear regions;

[0221] A coordinate calculation unit is configured to use a dynamic threshold method to construct a local map of the environment in units of linear regions, and the longitudinal coordinate calculation point expression is:

[0222]

[0223] wherein, ε i represents a dynamic threshold constant; y0 represents a local map construction parameter; y i+1 represents a local map construction error; the global map established by the Kinect sensor and the local map established by the laser radar are overlapped together by using a covariance function, so that the virtual map established according to the environment is refined, and the horizontal coordinate of the local map is:

[0224]

[0225] wherein, represents a covariance constant; S represents an overlapping coefficient of the global map and the local map; α r represents the weight of the virtual map.

[0226] In one embodiment, in the second planning module 4, the obstacle avoidance path planning according to the obstacle environment map includes:

[0227] A step calculation unit is configured to calculate a map complexity coefficient C according to the obstacle environment map, so as to calculate a step S according to the map complexity coefficient.

[0228]

[0229] wherein, A obstacle represents the obstacle area; A map represents the map area; D obstacle represents the obstacle distribution;

[0230] An extension unit is configured to take the point on the obstacle closest to the initial path as a target point, and take the point on the initial path at a set distance in the opposite direction of the obstacle as an end point N 终 ;

[0231] a new node unit configured to set a random tree from a starting point N 起 Start growing, let the starting point N 起 Randomly generate a set of sample points in the map for the root node of the random tree N 采 ={N1, N2, N3}, select the sample point closest to the end point as the final sample point N 采 , traverse the nodes in the random tree to find the node N 采 closest to the sample point N 近 ; add the node N 近 to the random tree, and define the parent node N 采 of the node N 新 in the direction of the sample point N 新 , and grow a step S without colliding with obstacles to obtain a new node N 新 , add the node N 近 to the random tree, and define the parent node N 新 ;

[0232] a region defining unit configured to divide the search space into two regions according to the node N 新2 , define the region where the end point is located as the effective region, and define the other region as the invalid region, and when sampling again at the next random sampling point, sample in the effective region to obtain a new node N ;

[0233] a node updating unit configured to, when the random tree cannot continue to grow after a set number of iterations of the current new node, determine that the new node is invalid, remove it from the random tree, and set the effective sampling region of the random tree to the effective region obtained based on the last new node, and the random tree starts growing again in the updated effective sampling region until the distance between the generated new node and the end point is less than the step S, and the line connecting the two points does not collide with the obstacle, and the end point is reached.

[0234] a connection unit configured to add the end point to the random tree, trace all nodes with parent nodes in the random tree in reverse, form a node set, and perform a pre-set selection connection according to the node order to obtain an obstacle avoidance path.

[0235] In one embodiment, the monitoring module 5 updates the remaining initial path according to the environmental obstacles, including:

[0236] a contour acquisition unit configured to monitor the position of the environmental obstacles in real time, and acquire the contour of the environmental obstacles when the environmental obstacles cover the remaining path;

[0237] The path segment division unit is configured to divide the remaining path into an initial path segment, a covering path segment and an ending path segment according to the environmental obstacle profile, and obtain a starting intersection point and an ending intersection point of the environmental obstacle profile and the remaining path.

[0238] The first target point selection unit is configured to select the starting intersection point and the ending intersection point as a starting target point and an ending target point when the starting intersection point and the ending intersection point are the points on the environmental obstacle profile closest to the initial path segment and the ending path segment respectively.

[0239] The second target point selection unit is configured to obtain a reference point on the environmental obstacle profile closest to the initial path segment and the ending path segment when the starting intersection point and the ending intersection point are not the points on the environmental obstacle profile closest to the initial path segment and the ending path segment respectively, draw a straight line through the reference point and perpendicular to the initial path segment and the ending path segment, select an intersection point of the straight line and the initial path segment and the ending path segment as the starting target point and the ending target point, and discard the starting intersection point and / or the ending intersection point closest to the target point.

[0240] The circle drawing unit is configured to draw a circle with a distance between the starting target point and the ending target point as a diameter and a line segment center between the starting target point and the ending target point as a center of the circle.

[0241] The first updating unit is configured to draw a chord closest to the environmental obstacle when the circle does not have an intersection point with the environmental obstacle, and select a path of a shortest half circle and chord on both sides of the remaining path as an updated path of the covering path segment.

[0242] The second updating unit is configured to draw a half circle with a distance between a point on the environmental obstacle farthest from the covering path segment and a center of the circle as a radius when the circle has an intersection point with the environmental obstacle, and select a profile of the half circle as an updated path of the covering path segment.

[0243] The above modules, units and sub-units are configured to correspondingly execute the steps in the above-mentioned multi-sensor-based visual obstacle auxiliary navigation method, and the specific implementation manners are described above in the method embodiment, which will not be described here again.

[0244] As shown in Figure 3 The present application also provides a computer device, which can be a server, and the internal structure thereof can be as shown in Figure 3The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store all data required by the process of the multi-sensor based visual impairment auxiliary navigation method. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the multi-sensor based visual impairment auxiliary navigation method.

[0245] Those skilled in the art can understand that, Figure 3 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied.

[0246] The computer program is executed by the processor to implement the multi-sensor based visual impairment auxiliary navigation method.

[0247] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium provided by the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM) and the like.

[0248] It is to be understood that the terminology "including", "comprising", or any other variation thereof, is intended to cover a non-exclusive inclusion such that process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0249] The above description is merely the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-sensor-based assisted navigation method for the visually impaired, characterized in that, include: LiDAR and visual sensors are used to acquire geometric distance data and optical image data of the environment, and SLAM algorithm is used to construct a two-dimensional grid map; An initial path is planned using a preset ant colony optimization algorithm in the two-dimensional grid map so that the visually impaired person can move along the initial path. A visual sensor is used to detect obstacles in the movement of the visually impaired person in real time, and voice prompts are given to the visually impaired person based on the obstacles. An obstacle environment map is constructed using Kinect sensors and LiDAR, and obstacle avoidance path planning is performed based on the obstacle environment map so that the visually impaired person moves along the obstacle avoidance path until returning to the initial path; wherein, constructing a local map using the LiDAR includes: The Kinect sensor is used to collect depth information of the current environment containing obstacles, and a global map based on the environment is built; wherein, the outline of the obstacle is the union of the outline of the obstacle in the target environment image and the outline of the obstacle at the final location; Using the central emission point of the lidar as the origin of the coordinate system, the lidar simultaneously scans data points in the environment and the global map. The coordinates of the data points are then transformed into global coordinates in the Cartesian system using the Cartesian formula, so that the data points can be connected into a divisible line segment structure. After the data point coordinates are transformed by the Cartesian formula, the coordinates of neighboring coordinates are grouped together by a linear function and connected together to form a vector line describing the linear region of the environment. Multiple vector lines divide the environment into multiple linear regions. Using linear regions as units, a local map of the environment is constructed using a dynamic thresholding method. The expression for calculating the vertical axis coordinates of the points is as follows: ； Where, ε i y represents the dynamic threshold constant; y0 represents the local map construction parameters; y i+1 This represents the local map construction error; the covariance function is used to overlay the global map built by the Kinect sensor and the local map built by the LiDAR, thus refining the virtual map built according to the environment. The horizontal axis of the local map is: ； in, Represents the covariance constant; S represents the overlap coefficient between the global map and the local map; α r Indicates the weight of the virtual map; The remaining path is monitored in real time using lidar and visual sensors, and the remaining initial path is updated based on the environmental obstacles.

2. The multi-sensor-based visually impaired assisted navigation method according to claim 1, characterized in that, The step of using a preset ant colony optimization algorithm to perform initial path planning in the two-dimensional grid map so that the visually impaired person can move along the initial path includes: In the two-dimensional grid map, the destination of the visually impaired person's movement is determined, and the ant system in the ant colony algorithm is initialized, with the number of ants set to n and the maximum number of iterations to N. max ; Initialize k=1; where k is the ant number, and k=1,2,…,n; Place ant k on the current node and update the visited nodes in the tabu list; If ant k is found to be deadlocked according to the taboo list, the preset deadlock handling mechanism is used to handle it, and the steps of placing ant k on the current node and updating the traversed nodes in the taboo list are returned. Calculate the state transition probability of ant k, and use the roulette wheel method to select the next feasible node. If the next feasible node has not reached the destination, return to the step of placing ant k on the current node and updating the nodes that have been traversed in the tabu list. When the next feasible node reaches the destination, if k is less than n, calculate k = k + 1, and return to the step of placing ant k on the current node and updating the traversed nodes in the tabu list; When k equals n, the pheromone is updated using a preset pheromone update method, and it is determined that the current iteration number N is less than the maximum iteration number N. max When N=N+1 is reached, return to the step of initializing k=1; Determine if the current iteration number N is equal to the maximum iteration number N max When the optimal path is output as the initial path, the visually impaired person moves along the initial path.

3. The multi-sensor-based visually impaired assisted navigation method according to claim 2, characterized in that, The step of determining, based on the tabu list, that when ant k is trapped in a deadlock state, to handle the situation using a preset deadlock handling mechanism includes the following: When a node has only one feasible neighbor, mark it as an obstacle and withdraw it to prevent the ant from falling into a U-shaped trap; and during the ant's pathfinding process, if the ant has no feasible nodes, let it die directly. In the steps of calculating the state transition probability of ant k, the calculation formula is as follows: ； in, Let be the state transition probability of ant k crawling from grid point i to grid point j; α and β are the influence factors of pheromone and heuristic information on the state transition probability, respectively. k τ represents the set of possible paths for the next stage of a node; ij η represents the pheromone concentration value along path i to j; ij The heuristic information for path i to j is calculated using the following formula: ; ； ； ； Where Q is a constant greater than 1; C(n) is the bending suppression factor; bend is the number of bends in the path from the previous node to the next node; angle is the angle between corners; phi1 and phi2 are the conversion coefficients for the number of turns and the degree of turn, respectively; g(n) is the cost of node n from the starting point; S (i) h(n) represents the distance moved from the parent node of node i to node i-1; h(n) is the estimated cost of node n from the destination; (n x ,n y ) represents the coordinates of the current node; (f x ,f y ) represents the coordinates of the target node.

4. The multi-sensor-based visually impaired assisted navigation method according to claim 2, characterized in that, In the step of updating pheromones using a preset pheromone update method when k equals n, the preset pheromone update method includes: Set the initial pheromone concentration as follows: ； Where, τ ij (0) represents the initial pheromone concentration, c is a constant, and d(j,l) is the distance between adjacent grid j and the line connecting the target point and the starting point l. The closer the grid point is to the line l, the higher the initial pheromone concentration, and the more the ant colony tends to choose it in the initial stage. Set the pheromone update rules as follows: ； ； Where Q1 and Q2 are constants greater than 1; L best This represents the current optimal path; Cal(l) is the sum of the cumulative turning angles on the optimal path; Turn(l) is the sum of the number of turns on the optimal path; x and y are the weighting coefficients of the turning angle and the number of turns.

5. The multi-sensor-based visually impaired assisted navigation method according to claim 1, characterized in that, The step of using a visual sensor to detect obstacles in real time during the movement of the visually impaired person, and issuing voice reminders to the visually impaired person based on the obstacles, includes: A visual sensor is used to detect obstacles in real time during the movement of the visually impaired person. When an obstacle is detected, the straight-line distance to the obstacle is calculated; wherein, the straight-line distance is the shortest distance to the outline of the obstacle. When the straight-line distance is greater than the threshold, no voice prompt is issued, the visually impaired person continues to move along the initial path, and returns to continue using the visual sensor to detect obstacles in the process of the visually impaired person's movement in real time; When the straight-line distance is less than or equal to the threshold, a voice prompt to stop is issued to the visually impaired person so that the visually impaired person stops moving and waits for a new prompt; The movement attributes of the obstacle are determined based on the time of obstacle detection and the location of the obstacle, so as to determine the final location of the obstacle based on the movement attributes of the obstacle.

6. The multi-sensor-based visually impaired assisted navigation method according to claim 5, characterized in that, The step of determining the movement attributes of the obstacle based on the time of obstacle detection and the location of the obstacle, and determining the final location of the obstacle based on the movement attributes of the obstacle, includes: When an obstacle is detected, the vision sensor captures an initial environmental image with the obstacle, and when the straight-line distance is less than or equal to a threshold, the vision sensor captures a target environmental image with the obstacle. Compare the center point positions of the obstacles in the initial environment image and the target environment image, and determine whether the difference between the center point positions exceeds a set value; If the difference between the positions of the center points does not exceed the set value, the movement attribute of the obstacle is determined to be a fixed obstacle, and its position is taken as the final position of the obstacle. If the difference between the positions of the center points exceeds a set value, the movement attribute of the obstacle is determined to be a moving obstacle, and the time required for the visually impaired person to move to the obstacle is calculated based on the straight-line distance as the time period; Calculate the average velocity of the obstacle within the time period it appears based on the time period, and predict the final position of the obstacle based on the average velocity; specifically including: The time when the obstacle appears is the time period from the initial environment image to the target environment image. The average speed per unit time is calculated within the time period, and then the average speed per unit time is calculated to obtain the average speed within the time period. The time required to cross the obstacle is estimated based on the visually impaired person's movement speed. The average speed is then multiplied by the sum of the time period and the crossing time to obtain the distance the obstacle travels, and finally, the final position of the obstacle is determined.

7. The multi-sensor-based visually impaired assisted navigation method according to claim 1, characterized in that, The step of constructing an obstacle environment map using Kinect sensors and LiDAR, and planning an obstacle avoidance path based on the obstacle environment map so that the visually impaired person can move along the obstacle avoidance path until returning to the initial path, includes the following steps: The map complexity coefficient C is calculated based on the obstacle environment map, and the step size S is calculated based on the map complexity coefficient. ； ； Among them, A obstacle Indicates the area of ​​the obstacle; A map D represents the area of ​​the map; obstacle Indicates the distribution of obstacles; The point on the obstacle closest to the initial path is taken as the target point, and the point N extending a predetermined distance from the target point along the obstacle in the opposite direction from the initial path is taken as the endpoint. 终 ; Based on the RRT algorithm, the random tree starts from point N. 起 Start growing, let the starting point N 起 As the root node of the random tree, a set N of sampling points is randomly generated in the map. 采 Given a set {N1, N2, N3}, select the sampling point closest to the endpoint as the final sampling point N. 采 Traverse the nodes in the random tree to find the distance N from the sampling point. 采 The nearest node N 近 ; Node N 近 Along sampling point N 采 When the directional growth step size is S and there is no collision with obstacles, a new node N is obtained. 新 , will node N 新 Add to a random tree and define node N. 新 Parent node N 近 ; According to the node N 新 The search space is divided into two regions: the region where the endpoint is located is defined as the valid region, and the other region is defined as the invalid region, with node N as the starting point. 新 Starting from this point, during the next random sampling, sampling is performed within the valid region to obtain a new node N. 新2 ; When the random tree fails to continue growing after a set number of iterations of the current new node, the new node is deemed invalid and removed from the random tree. The effective sampling area of ​​the random tree is set to the effective area obtained by segmenting the previous new node. The random tree starts growing again in the updated effective sampling area until the distance between the generated new node and the endpoint is less than the step size S, and the line connecting the two points does not collide with the obstacle. At this point, the endpoint is determined to have been reached. The endpoint is added to the random tree, and a set of nodes is formed by tracing back all nodes with parent nodes in the random tree. The nodes are then selected and connected according to the preset order to obtain the obstacle avoidance path.

8. The multi-sensor-based visually impaired assisted navigation method according to claim 1, characterized in that, The step of using lidar and visual sensors to monitor environmental obstacles on the remaining path in real time, and updating the remaining initial path based on the environmental obstacles, includes updating the remaining initial path based on the environmental obstacles, which includes: The location of the environmental obstacles is monitored in real time, and the outline of the environmental obstacles is obtained when the environmental obstacles cover the remaining path; The environmental obstacle contour divides the remaining path into an initial segment, a covered segment, and an ending segment, and obtains the starting and ending intersection points of the environmental obstacle contour and the remaining path. When the starting intersection point and the ending intersection point are the points on the outline of the environmental obstacle that are closest to the starting road segment and the ending road segment, respectively, the starting intersection point and the ending intersection point are selected as the starting target point and the ending target point; When the starting intersection point and the ending intersection point are not the points on the environmental obstacle outline that are closest to the starting road segment and the ending road segment, the points on the environmental obstacle outline that are closest to the starting road segment and the ending road segment are obtained as reference points. A straight line perpendicular to the starting road segment and the ending road segment is drawn through the reference point, and the intersection point of the straight line with the starting road segment and the ending road segment is taken as the starting target point and the ending target point. The starting intersection point and / or the ending intersection point that are closest to the target point are discarded. Draw a circle with the distance between the starting target point and the ending target point as the diameter and the center of the line segment between the starting target point and the ending target point as the center; When the circle does not intersect with the environmental obstacle, draw the chord closest to the environmental obstacle, and take the path of the shortest semicircle + chord on both sides of the remaining path as the updated path of the covered road segment. When the circle intersects with the environmental obstacle, a semicircle is drawn with the distance between the point on the environmental obstacle farthest from the covered road segment and the center of the circle as the radius, and the outline of the semicircle is used as the updated path of the covered road segment.

9. A multi-sensor-based visually impaired assistive navigation device, characterized in that, The device comprising the multi-sensor-based visual impairment-assisted navigation method according to any one of claims 1 to 8, wherein the device includes: The module is used to acquire geometric distance data and optical image data of the environment using LiDAR and visual sensors, and to build a two-dimensional grid map using the SLAM algorithm; The first planning module is used to perform initial path planning in the two-dimensional grid map using a preset ant colony optimization algorithm, so that the visually impaired person can move according to the initial path. The prompting module is used to detect obstacles in the movement of the visually impaired person in real time using a visual sensor, and to issue voice prompts to the visually impaired person based on the obstacles. The second planning module is used to construct an obstacle environment map using Kinect sensors and LiDAR, and to plan an obstacle avoidance path based on the obstacle environment map, so that the visually impaired person moves according to the obstacle avoidance path until returning to the initial path; The monitoring module is used to monitor environmental obstacles on the remaining path in real time using lidar and visual sensors, and update the remaining initial path based on the environmental obstacles.

Images