A multi-modal blind-assisted indoor navigation method and system

Abstract

The application discloses a multi-modal blind-assisting indoor navigation method and system, relates to data processing and blind-assisting indoor navigation technology, and comprises the following steps: receiving image video input and audio stream input; perceiving the environment through a vision model for the image video input, and converting the audio stream into text to construct a state environment description; analyzing the state environment description by using an LLM to generate a semantic map and perform target analysis on a user instruction; and matching the generated semantic map with the analyzed user instruction for navigation, so as to realize blind-assisting indoor navigation. The application provides an indoor navigation method combining a large language model and a vision model, realizes semantic-level environment perception under the condition of no preset map, conversational path decision and multi-modal collaborative control, and promotes the intelligent upgrading of barrier-free technology.

Description

Technical Field

[0003]

[0001] This application relates to the technical fields of data processing and blind indoor navigation, and particularly to a multimodal blind indoor navigation method and system. Background Art

[0002] At present, the travel navigation of visually impaired people in China mostly relies on guide dogs or human assistance, that is, accompanied by volunteers or family members. In particular, blind navigation has not been widely applied, and it is mostly used for outdoor navigation, that is, relying on wearable devices and the Global Positioning System (GPS) for joint positioning. In the context of indoor navigation, it can only rely on the existing general indoor navigation technology. The existing indoor navigation mainly relies on the following several technologies: wireless signal-based navigation, Bluetooth beacon positioning, and reinforcement learning and mapless navigation.

[0003] Wireless signal-based navigation uses Ultra-Wideband (UWB) or Beidou signals, etc., and combines multi-source fusion algorithms to achieve high-precision positioning; Bluetooth beacon positioning relies on Bluetooth beacons set inside buildings to periodically transmit broadcast signals, and the device measures the distance based on the received signal strength to locate the user; reinforcement learning and mapless navigation achieve mapless navigation through real-time decision-making of the reinforcement learning model.

[0004] The existing human-dependent solutions have the following defects respectively: (1) The training cost of guide dogs is relatively high, and the service coverage rate is seriously insufficient. According to statistics, there are about 17.31 million visually impaired people in the country at present, and the number of active guide dogs is only about 400.

[0005] (2) There are problems such as difficult time coordination and privacy protection when accompanied by volunteers or family members during travel, and it is difficult to meet the high-frequency independent travel needs.

[0006] In the indoor navigation scenario, GPS signals often fail due to building blockage. The existing general indoor navigation technology has the following defects when applied to blind navigation: (1) Lack of fine obstacle recognition ability. The general indoor navigation system only provides rough position information, and cannot recognize small obstacles such as steps, low furniture, etc., nor does it have reminders for dynamic obstacles such as pedestrians. Due to the difference in information acquisition ability between visually impaired people and the able-bodied, more refined obstacle classification is required.

[0007] (2) Lack of semantic environment description ability, and it is difficult to meet the cognitive needs of visually impaired people. General indoor navigation usually only provides direction guidance, relying on geometric maps and ignoring scene semantic information, resulting in users being unable to construct a mental and cognitive map, which affects travel. Summary of the Invention

[0008] The embodiments of this application provide a multi-modal indoor navigation method and system for the blind. For visually impaired people, it realizes scene information extraction and user natural language understanding based on multi-modal algorithms, and solves the problems of lack of fine obstacle recognition ability and semantic description ability in indoor navigation in the prior art.

[0009] The embodiments of this application provide a multi-modal indoor navigation method for the blind, including: Receiving image / video input and audio stream input; For the image / video input, perceiving the surrounding environment through a visual model, and converting the audio stream into text to construct a state environment description; Using the LLM to parse the state environment description, generating a semantic map, and performing target parsing on the user instructions; Matching and navigating the generated semantic map with the parsed user instructions to achieve indoor navigation for the blind.

[0010] Optionally, it further includes performing sound source localization on the received audio stream input, converting the audio stream determined to be a valid sound source into text, and discarding the invalid audio stream.

[0011] Optionally, the matching and navigation of the generated semantic map with the parsed user instructions includes: Traversing all nodes Vi∈V in the semantic map G, matching and scoring each node with the target location. The higher the matching degree between the node and the target location, the higher the obtained score; Selecting the k nodes with the highest scores as the target candidate node set V_goal.

[0012] Optionally, the matching and navigation of the generated semantic map with the parsed user instructions includes: Randomly selecting a node from the candidate target node set V_goal with probability P_goal; Completely randomly sampling a node within the entire space with probability P_random; Randomly selecting a highly connected non-target type node from the semantic map with probability P_semantic, where P_goal + P_random + P_semantic = 1; Taking the position of the selected node as the sampling point x_sample, where the probability of the node randomly selected from the candidate target node set V_goal is the sum of the probabilities of the nodes selected by the other two methods.

[0013] Optionally, the matching and navigation of the generated semantic map with the parsed user instructions further includes: Define the random tree T as a graph structure that continuously grows in the free space, with its root node being the navigation starting point S. By continuously expanding candidate nodes x_new from the existing node x_near in the direction of the sampling point x_sample and after verifying through collision detection, add the safe x_new nodes and their collision-free connecting edges to the graph; Determine the node x_near in the random tree T that is closest to the x_sample, where the random tree T is: Grow a step length from the node x_near in the direction of the node x_sample to obtain a new node x_new; Check whether the connection line from the node x_near to the node x_new intersects the geometric boundary of the obstacles marked in the semantic map. If it intersects, it is determined that a collision has occurred, and the node x_new is discarded.

[0014] Optionally, matching and navigating the generated semantic map with the parsed user instructions further includes: When a certain node in the random tree T enters the threshold range of any candidate target node, it is considered that the path search is successful; Backtrack from the successfully searched target node to the root node to obtain the original node sequence; For the original node sequence, perform pruning optimization according to the semantic reachability between nodes to obtain the navigation path.

[0015] Optionally, matching and navigating the generated semantic map with the parsed user instructions further includes: Perform a safety assessment on the generated navigation path. If there are risks in the path, perform emergency replanning; Output the passed navigation path in the form of voice broadcast.

[0016] An embodiment of this application also proposes a multimodal indoor navigation system for the blind, including a processor and a memory. A computer program is stored on the memory, and when the computer program is executed by the processor, it implements the steps of the above multimodal indoor navigation method for the blind.

[0017] This application provides an indoor navigation method that integrates a large language model and a visual model, realizing semantic-level environment perception, conversational path decision-making, and multimodal collaborative control without a pre-set map, promoting the intelligent upgrade of barrier-free technology.

[0018] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of this application more obvious and understandable, the following specifically presents the specific embodiments of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 This is a schematic diagram of the architecture of the multi-modal blind assistance indoor navigation system according to an embodiment of the present application. Specific embodiments

[0020] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0021] An embodiment of the present application provides a multi-modal blind assistance indoor navigation method, including the following steps: In step S101, receive image / video input and audio stream input.

[0022] In step S102, for the image / video input, perceive the surrounding environment through a vision model, and convert the audio stream into text to construct a state environment description. In a specific example, this step can be implemented by an environment perception module. The environment perception module receives the image / video input and the audio stream input in parallel. Perceive the real-time environment through the vision model, and monitor static obstacles, dynamic obstacles (such as pedestrians), markers (such as safety exits), and risk areas (such as slippery floors) in real time, and perform semantic segmentation on them.

[0023] In some embodiments, it further includes performing sound source localization on the received audio stream input, converting the audio stream determined to be a valid sound source into text, and discarding the invalid audio stream.

[0024] Convert the surrounding environment perceived by the vision model and the text converted from the audio stream into semantic information that can be understood by a large language model (LLM) as the state environment description.

[0025] In step S103, use the LLM to parse the state environment description, generate a semantic map, and perform target parsing on the user instruction.

[0026] In step S104, match and navigate the generated semantic map with the parsed user instruction to achieve blind assistance indoor navigation.

[0027] Through the collaborative work of the large language model and the visual model, this application constructs a complete set of automated blind navigation solutions, which can autonomously complete the entire process of environmental perception, semantic understanding, and dynamic planning, reducing the dependence on manual assistance in traditional blind services and enabling visually impaired people to achieve more independent and convenient independent travel.

[0028] This application is used to quickly generate an initial path with semantic rationality according to the target parsed from the user's instructions in the case of having only a real-time constructed semantic map without a pre-prepared geometric map. The embodiments of this application select and design a heuristic random tree algorithm for semantic goals. In some embodiments, matching and navigating the generated semantic map with the parsed user instructions includes: Traverse all nodes Vi∈V in the semantic map G, match and score each node with the target location. The higher the matching degree between the node and the target location, the higher the score obtained; Select the k nodes with the highest scores as the target candidate node set V_goal.

[0029] In some embodiments, matching and navigating the generated semantic map with the parsed user instructions includes: Randomly select a node from the candidate target node set V_goal with probability P_goal; Completely randomly sample a node within the entire space with probability P_random. Completely randomly sampling within the entire space can effectively solve the problem of getting stuck.

[0030] Randomly select a highly connected node of a non-target type from the semantic map with probability P_semantic. Highly connected nodes such as main corridor nodes are sampled to explore key channels. Among them, P_goal + P_random + P_semantic = 1.

[0031] Take the position of the selected node as the sampling point x_sample, where the probability of the node randomly selected from the candidate target node set V_goal and the sum of the probabilities of the nodes selected by the other two methods are 1.

[0032] Define the random tree T as a graph structure that continuously grows in the free space. Its root node is the navigation starting point S. By continuously expanding the candidate node x_new from the existing node x_near in the direction of the sampling point x_sample and after verifying through collision detection, add the safe x_new node and its collision-free connecting edge to the graph. Finally, form a safe path network covering the reachable space. This algorithm improves the traditional Rapidly-Exploring Random Tree (RRT), and it is guided by semantic information for sampling.

[0033] The innovative use of the guiding sampling strategy in the embodiments of this application is that the random tree T takes the starting point S as the root node, and by repeatedly executing the steps of "guided sampling - selecting the nearest node - expanding candidate nodes - collision detection", it gradually expands collision-free nodes and edges in the free space until its branch successfully reaches the navigation target. This sampling strategy enables the growth of the random tree T to have both clear goal orientation and exploration, and can quickly advance along the semantic key channels, avoiding wasting time in meaningless open areas.

[0034] In some embodiments, matching the generated semantic map with the parsed user instructions for navigation further includes: Determining the node x_near in the random tree T that is closest to x_sample; Growing a step length from the node x_near in the direction of the node x_sample to obtain a new node x_new; Collision detection: Checking whether the connection line from the node x_near to the node x_new intersects the geometric boundaries of the obstacles marked in the semantic map, such as walls, tables, etc. If it intersects, it is determined that a collision has occurred, and the node x_new is discarded. This is the only geometric operation in the method of this application, which depends on the object bounding boxes provided by real-time visual perception. If a collision occurs, the current x_new is discarded.

[0035] In some embodiments, matching the generated semantic map with the parsed user instructions for navigation further includes: When a certain node in the random tree T enters the threshold range of any candidate target node, it is considered that the path search is successful; Backtracking from the successfully searched target node to the root node to obtain the original node sequence; For the original node sequence, pruning and optimization are performed according to the semantic reachability between nodes to obtain the navigation path. For example, if the path is A→B→C→D, but A can directly see D and is semantically reachable, it is simplified to A→D to make the path more natural.

[0036] In some embodiments, matching the generated semantic map with the parsed user instructions for navigation further includes: Performing a safety assessment on the generated navigation path. If the path has risks, emergency replanning is performed; For the navigation path that passes the assessment, it is output in the form of voice broadcast.

[0037] In a specific example, a safety assessment is performed on the generated navigation instruction. If the path has risks, emergency replanning is performed, and the LLM is applied to re-verify its safety until it passes. For the navigation instructions that pass, the optimal path is obtained through the algorithm and output in the form of voice broadcast.

[0038] Different from the fixed-instruction mode of traditional navigation, this application proposes a conversational path planning method based on large language models. It supports fuzzy target parsing. For example, for the instruction "I want to take a break", the LLM will automatically recommend the best destination and generate a path in combination with the area function annotations in the semantic map.

[0039] This application integrates the large language model with the visual model, which can not only detect scenes but also understand complex semantic descriptions. The visual model converts the original image data into semantic descriptions, which are then handed over to the large language model for scene reasoning and decision-making. For example, when detecting that the road surface is slippery, instead of only prompting "Pay attention to obstacles", it will generate a more humanistic suggestion after LLM analysis: "The ground is slippery. It is recommended to walk slowly while holding the wall. There is a dry path on the right." The embodiment of this application also proposes a multimodal indoor navigation system for the blind, which includes a processor and a memory. A computer program is stored on the memory, and when the computer program is executed by the processor, it implements the steps of the multimodal indoor navigation method for the blind as described above. The architecture of the indoor navigation system for the blind in this application is as Figure 1 described. The system of this application includes an environmental perception module, a semantic understanding module, and a dynamic decision-making module, where: The environmental perception module receives image / video input and audio stream input in parallel. It perceives the real-time environment through the visual model and monitors static obstacles, dynamic obstacles (such as pedestrians), markers (such as safety exits), and risk areas in real time.

[0040] The semantic understanding module is used to parse the user's natural language instructions and convert them into executable navigation targets, and at the same time understand the environmental semantics to support dynamic decision-making. Through the LLM model, this module parses the state environment description output by the environmental perception module to generate a semantic map, and at the same time performs target parsing on the user's instructions, and matches the two for navigation.

[0041] The dynamic decision-making module is used to perform a safety assessment on the generated navigation instructions. If there are risks in the path, it will perform emergency replanning and use the LLM to re-verify its safety until it passes. For the passed navigation instructions, the optimal path is obtained through an algorithm and output in the form of voice broadcast.

[0042] Compared with traditional indoor navigation systems, the present application innovatively achieves refined recognition of various obstacles in complex environments. It can not only detect static obstacles such as steps and low furniture, but also identify dynamic obstacles such as pedestrians, and issue warnings to users through a multi-modal prompt mechanism, effectively solving the pain point problem that visually impaired people have insufficient perception of subtle and dynamic obstacles in the environment. The present application provides users with environmental explanations containing elements such as spatial relationships and object attributes through natural language. This way of cognitive assistance can enhance the visually impaired users' ability to understand the space of the environment, help them build an accurate mental map, and thus greatly enhance their confidence and safety in traveling.

[0043] It should be noted that in each embodiment of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0044] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0045] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0046] The above describes the embodiments of the present application in conjunction with the drawings, but the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and these all fall within the protection scope of the present application.

Claims

1. A multimodal indoor navigation method for the blind, characterized in that, include: Receives image / video input and audio stream input; For image and video input, the system uses a visual model to perceive the surrounding environment and converts audio streams into text to construct a state-environment description. LLM is used to parse the state environment description, generate a semantic map, and perform target parsing on user commands; The generated semantic map is matched with the parsed user commands for navigation, thus enabling indoor navigation for the visually impaired. Navigation involves matching the generated semantic map with the parsed user commands, including: Traverse all nodes Vi∈V in the semantic map G, match and score each node with the target location. The higher the matching degree between the node and the target location, the higher the score. Select the k nodes with the highest scores as the candidate target node set V_goal; Navigation involves matching the generated semantic map with the parsed user commands, including: A node is randomly selected from the candidate target node set V_goal with probability P_goal; Nodes are sampled completely randomly throughout the entire space with probability P_random; A non-goal type highly connected node is randomly selected from the semantic map with probability P_semantic, where P_goal+P_random+P_semantic=1; The position of the selected node is taken as the sampling point x_sample, where the probability of a node randomly selected from the candidate target node set V_goal is equal to the sum of the probabilities of nodes selected in the other two ways.

2. The multimodal indoor navigation method for the blind as described in claim 1, characterized in that, It also includes locating the sound source of the received audio stream input, converting the audio stream that is determined to be a valid sound source into text, and discarding invalid audio streams.

3. The multimodal indoor navigation method for the blind as described in claim 1, characterized in that, The navigation process, which involves matching the generated semantic map with the parsed user commands, also includes: Define a random tree T as a graph structure that grows continuously in free space, with its root node being the navigation starting point S. By expanding candidate nodes x_new from the existing node x_near toward the sampling point x_sample, safe x_new nodes and their collision-free connecting edges are added to the graph structure of the random tree T. Determine the node x_near that is closest to x_sample from the random tree T; Grove the node x_near towards the node x_sample by one step to obtain a new node x_new; Check whether the line connecting node x_near to node x_new intersects with the geometric boundary of the obstacle marked in the semantic map. If they intersect, a collision is determined, and node x_new is discarded.

4. The multimodal indoor navigation method for the blind as described in claim 3, characterized in that, The navigation process, which involves matching the generated semantic map with the parsed user commands, also includes: When a node in the random tree T enters the threshold range of any candidate target node, the path search is considered successful. Traverse back from the target node where the search was successful to the root node to obtain the original node sequence; The original node sequence is pruned and optimized based on the semantic reachability between nodes to obtain a navigation path.

5. The multimodal indoor navigation method for the blind as described in claim 4, characterized in that, The navigation process, which involves matching the generated semantic map with the parsed user commands, also includes: The generated navigation path is subjected to a safety assessment. If the path is found to be risky, emergency replanning is performed. For navigation routes that pass the evaluation, the route will be output via voice prompts.

6. A multimodal indoor navigation system for the blind, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the multimodal assistive indoor navigation method for the visually impaired as described in any one of claims 1-5.

Images