Scenarized agent navigation and environment understanding method for vision-limited people

Abstract

The invention discloses a scenarized agent navigation and environment understanding method for vision-limited people, and the method comprises the steps: multi-modal data collection, preprocessing and spatial prior construction, multi-modal data unified coding input, scene agent matching activation, multi-role AI instantiation and parallel execution, and dual-mode output. The method has the advantage of being capable of improving travel independence and life convenience of people with limited eyesight.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of blind guiding and auxiliary travel, and in particular to a scene-based intelligent agent navigation and environment understanding method for visually impaired people. BACKGROUND

[0002] With the development of computer vision and artificial intelligence, intelligent auxiliary technology for visually impaired people has gradually been applied to travel and life scenarios. Existing solutions mostly use visible light cameras to obtain environment images, combine general image recognition models to obtain scene elements, and feed back to users through voice broadcasting for navigation guidance, obstacle reminders, or object recognition.

[0003] Existing disclosed technologies such as the intelligent blind guiding method, device, electronic equipment and storage medium of CN119293719A, the reading map method, system, device and storage medium for assisting visually impaired users of CN120356225A, and the intelligent blind guiding glasses system based on a multi-modal large language model of CN120131404A have improved the travel safety and environmental perception ability of visually impaired people to some extent, but there are still some deficiencies as follows: First, the adaptability to complex lighting and dynamic environment is insufficient. Existing blind guiding solutions generally rely on a direct link between single visible light image recognition and voice broadcasting. In complex lighting and dynamic environments such as weak light, backlight, night streets, or traffic intersections, image quality decreases and model recognition stability is insufficient, resulting in incomplete prompt content and affecting safety. Second, the spatial understanding ability is limited. Existing technologies are mainly based on two-dimensional image content recognition, lack of spatial scale information support such as laser ranging and infrared perception, and cannot provide data assistance for large model understanding and reasoning of spatial environment. Third, there is a lack of scene-based strategy capability. Current systems mostly use general recognition logic and lack differentiated strategies for different life scenarios (such as indoors, communities, streets, etc.) and specific behaviors (crossing the road, reading text, finding objects, etc.), which can easily cause problems such as mismatch between broadcast information and actual needs, redundant or missing prompts. Fourth, the interaction method is insufficient. Existing interaction methods mostly stay at the level of single-round voice response and cannot support continuous dialogue, automatic cruising, immediate follow-up questions, and result confirmation, which are more in line with real use needs. In summary, existing technologies still lack a navigation and environment understanding method that combines multi-modal perception ability, scene adaptive strategy, multi-role intelligent collaboration, and dual-mode interaction ability, making it difficult to continuously and stably meet the comprehensive needs of visually impaired people for safe travel and daily information acquisition in complex real environments, and new technical solutions are urgently needed to improve. SUMMARY

[0005] The purpose of this invention is to provide a scenario-based intelligent agent navigation and environmental understanding method and its application for visually impaired individuals. This invention has the advantage of improving the travel safety and convenience of visually impaired individuals.

[0006] The technical solution of the present invention: A scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals includes the following process: A. Multimodal data acquisition: Acquire multimodal sensing information and synchronize it with timestamps, including at least image data, laser ranging data and audio signals; B. Preprocessing and spatial prior construction: The multimodal sensing information is preprocessed, the image is enhanced under poor lighting conditions, and the laser ranging data is parsed into near-range spatial geometry and orientation information to form spatial priors; C. Unified Encoding Input of Multimodal Data: The multimodal data is uniformly encoded into a combination of structured language summaries and images for parsing by multimodal large language models or other inference models; D. Scene Intelligent Agent Matching and Activation: Select and activate the matched scene intelligent agent system according to the user's instructions; the scene intelligent agent system is a configurable running unit, consisting of prompt word templates, toolchain configuration, strategy parameters, and attention weights, and supports hot switching and contextual inheritance during operation; the scene intelligent agent system provides a unified inference call interface and result publishing interface to the outside world. E. Multi-role AI instantiation and parallel execution: Under the scheduling of the scene intelligent agent system, the multimodal perception information is integrated and multiple AI role subsystems matching the current scene are instantiated and driven in parallel to generate candidate results. The central decision subsystem implements weighted fusion control based on the strategy parameters and attention weights of the scene intelligent agent system, and outputs unified scene decisions and executable instructions. F. Dual-mode output: The system provides two interaction modes within the scene's intelligent agent system: automatic cruise mode and interactive dialogue mode, and can seamlessly switch between the two modes under user commands.

[0007] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the preprocessing and spatial prior construction described in section B are as follows: B1. In low-light conditions, activate infrared supplementary lighting and coordinate with visible light acquisition; B2. Express the laser ranging data in the sensor coordinate system and map it to the ground coordinate system through the calibrated extrinsic parameters. Calculate the distance-azimuth element (r, θ) and generate a thresholded state based on a preset threshold to form structured spatial prior data with time stamp.

[0008] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the multimodal data unified encoding input mentioned in C is as follows: Multimodal data is uniformly encoded into structured language summaries and image representations, and then concatenated into a Prompt input that can be parsed by a large language model; Multimodal data includes visible light images, infrared-enhanced images, lidar laser ranging data, audio signals from speech, and historical context of dialogue and action recordings; Multimodal data comes with a collection timestamp, and semantic abstraction, structural labeling, and input assembly are completed through a unified module, adapting to mainstream multimodal large language model interfaces.

[0009] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the multimodal data unified encoding input described in C adopts a modular Prompt generation mechanism, the specific details of which are as follows: C1. Based on structured spatial prior data, the distance-orientation elements (r, θ) and thresholded states are transcribed into natural language text of spatial semantic summary; C2. Convert audio data into natural language content and extract user intent; C3. Organize the historical context of dialogue and action records into a concise language reference and retain time markers; C4. When infrared supplementary light is detected, low-light state event semantics are generated and injected in pairs with the current frame image at the same time to achieve image-state joint perception and prompting. C5. Modularly assemble all information modules into a text + image combination Prompt using structured tags, and input it into the large language model interface.

[0010] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the scenario-based intelligent agent matching and activation described in D is characterized by: D1. The scene intelligent agent is configured with an environmental scene library and a behavioral scene library. The environmental scene library includes multiple environmental scenes such as indoor, community, street, and crossing the road; the behavioral scene library includes multiple behavioral scenes such as text reading and dynamic object finding. D2. Each of the aforementioned scene intelligent agents is a configurable operating unit, and its configuration includes, Prompt word templates are used to define the assistant role and language constraints in this scenario / behavior, for generation and understanding by multimodal / large language models; Toolchain configuration, from the role AI subsystem, selects and enables a subset of role subsystems based on the target scenario / behavior; A set of strategy parameters is used to characterize the complexity of the scenario, including at least content density, dynamic intensity, and risk coefficient. Initial attention weights are used to set the initial weight vector π for the enabled role AI subsystem, serving as the starting value for the runtime attention allocation algorithm; D3. The scene intelligent agent is implemented in the form of a system, providing a unified inference call (text + image) interface and result publishing interface (text / audio) to the outside world.

[0011] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the scenario intelligent agent system described in E, which involves multi-role AI instantiation and parallel execution, operates using a multi-role AI collaborative mechanism, following the process described below: E1, Receive user key presses or voice commands; E2. Select and activate the scene intelligent agent system that matches the instruction, and load its prompt word template, toolchain configuration, policy parameters and attention weights; E3. Preprocess and route multimodal perception data, and distribute the data to the enabled role AI subsystems; E4. The scene intelligent agent system implements resource scheduling control for the activated role AI subsystem based on the policy parameters and attention weights it maintains. E5. Each role's AI subsystem executes in parallel according to the system schedule and outputs standardized candidate results on the system message bus. The candidate results include type, content, and timestamp. E6. Complete cross-role information collaboration and initial integration preparation on the message bus. Initial integration preparation includes deduplication, alignment and expiration management. E7. Under the constraint of time consistency, the central decision-making subsystem implements weighted fusion control based on the policy parameters and attention weights of the scene intelligent agent system to generate unified scene decisions and executable instructions. E8. Output the decision through the speech synthesis / navigation execution interface and write the key states back to the context storage to support hot switching and takeover during operation.

[0012] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the scenario-based intelligent agent system described in E4, based on its maintained policy parameters and attention weights, utilizes a scenario-driven policy activation and attention weight allocation mechanism for collaboration among multiple role-based AI subsystems, specifically including: E41. For each scene and behavior, provide corresponding complexity parameters upon activation, including: Content density The richness of scene elements; Dynamic intensity The number or speed of moving objects within the scene; Risk coefficient The degree of potential threats related to the safety of visually impaired individuals, and the risk of the scenario being affected by weather and uncertain factors; E42. Configure sensitivity coefficients for scene complexity parameters for each enabled role AI subsystem, including content sensitivity, dynamic sensitivity, and risk sensitivity; E43. Based on the scene complexity parameter and the sensitivity coefficient, calculate the importance score of each enabled role AI subsystem, and obtain the unnormalized weight through a monotonically non-decreasing mapping function; then normalize it on the set of currently enabled role AI subsystems, and apply upper / lower bounds and minimum safety quota constraints to obtain the attention weight vector. E44. The attention weight is used for resource scheduling (including sampling frequency, computing power share and call priority) to drive the AI ​​of each role in parallel, and also as a fusion weighting coefficient to participate in the result fusion of the central decision subsystem. Unactivated role AI subsystems are considered to have a weight of zero and do not participate in the current scheduling and fusion.

[0013] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the dual-mode output described in F is orchestrated by a mode-layer state machine, supporting both automatic cruise mode and interactive dialogue mode in parallel within the same scenario intelligent agent. The specific process is as follows: F1: Receive user voice / key commands; F2. Activate the target scene intelligent agent system; F31. Automatic cruise path: The adaptive broadcast rhythm controller determines the next acquisition / analysis time. The system autonomously acquires environmental images and triggers an intra-frame pipeline without user operation, and obtains the environmental analysis results in automatic cruise mode. F32. Interactive Dialogue Path: Triggers a single intra-frame pipeline as needed based on the user's voice intent to obtain the task answer or guidance result in the interactive dialogue mode. F4. Security Preemption: During the execution of steps F31 and / or F32, when the event priority arbitrator determines that there is a high-risk event, it will interrupt or temporarily preempt the current broadcast or dialogue, and take the high-risk alarm information as the priority output. F5. Publish the outputs generated in steps F31, F32 and F4 through the speech synthesis interface, and write back the key information to the context to support subsequent mode retention and switching.

[0014] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired people, the dual modes are supported in parallel within the same scenario intelligent agent, the two modes share multimodal perception information and the scenario intelligent agent system, and can be seamlessly switched under user commands; In automatic cruise mode, without requiring active user operation, the event priority arbitrator determines the broadcast focus, and the adaptive broadcast rhythm controller determines the next detection / broadcast time point to achieve low-interference continuous broadcasting. The interactive dialogue mode is explicitly triggered by the user via voice and / or button, and supports multiple rounds of follow-up questions and result confirmation; When a high-risk event is detected, the conversation can be prioritized and interrupted or temporarily preempted.

[0015] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the adaptive broadcast rhythm controller is used to determine the time point of the next detection or broadcast in automatic cruise mode, specifically including: Based on input factors including risk status, image stability, user status, and history of similar content broadcasts, and combined with a preset basic cycle, a state quantity for rhythm calculation is obtained. After each broadcast, the time point for the next detection or broadcast is updated according to the monotonous rules, including shortening the cycle when the risk increases, the image becomes unstable, or the user speeds up, and extending the cycle when the image remains stable or similar content has been broadcast recently. The update results are subject to minimum cooling time and maximum interval limits. Time smoothing or hysteresis is used to suppress rhythm jitter. If necessary, high-priority events are responded to trigger temporary early broadcasts.

[0016] In the aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the event priority arbitrator is used to sort and merge candidate events within the same period in automatic cruise mode, and includes: Maintain a configurable event priority queue with a basic hierarchy from high to low, including collision or fall risk, traffic and crossing, path guidance, and environmental description, to determine the order of broadcasts. For key events of concern, including steps or drops, night or low light, rain or snow or slippery conditions, and oncoming vehicles or bicycles, shorten the cooldown time of similar prompts, increase the relevant sampling rate, and allow one-time insertion when necessary. Candidate events that are close in location or semantically similar are merged and deduplicated, and only the highest priority summary is output. The arbitration result is written back to the context for subsequent rhythm control and arbitration strategy adjustment.

[0017] The aforementioned scenario-based intelligent agent navigation and environmental understanding method for visually impaired people also includes corresponding devices, systems, electronic devices, and storage media that apply the method.

[0018] The device includes: A multimodal data acquisition module is used to acquire multimodal sensing information and synchronize it with timestamps. The multimodal sensing information includes at least image data, laser ranging data, and audio signals. The preprocessing and spatial prior construction module is used to preprocess the multimodal sensing information, enhance the image under poor lighting conditions, and parse the laser ranging data into near-range spatial geometry and orientation information to form structured spatial prior data. The multimodal data unified encoding module is used to uniformly encode the image data, structured spatial prior data, audio signals and historical context into a combination of structured language summary and image input, and splice them into a Prompt that can be parsed by a multimodal large language model or other reasoning model; The scene agent matching and activation module is used to select and activate the matched scene agent according to the user's instructions. The scene agent is a configurable running unit, and its configuration includes at least prompt word template, toolchain configuration, strategy parameter set and initial attention weight value, which is used to support hot switching and context inheritance during operation. The multi-role AI instantiation and parallel execution module is used to integrate the multimodal perception information and instantiate and drive multiple AI role subsystems that match the current scene in parallel under the scheduling of the scene intelligent agent system to generate candidate results. The central decision subsystem implements weighted fusion control based on the strategy parameters and attention weights of the scene intelligent agent system to output unified scene decisions and executable instructions. The dual-mode interactive output module is used to provide two interactive modes within the same scene intelligent agent system: automatic cruise mode and interactive dialogue mode, and to seamlessly switch between the two modes under user commands.

[0019] Each module can be implemented in hardware, software functional modules, or a combination of hardware and software, and is not limited to a strict physical distinction.

[0020] The system includes: Wearable devices, intended to be worn on a user's head or eye area, include: Image acquisition component for acquiring visible light images and / or infrared-enhanced images; A laser ranging component is used to acquire laser ranging data in front or in a specified direction; Audio acquisition component, used to acquire user voice and environmental audio signals; Audio output component, used to broadcast navigation prompts and environmental descriptions to the user; The control and processing components are used to preprocess locally acquired multimodal data and interact with external processing units through the communication components. Communication components used to establish wired or wireless connections with mobile terminals, edge computing nodes, or cloud servers; Human-computer interaction components are used to receive user key presses or touch operations to trigger mode switching or task commands; The processing unit can be a processor inside a wearable device, a mobile terminal connected to it, or a remote server, and is configured to perform the steps of the method described in any of the preceding claims, including: multimodal data acquisition and preprocessing, spatial prior construction, multimodal unified encoding, scene intelligent agent matching and activation, multi-role AI collaboration, and dual-mode output control. The communication link is used to transmit multimodal perception data, scene agent configuration, decision results and control commands between the wearable device and the processing unit; The system achieves multimodal perception, contextual understanding, and voice navigation output of the user's surrounding environment through the collaborative work of wearable devices and processing units.

[0021] The electronic device includes: At least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the computer program being configured to, when executed by the at least one processor, implement the steps of the contextualized intelligent agent navigation and environment understanding method for visually impaired individuals as described in any of the preceding claims.

[0022] The computer-readable storage medium includes a computer program stored thereon, which, when executed by a processor, implements the steps of the scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals as described in any of the preceding claims.

[0023] Compared with the prior art, the beneficial effects of this application are as follows: 1. More stable perception in complex environments By using multimodal collaborative acquisition of "visible light image + infrared supplementary light enhanced image + laser ranging data", usable image and distance information can still be obtained in complex environments such as low light, backlight, nighttime streets, and traffic intersections. This significantly improves the problem of unstable perception in low light and backlight scenarios when traditional recognition relies solely on visible light images. 2. Possesses explicit spatial semantic understanding capabilities The laser ranging data is mapped to the ground coordinate system and the distance, orientation and other elements are calculated. It is then further transcribed into spatial semantic summary text, so that the system can not only identify "what is seen", but also accurately describe "how far away, in which direction, and whether it is close to the danger boundary" in natural language, thereby improving the accuracy of obstacle avoidance, path judgment and safety decision-making. 3. Contextualized intelligent agents bring differentiated strategies and adaptive scheduling. By introducing scene-based intelligent agents, different intelligent agent units are configured for various environmental / behavioral scenarios such as indoor, community, street, crossing the road, text reading, and dynamic object finding. Each intelligent agent is bound with prompt word templates, toolchain configurations, strategy parameter sets, and initial attention weights, enabling the system to automatically select appropriate AI role combinations and broadcasting strategies based on the scenario and user intent, resulting in output content that better meets actual needs. 4. Multi-role AI collaboration and attention weight control Within the same scene intelligent agent system, multiple AI subsystems with roles such as visual understanding, spatial reasoning, text dialogue, path planning, and central decision-making are set up. Attention weights are calculated based on scene complexity parameters such as content density, dynamic intensity, and risk coefficient. The resource allocation and result fusion of each AI role are weighted and controlled to realize a multi-agent collaborative mechanism where "scene determines strategy and strategy determines weight", making the overall decision more reliable and controllable. 5. It integrates automatic cruise control and interactive dialogue modes with low disturbance. Within the same framework, it provides an automatic cruise mode and an interactive dialogue mode, which are uniformly orchestrated by the mode layer state machine: the automatic cruise mode can continuously perceive the environment without user operation, and automatically adjust the broadcast frequency and filter key events through an adaptive broadcast rhythm controller and an event priority arbiter to achieve continuous safety prompts without hands or eyes; the interactive dialogue mode allows users to initiate task requests via voice or buttons and conduct multiple rounds of follow-up questions. High-risk events can also interrupt or preempt the current dialogue, effectively balancing safety and user comfort, and reducing information overload and frequent interruptions.

[0024] Furthermore, since the above methods can be implemented in electronic devices such as wearable smart glasses, mobile terminals, edge computing nodes or servers in the form of devices, systems and computer-readable storage media, the present invention not only improves the environmental perception and navigation capabilities of visually impaired people at the algorithm and process level, but also provides an implementation carrier that is easy to wear, deploy and upgrade in terms of device form, making it easy to promote and apply in actual products. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall system structure in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall process of the method of the present invention; Figure 3 This is a schematic diagram of the multimodal data preprocessing and unified coding logic in steps B and C of the present invention; Figure 4 This is a flowchart of the instantiation and parallel execution of scene intelligent agents and multi-role AI in steps D and E of the present invention; Figure 5This is a schematic diagram of the dual-mode output logic in step F of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0027] Example

[0028] Contextualized intelligent agent navigation and environmental understanding systems for visually impaired individuals, such as Figure 1 As shown, it includes: a wearable device, a main processing unit, and a communication link.

[0029] Wearable devices are worn on the user's head or eye area, serving as a perception layer to collect multimodal perception information of the user's surrounding environment and output navigation results to the user.

[0030] The wearable device uploads multimodal perception data to the main processing unit via a communication link and receives navigation and environmental understanding results and control commands issued by the main processing unit.

[0031] The main processing unit can be a processor inside the wearable device, a mobile terminal connected to the wearable device, or a remote server. As the main processing layer of the scene intelligent agent system, it is used to execute the various steps of the method of the present invention, including multimodal data preprocessing and spatial prior construction, multimodal unified encoding, scene intelligent agent matching and activation, multi-role AI instantiation and parallel execution, and dual-mode interactive output.

[0032] The communication link is used to transmit data between the wearable device and the main processing unit. It can establish an encrypted communication connection using Bluetooth, WiFi, or other wireless / wired methods. Through the communication link, the uplink transmission includes at least multimodal perception data such as image data, laser ranging data, and audio signals, while the downlink transmission includes at least voice broadcast content for navigation path and environmental understanding, mode switching instructions, and other control instructions.

[0033] The wearable device can integrate the following functional components: Image acquisition component for acquiring visible light images and / or infrared-enhanced images; Laser ranging components are used to collect laser ranging data in front of the user or in a specified direction; Audio acquisition component, used to acquire user voice and environmental audio signals; An audio output component is used to broadcast navigation prompts and environmental descriptions to the user; Human-computer interaction components are used to receive user key presses or voice commands to trigger mode switching or task commands; The control and preprocessing component is used to perform time synchronization and basic preprocessing on the locally acquired multimodal data, and to send the processed data to the main processing unit or receive control and broadcast content from the main processing unit through the communication component.

[0034] The main processing unit is configured to implement the functions described in this application, including the multimodal data acquisition module, the preprocessing and spatial prior construction module, the multimodal data unified encoding module, the scene intelligent agent matching and activation module, the multi-role AI instantiation and parallel execution module, and the dual-mode interactive output module.

[0035] The above functions can be implemented through software, hardware, or a combination of both. They do not require strict physical separation and can be distributed between the wearable device and the remote server according to actual deployment needs.

[0036] A scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, the overall process of which is as follows: Figure 2 As shown, it includes the following steps: A. Multimodal data acquisition: Multimodal sensing information is acquired by wearable smart devices and synchronized with timestamps. The multimodal sensing information includes visible light images, infrared supplementary enhanced images, laser ranging data, and audio signals. B. Preprocessing and spatial prior construction: Under poor lighting conditions, visible light images and infrared supplementary enhanced images are used together as the acquired images to analyze the laser ranging data into near-range spatial geometry and orientation information and form spatial priors. C. Unified Encoding Input of Multimodal Data: The multimodal data is uniformly encoded into a combination of structured language summary and image input; the encoding includes transcribing laser ranging data into spatial semantic summary, transcribing audio signals into user intent text, organizing historical context into language reference, and generating low-light state event semantics when infrared supplementary lighting is triggered and injecting it in pairs with the current frame image with the same time identifier; D. Scene Intelligent Agent Matching and Activation: Select and activate the matched scene intelligent agent system according to the user's instructions; the scene intelligent agent system is a configurable running unit, consisting of prompt word templates, toolchain configuration, strategy parameters, and attention weights, and supports hot switching and contextual inheritance during operation; the scene intelligent agent system provides a unified inference call interface and result publishing interface to the outside world. E. Multi-role AI Instantiation and Parallel Execution: Under the scheduling of the scene intelligent agent system, multimodal perception information is integrated and multiple AI role subsystems matching the current scene are instantiated and driven in parallel. The role AI subsystems include a visual recognition subsystem, a spatial reasoning subsystem, an obstacle avoidance analysis subsystem, a text dialogue subsystem, a path planning subsystem, and a central decision-making subsystem. The role AI subsystems generate candidate results based on standardized inputs, and the central decision-making subsystem implements unified weighted fusion control according to the policy parameters and attention weights of the scene intelligent agent, and generates unified scene decisions and executable instructions under time consistency constraints. F. Dual-mode output: Provides two interactive modes: automatic cruise mode and interactive dialogue mode, which can be seamlessly switched under user command; in automatic cruise mode, the event priority arbitration unit and the adaptive broadcast rhythm control unit are set to determine the broadcast focus and the next detection / broadcast time point respectively; in interactive dialogue mode, the user can complete the task response by explicitly triggering it through voice or button.

[0037] The following explains each step and its details: Step A: The wearable device collects multimodal perception information of the user's surrounding environment at a preset frame rate. The multimodal perception information includes at least: Image data: including visible light images and infrared-enhanced images acquired in low-light conditions; Laser ranging data includes ranging distance and echo intensity, used to describe the direction and distance of obstacles in the environment ahead or in a specified direction; Audio signals: including user voice and ambient sounds, used to recognize user commands and assist in understanding the environment.

[0038] The wearable device assigns a unified timestamp to each of the above modal data, and packages the multimodal data under the same timestamp into a frame of multimodal data for subsequent preprocessing and unified encoding.

[0039] like Figure 3 The diagram shown is a schematic diagram of the multimodal data preprocessing and unified encoding logic in steps B and C of this invention.

[0040] The preprocessing and spatial prior construction described in step B are as follows: B1. In low-light conditions, infrared supplementary light is activated and visible light is collected in conjunction with infrared supplementary light. The wearable device has a built-in photosensitive sensor. When the ambient brightness is detected to be lower than a preset threshold, the near-infrared LED supplementary light is automatically turned on, and visible light images and infrared supplementary light enhanced images are collected simultaneously with a synchronous timestamp. B2. Laser ranging data analysis and spatial prior construction, i.e. (1) Express the laser ranging data in the sensor coordinate system. Let the distance of a single ranging point be d, the corresponding pitch angle and horizontal angle be (φ, θ), and the sensor installation height be h. Then the position estimate of the obstacle in three-dimensional space can be obtained: horizontal forward distance r = d; the three-dimensional coordinates (x, y, z) can be calculated from d, φ, and θ according to the sensor installation posture. (2) Using the calibrated extrinsic parameters, the point cloud in the sensor coordinate system is mapped to the ground coordinate system to obtain the distance-azimuth elements (r, θ) and height information of the obstacle; (3) Based on the preset distance range and height threshold, map (r, θ) to a local grid map or equivalent data structure to generate thresholded states such as "near danger zone, attention zone, and safe zone"; (4) The above distance-orientation elements and thresholded states are organized into structured spatial prior data according to timestamps, which are used as inputs for subsequent multimodal unified coding and spatial semantic summarization generation.

[0041] Step C, which involves unified encoding input of multimodal data, details are as follows: C1. LiDAR Semantic Summarization Based on Spatial Priors Based on the structured spatial prior data obtained in step B2, the unified coding module transcribes the distance-orientation elements (r,θ) and thresholding states into natural language text of spatial semantic summary; For example: When a cluster of continuous obstacles with a height of about waist level is detected within 1 meter directly in front of the user, the semantics can be generated as: "There is a row of obstacles with a height of about waist level about 1 meter directly in front of the user". When a raised curb is detected 2 meters to the left front, the semantics can be generated as: "There is a step or raised curb about 2 meters to the left front".

[0042] The semantic summary is output in the format of "[LiDAR][Timestamp]Spatial Semantic Description", for example: "[LiDAR][08:21:33]Obstruction about 0.8 meters in front".

[0043] C2. Audio Transcription and User Intent Extraction The audio data is transcribed into natural language text by the speech recognition module, and then the intent recognition module extracts the user's task intent, target object, and constraints, which are then used as the intent module for subsequent Prompt generation. For example: "[Intention][08:21:30] I want to cross this area, are there any obstacles?"

[0044] C3. Historical Context Arrangement Organize historical information such as dialogue logs, past navigation results, and recent obstacle announcements into a concise language reference in chronological order, while retaining time markers; For example: "[Historical Reference][08:21:20] The mouse is on the right side of the keyboard, close to the screen." The historical context management module controls the number and time range of historical segments, ensuring a balance between relevance and conciseness.

[0045] C4. Low-light state event semantics and image pair injection When the built-in photosensor of the camera module triggers an extreme value, the system automatically activates the near-infrared LED fill light. This behavior is captured by the system and the fill light trigger flag is set. At this point: acquire the current frame's infrared supplemental lighting enhanced image I_IR, and encode it into an embeddable format such as base64, while recording the acquisition timestamp and binding it to the image; generate a structured language summary based on the supplemental lighting markers; For example: "

Status Reminder

[0046] When constructing a multimodal Prompt, the status reminder text and the I_IR image are injected in pairs with the same time identifier to achieve joint perception of "semantic status + image".

[0047] Example injection structure can be: { "role": "user", "content": [ { "type": "text", "text": "

Status Alert

[0048] C5, Modular Prompt Assembly The unified encoding module modularly splices the above information modules according to structured tags to form a text + image combination Prompt, which serves as the input to the large language model interface; One example structure is as follows: { "role": "user", "content": [ {"type": "text", "text": "

LiDAR

Intention

Historical Reference

Status Reminder

[0049] Through the above mechanism, a unified expression and parsable input of multimodal information on the language model side are achieved.

[0050] like Figure 4 The diagram shown is a flowchart of the instantiation and parallel execution of scene intelligent agents and multi-role AI in steps D and E of this invention.

[0051] Step D, which involves matching and activating the scene agent, is detailed below: D1. The scene intelligent agent is configured with an environmental scene library and a behavioral scene library. The environmental scene library includes at least indoor, community, street, and cross-road environmental scenes; the behavioral scene library includes at least text reading and dynamic object finding behavioral scenes. D2. Each of the aforementioned scene intelligent agents is a configurable operating unit, and its configuration includes, Prompts are used to define the assistant role, focus elements, and language constraints in this scenario / behavior, providing priors for the generation and understanding of multimodal large language models; Toolchain Configuration (Tools): Specifies the subset of AI subsystems to be enabled in this scenario, such as visual recognition AI, obstacle avoidance analysis AI, spatial reasoning AI, navigation AI, etc. Policy parameter set: used to characterize the complexity of the scene, including at least: content density C, dynamic intensity D, and risk coefficient R; Initial attention weights (π): The initial weight vector given to the enabled role AI subsystem, serving as the starting value for running the attention recalculation.

[0052] For example, the following strategies can be configured for different scenarios: Indoor environment: Visual recognition AI, obstacle avoidance analysis AI, and spatial reasoning AI are enabled, with C=0.50, D=0.30, and R=0.20; Community environment: Visual recognition AI, obstacle avoidance analysis AI, spatial reasoning AI, and navigation AI are enabled, with C=0.60, D=0.60, and R=0.40; Street environment: Visual recognition AI, obstacle avoidance analysis AI, spatial reasoning AI, and navigation AI are enabled, with C=0.60, D=0.70, and R=0.60; Crossing the road environment: Visual recognition AI, obstacle avoidance analysis AI, and spatial reasoning AI are enabled, with C=0.50, D=0.90, and R=0.80; Text reading: Primarily utilizes visual recognition AI, C=0.80, D=0.05, R=0.00; Dynamic object finding: Visual recognition AI and spatial reasoning AI are enabled, C=0.60, D=0.40, R=0.10.

[0053] D3. Unified Interface Implementation: The scene intelligent agent is implemented in the form of a system, providing a unified inference call interface and result publishing interface to the outside world.

[0054] Step E, which involves the instantiation and parallel execution of multi-role AI, proceeds as follows: E1, Receive user key presses or voice commands; E2. Select and activate the scene intelligent agent system that matches the instruction, and load its prompt word template, toolchain configuration, policy parameters and attention weights; E3. Preprocess and route multimodal perception data, and distribute the data to the enabled role AI subsystems; E4. The aforementioned scene intelligent agent system, based on the policy parameters and attention weights it maintains, uses a scene-driven policy activation and attention weight allocation mechanism for the role AI subsystem. E5. Each role's AI subsystem executes in parallel according to the system schedule and outputs standardized candidate results on the system message bus. The candidate results include type, content, and timestamp. E6. Complete cross-role information collaboration and initial integration preparation on the message bus. Initial integration preparation includes deduplication, alignment and expiration management. E7. Under the constraint of time consistency, the central decision-making subsystem implements weighted fusion control based on the policy parameters and attention weights of the scene intelligent agent to generate unified scene decisions and executable instructions. E8. The decision is output through the speech synthesis / navigation execution interface, and key states are written back to the context storage to support hot switching and transition during operation. Specifically, hot switching uses a lag window and cooling time to suppress frequent switching; old scenes exit in an orderly manner, and new scenes are loaded and warmed up in the background to avoid "quiet periods".

[0055] Step E4 specifically includes: E41. For each scene and behavior, provide corresponding complexity parameters upon activation, including: Content density The richness of scene elements; Dynamic intensity The number or speed of moving objects within the scene; Risk coefficient The degree of potential threats related to the safety of visually impaired individuals, and the risk of the scenario being affected by weather and uncertain factors; E42. Configure a sensitivity coefficient for scene complexity parameters, including content sensitivity, for each enabled role AI subsystem. Dynamic sensitivity Risk sensitivity ; E43. Based on the scene complexity parameter and the sensitivity coefficient, calculate the importance score of each enabled role AI subsystem, and obtain the unnormalized weight through a monotonically non-decreasing mapping function; then normalize it on the set of currently enabled role AI subsystems, and apply upper / lower bounds and minimum safety quota constraints to obtain the attention weight vector. First, the role AI subsystem is defined as including the visual recognition subsystem AI_V, the obstacle avoidance analysis subsystem AI_O, the spatial reasoning subsystem AI_S, and the path planning subsystem AI_N. , Secondly, based on the scene complexity parameter and sensitivity coefficient, the importance score of each enabled AI role is calculated, and the unnormalized weight is obtained through a monotonically non-decreasing mapping function. Where α is the overall sensitivity, and α > 0; Finally, the above weights are normalized on the currently enabled set of roles, and upper / lower bounds and minimum safety quota constraints are applied to obtain the attention weight vector, i.e. ; E44. The attention weights are used for resource scheduling to drive the AI ​​of each role in parallel, and also as a fusion weighting coefficient to participate in the result fusion of the central decision-making subsystem; the weights of unused roles are considered to be zero and do not participate in the current scheduling and fusion.

[0056] Through the aforementioned multi-role AI collaboration mechanism, an adaptive reasoning closed loop of "scenario determines strategy, and strategy determines weight" is achieved.

[0057] like Figure 5 The diagram shown is a schematic diagram of the F dual-mode output logic of the present invention.

[0058] The dual-mode output mentioned in step F has the following specific process: F1: Receive user voice / key commands; F2. Activate the target scene intelligent agent system; F3, Enter mode selection logic: To select the automatic cruise mode, press F31; To select interactive dialogue mode, press F32. F31, Automatic Cruise Path: The adaptive broadcast rhythm controller determines the next acquisition / analysis time. The system autonomously acquires environmental images without the need for active user operation, triggers a round of multimodal processing pipeline, and completes environmental understanding and navigation decision-making. F32. Interactive Dialogue Path: Trigger a single multimodal processing pipeline as needed based on the user's voice intent, and provide answers to user questions or perform specified tasks. F4. Security Preemption: During the execution of steps F31 and / or F32, when the event priority arbitrator determines that there is a high-risk event, it will interrupt or temporarily preempt the current broadcast or dialogue, and take the high-risk alarm information as the priority output. F5. Publish the outputs generated in steps F31, F32 and F4 through the speech synthesis interface, and write back the key information to the context to support subsequent mode retention and switching.

[0059] In automatic cruise mode, without requiring active user operation, the event priority arbitrator determines the broadcast focus, and the adaptive broadcast rhythm controller determines the next detection / broadcast time point to achieve low-interference continuous broadcasting. The interactive dialogue mode is explicitly triggered by the user via voice and / or button, and supports multiple rounds of follow-up questions and result confirmation; When a high-risk event is detected, the conversation can be prioritized and interrupted or temporarily preempted.

[0060] For example, users can switch back to automatic mode by giving a command like "resume cruise control"; Users can enter conversation mode by saying "I want to ask you a question" or using an "interact" command. Each switch preserves the environment context, reducing the burden of repetitive interactions.

[0061] The adaptive broadcast rhythm controller is used to determine the time point of the next detection or broadcast in automatic cruise mode, specifically including: Based on input factors including risk status, image stability, user status, and history of similar content broadcasts, and combined with a preset basic cycle, a state quantity for rhythm calculation is obtained. After each broadcast, the time point for the next detection or broadcast is updated according to the monotonous rules, including shortening the cycle when the risk increases, the image becomes unstable, or the user speeds up, and extending the cycle when the image remains stable or similar content has been broadcast recently. The update results are subject to minimum cooling time and maximum interval limits. Time smoothing or hysteresis is used to suppress rhythm jitter. If necessary, high-priority events are responded to trigger temporary early broadcasts.

[0062] The adaptive broadcast rhythm controller is used to control the frequency, and its specific details are as follows: The adaptive broadcast rhythm controller updates the next detection or broadcast time t_next after each round of broadcasting. The core input and judgment rules are as follows: risk_state indicates whether the current risk level is high (e.g., close-range obstacles / oncoming vehicles / drop). scene_stability: Whether the scene has been stable in the last N frames; user_state indicates whether the user is moving or stationary. Has talk_recent recently broadcast similar content? After each broadcast, the controller, in conjunction with the aforementioned state variables and the preset basic cycle, updates the next detection / broadcast time t_next according to the monotonic rule of "shortening the cycle when the risk increases, the image becomes unstable, or the user accelerates, and extending the cycle when the scene remains stable and similar content has been broadcast recently." It imposes constraints on t_next with minimum cooldown time and maximum interval limit, and suppresses rhythm jitter through time smoothing or hysteresis. If necessary, it responds to high-priority events to trigger temporary early broadcasts.

[0063] For example, setting a base cycle , ; ; The event priority arbitrator is used to sort and merge candidate events within the same period in automatic cruise mode, and includes: Maintain a configurable event priority queue with a basic hierarchy from high to low, including collision or fall risk, traffic and crossing, path guidance, and environmental description, to determine the order of broadcasts. For key events of concern, including steps or drops, night or low light, rain or snow or slippery conditions, and oncoming vehicles or bicycles, shorten the cooldown time of similar prompts, increase the relevant sampling rate, and allow one-time insertion when necessary. Candidate events that are close in location or semantically similar are merged and deduplicated, and only the highest priority summary is output. The arbitration result is written back to the context for subsequent rhythm control and arbitration strategy adjustment.

[0064] Through the aforementioned dual-mode output mechanism, this invention achieves coordinated operation of automatic navigation and interactive dialogue under the same scene intelligent agent, ensuring the safety of visually impaired users while taking into account the integrity of information and the low-intrusion nature of interaction.

[0065] Application Examples Based on the scenario-based intelligent agent navigation and environmental understanding system for visually impaired people provided in the above embodiments, the application process of the method of the present invention will be described in detail using the actual scenario of a visually impaired user walking on a city street and crossing an intersection as an example.

[0066] Users start from the entrance / exit of the community, walk along the pedestrian walkway to the street intersection, cross the street safely at the traffic light-controlled zebra crossing, and finally arrive at the target bus stop.

[0067] It should be noted that the following description uses a city street scene as an example, which is only used to illustrate the technical solution of the present invention and does not constitute a limitation on the scope of protection of the present invention.

[0068] Users with visual impairments can wear the wearable device provided by this invention. The device includes an image acquisition component, a laser ranging component, an audio acquisition component, an audio output component, and a human-computer interaction component.

[0069] Users can issue task requests at the community entrances and exits via buttons or voice commands, for example: "Activate pedestrian navigation and navigate to the XXXX bus stop."

[0070] The human-computer interaction component uploads the instruction to the processing unit; the scene intelligent agent matching and activation module selects the "city street intelligent agent" from the intelligent agent configuration library according to the user's task type and current location information, and loads the prompt word template, toolchain configuration, strategy parameters and initial value of attention weight corresponding to the street scene.

[0071] The intelligent agent focuses on scene elements including: sidewalk boundaries, changes in curb height, pedestrian and non-motorized vehicle traffic, temporary obstacles, shop signs and bus stop signs, etc.

[0072] Meanwhile, the system defaults to automatic cruise mode, with the subsequent broadcast rhythm and information output method controlled by the dual-mode interactive output module.

[0073] The user walks forward along the sidewalk; the wearable device collects multimodal perception information of the environment in front of the user at a preset frame rate (e.g., 5 frames / second), and synchronizes it with a unified timestamp: The image acquisition component captures a visible light image in front of the user, covering a horizontal field of view of approximately 160°; The laser ranging component periodically emits laser beams within a fan-shaped area of ​​±30° directly in front of the user, and collects the distance values ​​d and echo intensity I at several ranging points; The audio acquisition component continuously collects user voice and environmental audio to identify new user commands and perceive surrounding environmental sounds such as traffic noise and horn sounds.

[0074] The preprocessing and spatial prior construction module performs the following processing on the above multimodal data: The visible light image is denoised, its brightness and contrast are adjusted, and the ground area is enhanced to better identify details such as curbs, steps, and speed bumps. The laser ranging data is used to generate corresponding structured spatial prior data to describe the distribution of obstacles within a range of 0.2 to 18 meters ahead; For example, when a row of shared bicycles occupies the sidewalk 1.2 meters in front of the user, the laser ranging component obtains dense near-range ranging points in the corresponding direction, and the corresponding grid in the spatial prior data is marked as an obstacle area with an "obstacle + occupancy distance of about 1.0 to 1.4 meters".

[0075] To enable the model to understand the aforementioned spatial structure, the system converts laser ranging and spatial priors into a linguistic spatial semantic summary; based on obstacle distance and direction, obstacle risks are classified into levels such as near-field danger zone, attention zone, and background zone.

[0076] When an obstacle cluster is detected 1.2 meters directly in front of the user, and the obstacle area largely coincides with the current walking direction, the multimodal data unified encoding module generates a data structure like this: “

LiDAR

[0077] The spatial semantic summary and the current frame image are used together as input fragments and spliced ​​into a unified prompt. The visual recognition subsystem identifies the obstacle as a shared bicycle at waist height based on the image. The spatial reasoning subsystem combines the grid map to conclude that "the obstacle is mainly concentrated in the middle of the sidewalk, and there is a passable gap of 0.6 meters about 0.8 meters to the left front". Based on this, the obstacle avoidance analysis subsystem provides detour suggestions.

[0078] The central decision-making subsystem, based on the policy parameter set configured for the city street intelligent agent (e.g., content density C=0.60, dynamic intensity D=0.70, risk coefficient R=0.60) and corresponding attention weights, weights and fuses candidate results from visual recognition AI, obstacle avoidance analysis AI, spatial reasoning AI, and path planning AI to form a unified decision. For example: "It is recommended to shift slightly to the left by half a step to avoid the obstacle, and then walk straight for two meters back to the center of the sidewalk."

[0079] The dual-mode interactive output module, based on the current risk_state, determines that the event is of medium to high risk and triggers a timely broadcast, alerting the user via the audio output component. "There is a row of parked bicycles about one meter ahead. Move half a step to the left to go around them." At the same time, the broadcast time is recorded for adaptive adjustment of the broadcast rhythm in subsequent broadcasts.

[0080] When the user continues walking along the street and enters a shady area or a section of road with poor street lighting, the photosensor detects that the ambient brightness is lower than a preset threshold, and the infrared supplementary lighting component automatically turns on.

[0081] The preprocessing and spatial prior construction module marks the current timestamp ts_lowlight as a "low-light state," while the image acquisition component simultaneously acquires infrared-enhanced images. The multimodal data unified encoding module generates state event semantics, such as: "[Status Alert][08:22:10] The current ambient light is dim, and infrared night vision mode has been activated." The status alert text and the corresponding infrared enhanced image are injected into a unified prompt in pairs at the same time, enabling the multimodal large language model to perceive that the current scene is low light and adjust its tolerance for image quality appropriately when performing object recognition and environmental understanding.

[0082] In low-light conditions, the risk weight in the scene agent's policy parameters is increased, and the adaptive broadcast rhythm control unit shortens the next detection / broadcast interval t_next, enabling the system to check obstacles and terrain changes more frequently in poor lighting conditions.

[0083] When the user is approximately 10 meters from the intersection ahead, the visual recognition subsystem detects an intersection due to the termination of the pedestrian crossing, the appearance of the traffic lane, and the display of road markings. In the spatial prior, the pedestrian crossing grid gradually narrows and becomes adjacent to the traffic lane grid. At this point, the city street agent, based on environmental characteristics and policy parameters, determines that "a pedestrian crossing scenario is about to begin," triggering the scenario switching condition.

[0084] In one approach, the system can proactively provide prompts: "There is an intersection about ten meters ahead. If you need to cross the road, you can say 'Help me cross the road safely'." The user issues the following via voice command: "Help me find the zebra crossing and cross the road safely"; After receiving the instruction, the scene intelligent agent matching and activation module switches to activate the "crossing the road intelligent agent" from the intelligent agent library and loads the prompt word template and strategy parameters specifically for the crossing the road scenario. This intelligent agent focuses more on elements such as traffic light status, traffic flow direction and speed, zebra crossing position, curb height changes, and safe crossing time window.

[0085] Under the scheduling of the pedestrian crossing agent, the attention weights of the multi-role AI subsystems are redistributed, and the policy parameters for this scenario are set as content density C=0.50, dynamic intensity D=0.90, and risk coefficient R=0.80. Visual recognition AI, obstacle avoidance analysis AI, and spatial reasoning AI are enabled as the main role subsystems.

[0086] The visual recognition subsystem detects pedestrian crossing markings, traffic light surfaces, and vehicles and pedestrians near the intersection in the current frame and adjacent frames. The obstacle avoidance analysis subsystem uses visual recognition results and LiDAR spatial prior data to perform fusion analysis on static obstacles (such as curbs, steps, etc.) and dynamic obstacles (such as vehicles, bicycles and pedestrians crossing laterally) located in the user's direction of travel, and generates obstacle avoidance suggestions. The spatial reasoning subsystem estimates the deviation distance between the user's current position and the center line of the pedestrian crossing based on spatial priors and visual recognition results, and determines the alignment direction. If a significant increase in traffic noise or rapid honking is detected in the continuously collected audio signals, the system will dynamically increase the risk factor R.

[0087] The central decision-making subsystem, combining the outputs from the aforementioned multiple roles, provides the following phased instructions: Approaching the zebra crossing: "There is a zebra crossing three meters ahead. Please turn slightly to the right to align with the zebra crossing." During red lights or periods of heavy traffic: "The light is red. Please wait here. There are vehicles approaching ahead." When the traffic light turns green and there is sufficient gap between vehicles: "The light is green now. The vehicles ahead have stopped. You can cross the road directly. It will take about ten seconds." During the crossing, the system maintains a high frequency of detection. When it detects a non-motorized vehicle or motorized vehicle approaching at high speed from the side, and determines that there is a potential collision risk, it will issue a high-priority warning: "A vehicle is approaching from the right. Please slow down slightly and be aware of surrounding vehicles."

[0088] While walking or waiting to cross the street, users can activate the interactive dialogue mode at any time via button or voice, for example, by asking a question: What shop is on the right side ahead? How far am I from the bus stop? After receiving a query, the system briefly switches to interactive dialogue mode. While preserving the current scene agent and multimodal context, it reconstructs a unified Prompt and inputs the user's query text, the current frame image, the most recent spatial semantic summary, and state events into the multimodal large language model.

[0089] For example, the visual recognition subsystem identifies the text "Convenience Store" on the shop sign on the right, the path planning subsystem determines that the current distance to the target bus stop is approximately 50 meters, the central decision-making subsystem passes the response to the text dialogue subsystem to organize it into natural language, and finally broadcasts it through the audio output component: "There is a convenience store three meters to your right, and the bus stop you need is about fifty meters straight ahead." After the task is completed, under the control of the dual-mode interactive output module, the user can switch to the automatic cruise mode to continue navigation and safety monitoring according to the strategy of the road crossing agent.

[0090] When the visual recognition subsystem and spatial reasoning subsystem detect a bus stop sign, stop name sign, or distinctive bus shelter structure at close range ahead, and this matches the user's preset target, the central decision-making subsystem generates a "reach the target" scenario decision, for example: "The bus stop you need is just one meter ahead. You can stop now." The system announces the arrival information and marks the current task status as completed.

[0091] Users can end the navigation service by pressing a button or using voice commands, or continue using the interactive dialogue mode to inquire about waiting information or other content while still in place.

[0092] As can be seen from the above street scene examples, the scenario-based intelligent agent navigation and environmental understanding method of the present invention can provide visually impaired users with continuous, detailed and safety-priority navigation and environmental understanding services by using multimodal data collected by wearable devices and combining the collaborative decision-making of scenario intelligent agents and multi-role AI in complex scenarios such as real streets, low light and crossing the street.

Claims

1. A scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals, characterized in that, The process includes the following steps: A. Multimodal data acquisition: Acquire multimodal sensing information and synchronize it with timestamps, including at least image data, laser ranging data and audio signals; B. Preprocessing and spatial prior construction: The multimodal sensing information is preprocessed, the image is enhanced under poor lighting conditions, and the laser ranging data is parsed into near-range spatial geometry and orientation information to form spatial priors; C. Unified Encoding Input of Multimodal Data: The multimodal data is uniformly encoded into a combination of structured language summaries and images for parsing by multimodal large language models or other inference models; D. Scene intelligent agent matching and activation: Select and activate the matching scene intelligent agent system according to the user's instructions; The scene intelligent agent system is a configurable operating unit, consisting of prompt word templates, toolchain configurations, policy parameters, and attention weights, supporting hot switching and context handling during operation; the scene intelligent agent system provides a unified inference call interface and result publishing interface to the outside world; E. Multi-role AI instantiation and parallel execution: Under the scheduling of the scene intelligent agent system, the multimodal perception information is integrated and multiple AI role subsystems matching the current scene are instantiated and driven in parallel to generate candidate results. The central decision subsystem implements weighted fusion control based on the strategy parameters and attention weights of the scene intelligent agent system, and outputs unified scene decisions and executable instructions. F. Dual-mode output: The system provides two interaction modes within the scene's intelligent agent system: automatic cruise mode and interactive dialogue mode, and can seamlessly switch between the two modes under user commands.

2. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 1, characterized in that, The preprocessing and spatial prior construction described in section B are as follows: B1. In low-light conditions, activate infrared supplementary lighting and coordinate with visible light acquisition; B2. Express the laser ranging data in the sensor coordinate system and map it to the ground coordinate system through the calibrated extrinsic parameters. Calculate the distance-azimuth element (r, θ) and generate a thresholded state based on a preset threshold to form structured spatial prior data with time stamp.

3. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 1, characterized in that, The unified encoding input of multimodal data described in C is as follows: Multimodal data is uniformly encoded into structured language summaries and image representations, and then concatenated into a Prompt input that can be parsed by a large language model; Multimodal data includes visible light images, infrared-enhanced images, lidar laser ranging data, audio signals from speech, and historical context of dialogue and action recordings; Multimodal data comes with a collection timestamp, and semantic abstraction, structural labeling, and input assembly are completed through a unified module, adapting to mainstream multimodal large language model interfaces.

4. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 3, characterized in that, The multimodal data unified encoding input described in C adopts a modular Prompt generation mechanism, the details of which are as follows: C1. Based on structured spatial prior data, the distance-orientation elements (r, θ) and thresholded states are transcribed into natural language text of spatial semantic summary; C2. Convert audio data into natural language content and extract user intent; C3. Organize the historical context of dialogue and action records into a concise language reference and retain time markers; C4. When infrared supplementary lighting is triggered, low-light state event semantics are generated and injected in pairs with the current frame image at the same time to achieve image-state joint perception and prompting. C5. Modularly assemble all information modules into a text + image combination Prompt using structured tags, and input it into the large language model interface.

5. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 1, wherein the scenario intelligent agent matching and activation described in claim 1 is characterized in that: D1. The scene intelligent agent is configured with an environmental scene library and a behavioral scene library. The environmental scene library includes multiple environmental scenes such as indoor, community, street, and crossing the road; the behavioral scene library includes multiple behavioral scenes such as text reading and dynamic object finding. D2. Each of the aforementioned scene intelligent agents is a configurable operating unit, and its configuration includes, Prompt word templates are used to define the assistant role and language constraints in this scenario / behavior, for generation and understanding by multimodal / large language models; Toolchain configuration: Select and enable a subset of the character AI subsystems from the character AI subsystems based on the target scenario / behavior. A set of strategy parameters is used to characterize the complexity of the scenario, including at least content density, dynamic intensity, and risk coefficient. Initial attention weights are used to set the initial weight vector π for the enabled role AI subsystem, serving as the starting value for the runtime attention allocation algorithm; D3. The scene intelligent agent is implemented in the form of a system, providing a unified inference call interface and result publishing interface to the outside world.

6. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 1, characterized in that: The scene intelligent agent system described in E consists of multiple role AI subsystems. Each role AI subsystem is responsible for a specific task. The role AI subsystems include, but are not limited to, a visual recognition subsystem, a spatial reasoning subsystem, an obstacle avoidance analysis subsystem, a text dialogue subsystem, a path planning subsystem, and a central decision-making subsystem.

7. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 1, characterized in that, In the multi-role AI instantiation and parallel execution described in E, the scene intelligent agent system operates with multiple role AI subsystems working together, following the process: E1, Receive user key presses or voice commands; E2. Select and activate the scene intelligent agent system that matches the instruction, and load its prompt word template, toolchain configuration, policy parameters and attention weights; E3. Preprocess and route multimodal perception data, and distribute the data to the enabled role AI subsystems; E4. The scene intelligent agent system implements resource scheduling control for the activated role AI subsystem based on the policy parameters and attention weights it maintains. E5. Each role's AI subsystem executes in parallel according to the system schedule and outputs standardized candidate results on the system message bus. The candidate results include type, content, and timestamp. E6. Complete cross-role information collaboration and initial integration preparation on the message bus. Initial integration preparation includes deduplication, alignment and expiration management. E7. Under the constraint of time consistency, the central decision-making subsystem implements weighted fusion control based on the policy parameters and attention weights of the scene intelligent agent system to generate unified scene decisions and executable instructions. E8. Output the decision through the speech synthesis / navigation execution interface and write the key states back to the context storage to support hot switching and takeover during operation.

8. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 7, characterized in that, The scene-based intelligent agent system described in E4, based on its maintained policy parameters and attention weights, utilizes a scene-driven policy activation and attention weight allocation mechanism for collaboration among multiple role-based AI subsystems, specifically including: E41. For each scene and behavior, provide corresponding complexity parameters upon activation, including: Content density The richness of scene elements; Dynamic intensity The number or speed of moving objects within the scene; Risk coefficient The degree of potential threats related to the safety of visually impaired individuals, and the risk of the scenario being affected by weather and uncertain factors; E42. Configure sensitivity coefficients for scene complexity parameters for each enabled role AI subsystem, including content sensitivity, dynamic sensitivity, and risk sensitivity; E43. Based on the scene complexity parameter and the sensitivity coefficient, calculate the importance score of each enabled role AI subsystem, and obtain the unnormalized weight through a monotonically non-decreasing mapping function; then normalize it on the set of currently enabled role AI subsystems, and apply upper / lower bounds and minimum safety quota constraints to obtain the attention weight vector. E44. The attention weights are used for resource scheduling to drive the AI ​​of each role in parallel, and also as fusion weighting coefficients to participate in the result fusion of the central decision-making subsystem. Unactivated role AI subsystems are considered to have a weight of zero and do not participate in the current scheduling and fusion.

9. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 1, characterized in that, The dual-mode output described in F is orchestrated by a mode-layer state machine, supporting both automatic cruise mode and interactive dialogue mode in parallel within the same scene's intelligent agent system. The specific process is as follows: F1: Receive user voice / key commands; F2. Activate the target scene intelligent agent system; F3: Enters the mode selection logic. Selecting the automatic cruise mode will execute F31; selecting the interactive dialogue mode will execute F32. F31. Automatic cruise path: The adaptive broadcast rhythm controller determines the next acquisition / analysis time. The system autonomously acquires environmental images and triggers an intra-frame pipeline without user operation, and obtains the environmental analysis results in automatic cruise mode. F32. Interactive Dialogue Path: Triggers a single intra-frame pipeline as needed based on the user's voice intent to obtain the task answer or guidance result in the interactive dialogue mode. F4. Security Preemption: During the execution of step F31 or step F32, when the event priority arbitrator determines that there is a high-risk event, it will interrupt or temporarily preempt the current broadcast or dialogue, and take the high-risk alarm information as the priority output. F5. Publish the outputs generated in steps F31, F32 and F4 through the speech synthesis interface, and write back the key information to the context to support subsequent mode retention and switching.

10. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 9, characterized in that: The dual modes are supported in parallel within the same scene intelligent agent. The two modes share multimodal perception information and scene intelligent agent system, and can be seamlessly switched under user commands. In automatic cruise mode, without requiring active user operation, the event priority arbitrator determines the broadcast focus, and the adaptive broadcast rhythm controller determines the next detection / broadcast time point to achieve low-interference continuous broadcasting. The interactive dialogue mode is explicitly triggered by the user via voice and / or button, and supports multiple rounds of follow-up questions and result confirmation; When a high-risk event is detected, the conversation can be prioritized and interrupted or temporarily preempted.

11. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 10, characterized in that, The adaptive broadcast rhythm controller is used to determine the time point of the next detection or broadcast in automatic cruise mode, specifically including: Based on input factors including risk status, image stability, user status, and history of similar content broadcasts, and combined with a preset basic cycle, a state quantity for rhythm calculation is obtained. After each broadcast, the time point for the next detection or broadcast is updated according to the monotonous rules, including shortening the cycle when the risk increases, the image becomes unstable, or the user speeds up, and extending the cycle when the image remains stable or similar content has been broadcast recently. The update results are subject to minimum cooling time and maximum interval limits. Time smoothing or hysteresis is used to suppress rhythm jitter. If necessary, high-priority events are responded to trigger temporary early broadcasts.

12. The scenario-based intelligent agent navigation and environmental understanding method for visually impaired individuals according to claim 10, characterized in that, The event priority arbitrator is used to sort and merge candidate events within the same period in automatic cruise mode, and includes: Maintain a configurable event priority queue with a basic hierarchy from high to low, including collision or fall risk, traffic and crossing, path guidance, and environmental description, to determine the order of broadcasts. For key events of concern, including steps or drops, night or low light, rain or snow or slippery conditions, and oncoming vehicles or bicycles, shorten the cooldown time of similar prompts, increase the relevant sampling rate, and allow one-time insertion when necessary. Candidate events that are close in location or semantically similar are merged and deduplicated, and only the highest priority summary is output. The arbitration result is written back to the context for subsequent rhythm control and arbitration strategy adjustment.

Images