A method and device for active interaction control of streaming video for a blind assistance scene, and a medium

Abstract

A kind of flow video active interaction control method, equipment and medium for the scene of helping blind, construct the hierarchical task system for the scene of helping blind, train the semantic understanding and behavior alignment of multimodal large model to specific task of helping blind;Introduce the active interaction control mechanism for flow video, under the premise of not changing the bottom model parameter, realize the trigger output of environmental perception by the dynamic suffix switching of determination mode and generation mode;Finally, the incremental context updating strategy is used to optimize the inference process, while keeping the timing consistency and reducing the calculation redundancy.The invention solves the technical problems of lack of special data, passive interaction logic and calculation redundancy of multimodal large model in flow environment through the synergistic effect of data-driven task adaptation and active interaction mechanism.The general framework proposed in the invention provides an important theoretical basis and technical paradigm for the paradigm evolution of flow video understanding from "passive analysis" to "active interaction".

Description

Technical Field

[0001] This invention belongs to the field of computer artificial intelligence technology, and relates to computer vision, video understanding and video analysis. Specifically, it is a streaming video active interactive control method, device and medium for assistive visually impaired scenarios. Background Technology

[0002] Vision is the primary way humans acquire information from the outside world and interact with their environment, playing a vital role in daily life and social activities. However, for people with visual impairments, the lack or limitation of visual function poses numerous difficulties in areas such as environmental perception, target recognition, and travel safety.

[0003] To assist visually impaired individuals in environmental perception and safe travel, existing technologies include various assisted navigation methods based on traditional computer vision algorithms. These methods utilize techniques such as object detection, image classification, and semantic segmentation to identify obstacles, traffic lights, and zebra crossings. However, most of these methods are designed for single tasks, with different research efforts typically focusing on only one specific task, such as obstacle avoidance, traffic signal recognition, or road sign detection, lacking a unified multi-task collaboration and fusion mechanism. In practical applications, requiring visually impaired users to switch between different applications for different scenarios is impractical. Furthermore, traditional methods rely on closed-set training with specific labels, resulting in limited generalization and high-level semantic understanding capabilities. The lack of broad world knowledge makes it difficult for them to adapt to complex and ever-changing real-world environments and achieve deep scene understanding.

[0004] Therefore, despite a large number of related studies, visually impaired people still mainly rely on traditional assistive methods such as guide canes and guide dogs in their daily lives. However, the perception range of guide canes is limited to near-ground physical contact, resulting in a "perceptual blind spot"; while guide dogs can assist in travel to some extent, their training and maintenance costs are high, and they have significant limitations in complex semantic interactions and task understanding, making it difficult to flexibly adapt to the diverse and dynamically changing needs of users.

[0005] In recent years, with the rapid development of multimodal large model technology, artificial intelligence systems based on joint vision and language modeling have made significant progress in image and video understanding tasks, providing a new technical path for building intelligent interactive systems to assist visually impaired users. However, most existing multimodal large models focus on offline or post-event video analysis and processing, making it difficult to meet the needs of real-time environmental perception. At the same time, their interaction methods usually rely on a question-and-answer model where the user actively asks questions, lacking a proactive perception and timely feedback mechanism for key environmental events. Furthermore, due to the lack of dedicated datasets and targeted optimization training for visually impaired assistance scenarios, existing models struggle to accurately understand the user's state and generate optimal assistance decision-making information.

[0006] Therefore, how to construct a real-time video understanding and active interaction system for visually impaired assistance scenarios, realize continuous perception and intelligent response to dynamic environmental information, take into account multi-task processing capabilities and high-level semantic understanding capabilities, and break through the limitations of existing training data scale and quality, remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The technical problem this invention aims to solve is that existing visual aids for the blind are insufficient to fully meet practical needs, mainly due to the following issues:

[0008] 1) Poor task fragmentation and semantic generalization ability: Traditional vision algorithms are usually designed for a single closed set of tasks, lacking deep semantic understanding of complex open scenes and multi-task collaborative processing ability, making it difficult to adapt to the diverse needs of visually impaired users in open scenes.

[0009] 2) Passive interaction mode and delayed feedback: Existing multimodal large models mostly adopt a passive "query-response" interaction logic, which cannot achieve proactive perception and early warning based on environmental changes and potential risk levels in continuous visual flow; at the same time, due to the limitations of offline / post-event analysis architecture, it is difficult to meet the requirements of real-time and continuous perception in assistive visual scenarios.

[0010] 3) Lack of dedicated training data and construction methods for assistive visually impaired scenarios: Existing video dialogue and visual understanding datasets are mostly geared towards general scenarios, lacking high-quality training data that is centered on the perspective of visually impaired users and achieves multi-level alignment of "target-semantics-interaction"; in addition, the cost of relying entirely on manual annotation to construct large-scale video datasets is high, which seriously restricts the training and iteration of dedicated models for assistive visually impaired scenarios.

[0011] The technical solution of this invention is: a streaming video active interactive control method for assistive visually impaired scenarios, which configures an active interactive control plugin and embeds it into the inference process of a multimodal large model in the form of a decoupled plugin. Using streaming video from the user's first-person perspective as input, it enables active interaction between the user and the real-time environment, including:

[0012] 1) Constructing tasks to assist the blind: Constructing a hierarchical task system that includes active reminder tasks, visual question-answering tasks, and visually guided interactive tasks;

[0013] 2) Based on a multimodal large model, inference is performed on the input streaming video and user commands. The active interactive control plugin implements active interactive control, including:

[0014] 2.1) State maintenance: For proactive reminder tasks, continuously acquire streaming video frames from the user's first-person perspective and maintain a key-value cache (KV Cache) consisting of task definition information and historical video frame features. The task definition information is used to represent the current task content.

[0015] 2.2) Input Construction: The input sequence is constructed as: task definition information + video frame sequence token + functional suffix. The functional suffix is ​​dynamically switched between judgment mode and generation mode, and is divided into judgment suffix and generation suffix. The judgment suffix is ​​configured according to the task definition information and is used to guide the model to perceive and evaluate the environmental state represented by the current video frame. It determines whether to provide auxiliary feedback to the user according to the task definition information. When the judgment result is yes, the judgment suffix is ​​replaced with the generation suffix, guiding the model to enter the autoregressive decoding stage and generate blind-aid auxiliary feedback corresponding to the current environmental event.

[0016] 2.3) Trigger determination: Incremental context update is performed by combining the key-value cache (KV Cache), forward inference is performed on the input sequence of the current frame, and the current frame is determined to meet the trigger condition of the determination suffix based on the output probability distribution.

[0017] 2.4) Dynamic mode switching: If the triggering condition is met, the judgment suffix is ​​dynamically replaced with the generation suffix, and the autoregressive decoding stage is entered to generate blind assistance feedback. The blind assistance feedback is configured according to the response content in the blind assistance task. After the blind assistance feedback is completed, the key-value cache (KV Cache) is maintained, the functional suffix is ​​switched to the judgment suffix, and the judgment cycle of the next frame is entered, forming a continuous real-time interactive closed loop.

[0018] If the triggering condition is not met, maintain the key-value cache (KV Cache) and directly proceed to the incremental processing flow of the next video frame;

[0019] It enables dynamic switching between judgment mode and generation mode under continuous video input conditions, and completes environmental perception, trigger judgment and autoregressive auxiliary feedback output.

[0020] The tasks for assisting the blind include:

[0021] Active alert task: Used to automatically identify potential hazards or critical environmental information in continuous video streams and actively alert users when trigger conditions are met, while remaining silent at other times;

[0022] Visual question answering task: It is used to meet users' immediate visual information query needs. Users ask questions in natural language, and the model generates targeted answers by combining current and historical visual information.

[0023] Visually guided interaction: This is used to support users in completing specific target operations. The model first generates action instructions based on the current visual state to guide the user to adjust their position or camera view. After the user performs the corresponding operation, visual analysis is performed again and new guidance instructions are output. Through multiple rounds of interactive iteration, the model assists the user in completing the target task, forming a closed-loop interaction process.

[0024] Furthermore, post-training is performed on the large multimodal model, including...

[0025] First-person perspective videos were collected and labeled according to the visual aid task. Each labeled sample includes a time interval. and its corresponding natural language description For tasks requiring proactive reminders and visually guided interactions, Indicates the starting point for triggering an active reminder or interaction. This indicates the latest time when the current event ends or the user completes their action. Used to represent key information that needs to be conveyed to the user within a time interval, for visual question answering tasks. and At the same time, The model is used to represent responses and is trained for task adaptation based on constructed temporal interval-level assistive video data.

[0026] Furthermore, based on the task of assisting the blind, an automated dataset for assisting the blind is constructed using expert knowledge guidance for fine-tuning training of a multimodal large model: A publicly available first-person perspective video dataset is selected as the original data source; an open-vocabulary object detection model is used for object detection; frame-level semantic description generation is performed frame-by-frame using a multimodal large model, followed by annotation, to construct training data for assisting the blind scenario, including the following steps:

[0027] 3.1) Video frame extraction and temporal downsampling

[0028] A low-frequency sampling strategy is adopted to perform frame extraction on the video, thereby limiting the amount of video frames processed.

[0029] 3.2) Frame-level filtering based on object detection

[0030] A frame-level filtering mechanism based on object detection is established, and differentiated processing strategies are adopted for different types of assistive blindness tasks. The tasks are divided into the following two categories:

[0031] Open set task: For open domain scenarios, perform frame-by-frame annotation on the extracted frame sequence;

[0032] Goal-driven tasks: For tasks with a clear target shape, the frame sequence is first filtered using a target detection model, and only continuous frame segments with detected relevant targets are input into the multimodal large model for description generation;

[0033] 3.3) Automatic generation of frame-level semantic descriptions

[0034] The selected video frame sequence is input into the multimodal large model, which automatically generates frame-by-frame semantic description text to characterize the current environmental state, key targets and potential risk factors, forming initial frame-level annotation data;

[0035] 3.4) Event-level semantic merging and time interval generation

[0036] By using a large language model, similarity analysis and semantic aggregation are performed on the semantic descriptions of adjacent frames. For continuous description segments that meet the semantic similarity setting, the corresponding event-level descriptions are automatically merged and their start and end timestamps are determined simultaneously to construct structured time-series interval-level labeled samples.

[0037] Finally, based on the constructed temporal interval-level assistive video dialogue data, the multimodal large model is trained for streaming structure adaptation. During the training process, the video frame representation and text prompts are interleaved in chronological order to construct the input sequence to simulate real streaming video interaction scenarios. During training, the loss function is calculated only for the text output part to enhance the model's semantic alignment ability and active trigger judgment ability in continuous video environments.

[0038] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to realize the above-mentioned active interactive control method for streaming video in assistive visually impaired scenarios.

[0039] The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein when the at least one instruction or at least one program is executed, the above-described active interactive control method for streaming video in assistive visually impaired scenarios is implemented.

[0040] This invention addresses the open environment perception needs in assistive devices for the blind, designing a layered collaborative solution across three levels: task system construction, inference control mechanism, and data support system.

[0041] 1) At the task system level, a hierarchical task system is constructed for assistive visual scenarios. Based on the existing multimodal large model's open semantic understanding and multi-task processing capabilities, this invention provides a structured definition of tasks such as proactive reminders, visual question answering, and visually guided interaction. Through a unified task organization method and temporal interval-level annotation specifications, the model's multi-task capabilities can be systematically expressed and applied in assistive visual scenarios, thereby breaking through the adaptability bottleneck of traditional single-task visual algorithms in open and complex environments.

[0042] 2) At the inference control level, an active interactive control mechanism for streaming video was designed. Without changing the underlying model structure and parameter configuration, it can realize real-time active perception and trigger-based output of continuous video streams, thereby overcoming the limitations of existing multimodal models that rely on passive question answering and offline analysis.

[0043] 3) At the data support level, we propose an automated dataset construction method for assisting the blind based on expert knowledge guidance. This method is used to train the underlying model for task adaptation and behavior alignment, making up for the lack of dedicated data in the field of assisting the blind and enhancing the model's judgment accuracy and interaction stability in specific scenarios.

[0044] Through the synergistic effect of the above-mentioned task system construction, reasoning control mechanism design, and data support system construction, this invention forms a complete technical architecture for assistive blind scenarios, realizing real-time closed-loop interaction from continuous environmental perception and event triggering judgment to conversational auxiliary feedback output.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1) Achieving unified expression and collaborative modeling across multiple tasks, enhancing adaptability to open scenarios. Compared to traditional visual aids for the blind, which are typically designed separately for single closed-set tasks, this invention leverages the open semantic understanding and world knowledge modeling capabilities of multimodal large models to construct a hierarchical aid task system. Based on decoupled plug-in active interaction control plugs, active reminders, visual question answering, and visual guidance interactions are systematically organized and expressed within a unified framework. This overcomes the generalization bottleneck of traditional single-task algorithms in complex open environments, significantly enhancing the system's adaptability to diverse scenarios and dynamic environmental changes.

[0047] 2) Introducing a model architecture-independent active interactive control mechanism to achieve real-time triggered feedback. Unlike existing multimodal systems that rely on passive question answering or offline analysis, this invention achieves active perception and triggered output of continuous video streams through a dynamic switching mechanism between judgment and generation modes, without changing the underlying model structure and parameter configuration. Simultaneously, by combining KV Cache reuse and incremental context update strategies, it effectively reduces streaming inference latency and computational redundancy, ensuring real-time performance while maintaining system stability and resource utilization efficiency.

[0048] 3) Constructing an expert-guided automated data construction system significantly reduces annotation costs and improves scalability. This invention achieves high-quality construction of large-scale assistive video dialogue data through structured temporal interval annotation specifications and automated semantic generation processes, reducing reliance on purely manual frame-by-frame annotation. This data construction framework has good transferability and scalability, and can be extended to other streaming visual understanding and active interaction scenarios, providing efficient data support for model training in related fields. Attached Figure Description

[0049] Figure 1 This is an overall framework diagram of the method of the present invention.

[0050] Figure 2 This is an overview diagram of the blind-assistance task constructed in this invention.

[0051] Figure 3 This is a diagram of the automated assistive dataset construction method proposed in this invention.

[0052] Figure 4 This is a diagram of the active interactive control method proposed in this invention.

[0053] Figure 5 This is a comparison of the results of this invention with existing methods on manually labeled test sets. Detailed Implementation

[0054] This invention addresses the needs of assistive visually impaired tasks under streaming video input conditions. It constructs a hierarchical task system for assistive visually impaired scenarios, proposes an automated data construction method for assistive visually impaired tasks guided by expert knowledge, and designs an active interactive control mechanism for real-time environments. This enables multimodal large models to achieve multi-task collaborative expression, event triggering judgment, and conversational auxiliary feedback output in continuous video environments, thereby forming a real-time closed-loop interactive system suitable for assistive visually impaired scenarios. This enhances the active triggering interactive capabilities of multimodal large models in continuous visual environments, including the following aspects.

[0055] 1) Constructing a task system for assisting the blind

[0056] Based on first-person videos of real visually impaired users, the needs are abstracted to form a hierarchical task system including proactive reminder tasks, visual question-and-answer tasks, and visual guidance interactions, and a unified task organization method and time-series interval labeling standard are established.

[0057] 2) Active interactive control for streaming video

[0058] In the inference stage of the multimodal large model, without changing the original structure and parameter configuration of the multimodal large model, an active interactive control mechanism is introduced. An active interactive control plugin is embedded in the form of a decoupling plugin, which enables the model to dynamically switch between the judgment mode and the generation mode under continuous video input conditions, and complete environmental perception, trigger judgment and autoregressive auxiliary feedback output.

[0059] Embedding the interactive control plugin of this invention enables large models to perform proactive reminder tasks for the blind. To address the performance requirements of tasks in blind assistance scenarios, this invention further adapts and trains the multimodal large model, fine-tuning the training to further improve performance in blind assistance scenarios. The interactive control plugin and the adaptation training are two independent improvements in this invention; the plugin enables proactive output, while the training improves performance. This invention's plugin is training-free. As a preferred implementation, the large model can be trained for the scenario first, and then the plugin can be added, achieving a dual improvement in both functionality and performance.

[0060] The adaptation training for multimodal large model design in this invention includes:

[0061] 1) Construction of an automated assisted visual aid dataset guided by expert knowledge

[0062] Based on the aforementioned task system, first-person videos are annotated at the temporal interval level to construct structured video dialogue data for the visually impaired; the data construction process includes video frame extraction, target detection and filtering, frame-level semantic generation, and event-level semantic merging.

[0063] 2) Based on the data, perform task-adaptive training on the multimodal large model.

[0064] We use the constructed temporal interval-level video dialogue data for assistive visual impairment to train a multimodal large model for task adaptation. We construct the input sequence by interleaving video frame representations and text prompts in chronological order to enhance the model's semantic alignment and trigger judgment capabilities in continuous video scenes.

[0065] This invention, through the collaborative design of offline data construction and online inference control processes, enables the model to complete environmental perception, trigger determination, and conversational auxiliary feedback output under streaming video input conditions. This improves real-time response capabilities and interaction stability in assistive scenarios for the blind, constructing a complete technical closed loop from data support to inference control. The implementation of this invention is described in detail below.

[0066] 1) Constructing a task system for assisting the blind

[0067] To accurately characterize the core needs of visually impaired users, this invention analyzes first-person perspective video data posted by real blind users on public video platforms, and combines this with task settings in traditional visual aid research to gradually summarize and form a preliminary task framework.

[0068] Based on this, this invention constructed an initial version of the task definition system and annotation manual for assisting the blind, and organized 30 volunteers to conduct trial annotation operations on real videos of blind people. During the trial annotation process, the task boundaries, description specifications, and annotation granularity were revised and optimized multiple times based on the feedback from the annotators, ultimately forming a stable annotation specification system. Approximately 4,000 high-quality manually annotated data points were obtained in this stage. Given the scarcity of real video resources of blind people and the high reliability of manually annotated data, this invention uses this data as an evaluation set and conducts subsequent automated annotation work based on the established task system.

[0069] Ultimately, the task of assisting the blind is divided into the following three core sub-tasks: Figure 2 Examples of these tasks are shown:

[0070] Proactive Alert Task: This task automatically identifies potential hazards or critical environmental information in continuous video streams and proactively alerts the user when necessary, remaining silent at other times. Examples include obstacle warnings, pedestrian traffic light recognition, and tactile paving detection. Users do not need to actively initiate commands; the system autonomously triggers alerts based on environmental changes, achieving real-time perception and feedback of potential risks and useful information.

[0071] Visual question answering task: This task aims to meet the immediate visual information query needs of visually impaired users. Users ask questions in natural language, and the model generates targeted answers by combining current and historical visual information. Unlike traditional visual question answering tasks, visually impaired users cannot obtain real-time visual feedback, so their questions may be weakly related to or even unrelated to the current scene, and the image quality is often low. In such scenarios, the model is more prone to hallucinations. Therefore, this task places higher demands on multimodal semantic alignment and robust understanding capabilities.

[0072] Visually Guided Interaction: This task supports users in completing specific target operations. The model first generates action instructions based on the current visual state, guiding the user to adjust their position or camera view. After the user performs the corresponding operation, the system performs visual analysis again and outputs new guidance instructions. Through multiple rounds of interactive iterations, the system assists users in completing the target task, forming a closed-loop interaction process.

[0073] The three types of tasks mentioned above progress from simple to complex, are mutually complementary, and together constitute a complete assistive application system for the blind. Users can control the multimodal large model to switch between different tasks through voice commands and other means. For example, when a blind person is traveling, they can instruct the system to perform an active reminder task, so that the model can actively remind the blind person when it sees an obstacle, without having to ask the model in real time whether there is an obstacle in front of them. Blind users can also ask questions at any time, and the system will perform visual question-and-answer based on the received questions to complete the visual question-and-answer task. If a blind person issues a command to request the model to guide / assist in completing a task, the model will perform a corresponding visual guidance interaction task.

[0074] 2) An automated dataset construction method for assistive visual impairment based on expert knowledge guidance

[0075] Because existing general-purpose multimodal models lack dedicated training data for visually impaired users, especially in areas such as active perception, trigger judgment, and time-series interaction, supervision signals are insufficient, making it difficult for existing models to accurately understand the actual needs of visually impaired users and their environmental conditions. Therefore, this invention further proposes an automated method for constructing an assistive video dialogue dataset based on expert knowledge guidance. This method provides high-quality data support for the training and fine-tuning of multimodal models in assistive scenarios by constructing structured video dialogue data oriented towards assistive task systems.

[0076] This invention combines prior human knowledge with the capabilities of large-scale multimodal models to automate the construction of large-scale, high-quality video dialogue data for the visually impaired, while ensuring consistency in annotation, accuracy in temporal alignment, and standardization of semantic expression. The object detection step employs an open-vocabulary object detection model, and the frame-level semantic description generation step utilizes a multimodal model for frame-by-frame semantic generation, thereby improving the accuracy of object recognition and the consistency of semantic expression during data annotation. The method flow is as follows: Figure 3 As shown, the specific construction process is as follows:

[0077] 2.1) Data Sources and Collection Strategies

[0078] Due to the scarcity of real-world video data from visually impaired users, this invention primarily selects publicly available first-person perspective video datasets as the original data source, such as Ego4D and Sekai. Based on the aforementioned annotation system, these datasets are re-annotated and task-mapped to construct training data for assistive visually impaired scenarios. Here, task mapping refers to the specific task to which the generated annotation belongs; for example, does "There is a tree in front of you" belong to the "obstacle recognition" or "pedestrian traffic light recognition" task?

[0079] 2.2) Definition of Time-Series Interval-Level Labeling

[0080] To address the real-time and continuous nature of tasks requiring assistance for the blind, this invention employs a time-series interval-level annotation format, with each annotated sample comprising a time interval. and its corresponding natural language description For tasks requiring proactive reminders and visually guided interactions, Indicates the starting point for triggering an active reminder or interaction. This indicates the latest time when the current event ends or the user completes their action. Used to represent key information that needs to be conveyed to the user within a time interval, Figure 2 For example, the first proactive reminder label in the text:

[0081] This indicates the earliest time when it is appropriate to trigger an active reminder or interaction. At 59.8 seconds, the model confirms that the trees ahead will affect the blind person's movement. For earlier times, such as 56 seconds, when the blind person may still be far away from the tree and it is uncertain whether the tree will affect the blind person, premature prompts would be redundant.

[0082] This indicates the latest time when the current event has ended or when the user can complete the avoidance or action. At 62.4 seconds, this is the latest time when a blind person can avoid the tree in front of them. If the model provides the alert later than this time, the blind person may have already crashed into the tree.

[0083] This is used to represent the key information that needs to be conveyed to visually impaired users within this time interval. Between 59.8 and 62.4 seconds, the model needs to remind the blind person, "There is a tree ahead, please go around to the left."

[0084] Figure 2 The text highlighted in red indicates key information in the natural language description.

[0085] Proactive reminders and visually guided interactions require time interval annotations, while visual question answering only needs a single timestamp. and For example, if a blind person asks "Where is the hospital gate?" at the 29th second, in actual annotation, we set both the start and end times to 29 seconds for this annotation. This is used to represent the model's ability to respond, for example, if a question is asked at the 29th second of a video stream, the model needs to respond at that time.

[0086] 2.3) Video frame extraction and temporal downsampling

[0087] Considering the real-time requirements of assistive visual impairment tasks and the computational resource consumption during annotation, this invention adopts a unified low-frequency sampling strategy, processing the original video at 1 fps or 2 fps to avoid excessively large models for each frame. For example, some videos originally run at 30 fps, and excessively large models for each frame result in high model overhead and slow annotation speed. Secondly, video features are relatively redundant, and adjacent frames may be quite similar, so it is not necessary to annotate every frame. The specific frame extraction can be determined according to the scenario. By reasonably constraining the sampling frequency, while ensuring that key environmental change events are continuously traceable in the time dimension, the number of redundant frame data is effectively reduced, lowering the computational cost of subsequent multimodal model annotation, thereby achieving efficient construction of large-scale labeled data.

[0088] 2.4) Frame-level filtering based on object detection

[0089] To avoid the high computational resource consumption caused by large-scale frame-by-frame annotation of all extracted frames, this invention introduces a frame-level filtering mechanism based on object detection, and adopts differentiated processing strategies for different types of blindness assistance tasks, dividing the tasks into the following two categories:

[0090] Open set tasks: such as obstacle recognition, public facility detection, road condition analysis and other open domain scenarios, the target categories are difficult to exhaustively list in advance and cannot be covered by a fixed label system. Therefore, frame-by-frame labeling is performed directly on the extracted frame sequence.

[0091] Target-driven tasks: such as pedestrian traffic lights, tactile paving, zebra crossings, etc., which have clear target shapes, firstly, the frame sequence is filtered using a target detection model (such as MM-Grounding-Dino), and only continuous frame segments with detected relevant targets are input into the multimodal large model for description generation.

[0092] like Figure 3 As shown, Figure 3 The "(optional)" refers to the fact that for tasks involving specific targets, such as zebra crossings, pedestrian traffic lights, and stairs, an open-vocabulary target detection model can be used to detect them from video frames first, and then a large model can be used to annotate the corresponding segments, avoiding the waste of resources caused by using an excessively large model for each frame. For targets that cannot be predefined into fixed categories, such as "obstacles" in a general sense, i.e. any unknown object that may cause potential harm to visually impaired people, a multimodal large model can be used directly for frame-by-frame annotation to ensure the comprehensiveness and safety of scene analysis.

[0093] 2.5) Automatic generation of frame-level semantic descriptions

[0094] The selected video frame sequence is input into a multimodal large model, which automatically generates frame-by-frame semantic description text to characterize the current environmental state, key targets, and potential risk factors, thus forming initial frame-level labeled data. As an example, this uses a high-performance multimodal large model to construct the aided visual data, such as GPT-5, Gemini-3-Pro, or a high-performance open-source model, such as Qwen3.5-397B-A17B.

[0095] 2.6) Event-level semantic merging and time interval generation

[0096] Given the strong temporal continuity and semantic redundancy of frame-by-frame descriptions, this invention further introduces a large language model to perform similarity analysis and semantic aggregation on adjacent frame descriptions. For continuous description fragments with high semantic consistency, corresponding event-level descriptions are automatically merged, and their start and end timestamps are determined simultaneously, thereby constructing structured temporal interval-level labeled samples.

[0097] 3) Based on the data, perform task-adaptive training on the multimodal large model.

[0098] Based on the constructed temporal interval-level assistive video dialogue data, a multimodal large model is trained using streaming structure adaptation. During training, video frame representations and text prompts are interleaved in chronological order to construct the input sequence, simulating real-world streaming video interaction scenarios. The loss function is calculated only for the text output during training, thereby enhancing the model's semantic alignment ability and proactive triggering judgment ability in continuous video environments.

[0099] 4) Active interactive control mechanism for streaming video

[0100] To address the problems of existing multimodal large-scale models relying on passive question answering, lacking active triggering capabilities, and being difficult to adapt to real-time streaming scenarios during video understanding, this invention proposes an active interactive control mechanism for streaming video. This active interactive control mechanism is decoupled from the underlying multimodal large-scale model's network structure and can be embedded as an independent plug-in module into different model architectures for integration, thereby enhancing the system's versatility and scalability. Figure 4 As shown, this mechanism extends the reasoning process of the multimodal large model while retaining its original structure and parameter configuration, transforming the model from the traditional passive interaction paradigm of "query-response" to an active interaction mode of "perception-trigger-feedback" for blind assistance scenarios.

[0101] The mechanism is embedded into the model inference process as a decoupled plug-in, independent of specific model structures or network implementations. Under continuous video input conditions, it enables automatic perception, trigger determination, and decoding scheduling control of critical environmental events. Through this mechanism, the system can adaptively determine the activation timing of generated tasks based on dynamic changes in the environmental state during real-time video stream input. In dangerous or critical scenarios, it proactively outputs auxiliary commands to visually impaired users, while maintaining silent operation in normal environments, thus ensuring both real-time performance and computational efficiency. Its specific implementation method is as follows.

[0102] 4.1) Construction and segmentation of dynamic input sequences

[0103] During model inference, the input sequence is uniformly constructed as: "Task Definition Information + Video Frame Sequence Token + Functional Suffix". Here, the task definition information refers to the content of the current task, such as subtasks within an active reminder task, like obstacle reminders or target finding. Figure 4 The corresponding part is the task template, and the video frame sequence is Frame t-1, Frame t, etc.; the functional suffix dynamically switches between the following two modes according to the real-time running status of the system:

[0104] Decision pattern – Decision suffix: Used to guide the model to perceive and evaluate the environmental state represented by the current video frame in order to determine whether it is necessary to provide auxiliary feedback to visually impaired users. For example, the decision suffix can be designed as "Does the current visual condition meet the task definition? If it does, output A; otherwise, output B", thus transforming the active output requirement into a modelable discrete binary classification decision problem;

[0105] Generation Mode - Generation Suffix: When the judgment result meets the triggering condition (triggers signal A), the system replaces the judgment suffix with the generation guidance mark (such as "Response:") and enters the autoregressive decoding stage, guiding the model to generate blind assistance instructions corresponding to the current environmental event.

[0106] This dynamic input construction mechanism enables the model to flexibly switch between decision mode and generation mode within a unified reasoning framework.

[0107] 4.2) Incremental Context Update

[0108] To meet the real-time requirements of low latency and continuous perception in assistive devices for the visually impaired, the active interaction control plugin employs an incremental context update mechanism to avoid redundant calculations of historical input sequences. During operation, its core processes include:

[0109] KV Cache State Maintenance: The system continuously maintains a basic context key-value cache (KV Cache) consisting of "task definition information + historical video frame features", where video frame features are visual features.

[0110] Single-frame incremental calculation: When a new video frame arrives, the system performs forward calculation only once on the visual features corresponding to that frame, and then appends the generated KV value to the end of the basic context sequence, i.e., the end of the basic context key-value cache, in an incremental manner.

[0111] Temporal aggregation perception: Relying on the self-attention mechanism of Transformer, the current model state is associated with and utilizes historical visual information during a single forward propagation, thereby achieving environmental understanding and proactive interaction determination with temporal consistency.

[0112] By using an incremental context update mechanism, computational redundancy and inference latency are effectively reduced while ensuring long-term time-series modeling capabilities.

[0113] 4.3) Trigger detection and decoding mode switching

[0114] In decision mode, the system constructs a complete input sequence consisting of "task definition information + visual representation of the video frame at the current moment + decision suffix", and performs one forward inference to extract the hidden state at the end of the sequence. The probability distribution of the next lexical term is obtained through language modeling head mapping. The switching process is as follows:

[0115] Trigger determination: The calculation model outputs a flag indicating the environmental state represented by the current video frame. Predicted probability When satisfied When the system determines that the current environment meets the triggering conditions for the determination suffix, it terminates the determination mode and switches to the generation mode. As an example, a preset trigger threshold is used. Values .

[0116] Decoding Scheduling and State Switching: In generation mode, the system appends a generation suffix to the basic context, initiates the autoregressive decoding process, and generates the corresponding assistive instructions for the blind. Once decoding is complete, the system releases the temporary state generated during the generation phase and restores to the basic context state.

[0117] Looping operation: If the triggering condition is not met, the system maintains the current KV Cache and directly enters the incremental processing flow of the next video frame; if the trigger is successful, it enters the next judgment cycle after completing the output. This forms a stable and continuous real-time interactive closed loop, enabling the system to be in a low-power continuous monitoring state most of the time, triggering the high-load generation process only when necessary.

[0118] To verify the effectiveness of the active interactive control mechanism for streaming video described in this invention, comparative experiments were conducted on a manually annotated dataset of assistive devices for the visually impaired. The proposed active interactive control mechanism was embedded as an independent plug-in module into the general multimodal large-scale models Qwen2.5VL-7B and InternVL3.5-8B, respectively. Performance was compared with existing active interactive streaming models VideoLLM-Online-8B and LiveCC. The experimental results are as follows: Figure 5 As shown in the figure. Experimental results show that the general model applying the scheme of this invention exhibits an overwhelming advantage in the "active output" metric, which measures the level of active perception of the system. The InternVL3.5-8B, after integrating this mechanism, achieves an active output score of 18.5, which is several times or even tens of times higher than the dedicated models LiveCC (3.1) and VideoLLM-Online (0.2) specifically designed for streaming interactive training. This fully verifies that the "decision / generation dual-mode dynamic switching" mechanism proposed in this invention can effectively activate the model's active response capability to the streaming environment. Experimental data further proves that the active interactive control mechanism proposed in this invention has excellent architectural adaptability. By applying this mechanism to different series of general large models, such as Qwen2.5VL and InternVL series, significant and robust performance gains have been achieved. This confirms that the dynamic suffix switching and KV Cache incremental update mechanism described in this invention does not depend on specific model underlying parameters or specific pre-training tasks, and can be widely adapted to various multimodal large model architectures as a general control protocol, demonstrating extremely high industrial application value and technology transferability.

[0119] This invention is based on a computer program. Based on this, the invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the aforementioned active interactive control method for streaming video in assistive scenarios for the visually impaired. The invention also provides a computer-readable storage medium storing at least one instruction or at least one program segment, which, when executed, implements the aforementioned active interactive control method for streaming video in assistive scenarios for the visually impaired. The instructions or program code used to implement the method of this invention can be written in any combination of one or more programming languages. The instructions or program code can be executed entirely on a data processor, partially on a processor, partially on a remote device as a standalone software package, or entirely on a remote device or server.

Claims

1. A streaming video active interactive control method for assistive visually impaired scenarios, characterized by: Configure an active interaction control plugin, embedding it as a decoupled plugin into the inference process of a multimodal large model. Using streaming video from the user's first-person perspective as input, it enables active interaction between the user and the real-time environment, including: 1) Constructing tasks to assist the blind: Constructing a hierarchical task system that includes active reminder tasks, visual question-answering tasks, and visually guided interactive tasks; 2) Based on a multimodal large model, inference is performed on the input streaming video and user commands. The active interactive control plugin implements active interactive control, including: 2.1) State maintenance: For proactive reminder tasks, continuously acquire streaming video frames from the user's first-person perspective and maintain a key-value cache (KV Cache) consisting of task definition information and historical video frame features. The task definition information is used to represent the current task content. 2.2) Input Construction: The input sequence is constructed as: task definition information + video frame sequence token + functional suffix. The functional suffix is ​​dynamically switched between judgment mode and generation mode, and is divided into judgment suffix and generation suffix. The judgment suffix is ​​configured according to the task definition information and is used to guide the model to perceive and evaluate the environmental state represented by the current video frame. It determines whether to provide auxiliary feedback to the user according to the task definition information. When the judgment result is yes, the judgment suffix is ​​replaced with the generation suffix, guiding the model to enter the autoregressive decoding stage and generate blind-aid auxiliary feedback corresponding to the current environmental event. 2.3) Triggering determination: Incremental context update is performed in conjunction with KV Cache. Forward inference is performed on the input sequence of the current frame. The current frame is determined to meet the triggering condition of the determination suffix based on the output probability distribution. 2.4) Dynamic mode switching: If the triggering condition is met, the judgment suffix is ​​dynamically replaced with the generation suffix, and the autoregressive decoding stage is entered to generate blind assistance feedback. The blind assistance feedback is configured according to the response content in the blind assistance task. After the blind assistance feedback output is completed, the KV Cache is maintained, the functional suffix is ​​switched to the judgment suffix, and the judgment cycle of the next frame is entered, forming a continuous real-time interactive closed loop. If the triggering condition is not met, maintain the KV Cache and directly proceed to the incremental processing flow of the next video frame; It enables dynamic switching between judgment mode and generation mode under continuous video input conditions, and completes environmental perception, trigger judgment and autoregressive auxiliary feedback output.

2. The active interactive control method for streaming video in assistive visually impaired scenarios according to claim 1, characterized in that: Assistance tasks for the blind include: Active alert task: Used to automatically identify potential hazards or critical environmental information in continuous video streams and actively alert users when trigger conditions are met, while remaining silent at other times; Visual question answering task: It is used to meet users' immediate visual information query needs. Users ask questions in natural language, and the model generates targeted answers by combining current and historical visual information. Visually guided interaction: This is used to support users in completing specific target operations. The model first generates action instructions based on the current visual state to guide the user to adjust their position or camera view. After the user performs the corresponding operation, visual analysis is performed again and new guidance instructions are output. Through multiple rounds of interactive iteration, the model assists the user in completing the target task, forming a closed-loop interaction process.

3. The active interactive control method for streaming video in assistive visually impaired scenarios according to claim 1 or 2, characterized in that: Post-training of large multimodal models, including First-person perspective videos were collected and labeled according to the visual aid task. Each labeled sample includes a time interval. and its corresponding natural language description For tasks requiring proactive reminders and visually guided interactions, Indicates the starting point for triggering an active reminder or interaction. This indicates the latest time when the current event ends or the user completes their action. Used to represent key information that needs to be conveyed to the user within a time interval, for visual question answering tasks. and At the same time, The model is used to represent responses and is trained for task adaptation based on constructed temporal interval-level assistive video data.

4. The active interactive control method for streaming video in assistive visually impaired scenarios according to claim 3, characterized in that: Based on the task of assisting the blind, an automated dataset for assisting the blind is constructed using expert knowledge guidance for fine-tuning training of a multimodal large model. This involves selecting a publicly available first-person perspective video dataset as the original data source, employing an open-vocabulary object detection model for object detection, and using a multimodal large model to generate frame-by-frame semantic descriptions. After annotation, training data for assisting the blind is constructed, including the following steps: 3.1) Video frame extraction and temporal downsampling A low-frequency sampling strategy is adopted to perform frame extraction on the video, thereby limiting the amount of video frames processed. 3.2) Frame-level filtering based on object detection A frame-level filtering mechanism based on object detection is established, and differentiated processing strategies are adopted for different types of assistive blindness tasks. The tasks are divided into the following two categories: Open set task: For open domain scenarios, perform frame-by-frame annotation on the extracted frame sequence; Goal-driven tasks: For tasks with a clear target shape, the frame sequence is first filtered using a target detection model, and only continuous frame segments with detected relevant targets are input into the multimodal large model for description generation; 3.3) Automatic generation of frame-level semantic descriptions The selected video frame sequence is input into the multimodal large model, which automatically generates frame-by-frame semantic description text to characterize the environmental state, key targets and potential risk factors at the current moment, forming initial frame-level annotation data; 3.4) Event-level semantic merging and time interval generation By using a large language model, similarity analysis and semantic aggregation are performed on the semantic descriptions of adjacent frames. For continuous description segments that meet the similarity setting, the corresponding event-level descriptions are automatically merged and their start and end timestamps are determined simultaneously to construct structured time-series interval-level labeled samples. Finally, based on the constructed temporal interval-level assistive video dialogue data, the multimodal large model is trained for streaming structure adaptation. During the training process, the video frame representation and text prompts are interleaved in chronological order to construct the input sequence to simulate real streaming video interaction scenarios. During training, the loss function is calculated only for the text output part to enhance the model's semantic alignment ability and active trigger judgment ability in continuous video environments.

5. The active interactive control method for streaming video in assistive visually impaired scenarios according to claim 1, characterized in that: The configuration of the active interaction control plugin includes: Dynamic input sequence construction: The input sequence is constructed as "task definition information + video frame sequence token + functional suffix". The functional suffix dynamically switches between judgment mode and generation mode according to the real-time running status. An incremental context update mechanism is adopted, including: KV Cache State Maintenance: Continuously maintain the basic context key-value cache (KV Cache) consisting of "task definition information + historical video frame representation"; Single-frame incremental computation: When a new video frame arrives, only one forward computation is performed on the visual features corresponding to that frame, and the generated KV value is incrementally concatenated to the end of the basic context key-value cache; Temporal aggregation perception: Relying on the Transformer self-attention mechanism, the current model state is associated with and utilizes historical visual feature information during a single forward propagation, thereby achieving environmental understanding and proactive interaction determination with temporal consistency.

6. The active interactive control method for streaming video in assistive visually impaired scenarios according to claim 1 or 5, characterized in that it is active... The trigger judgment and dynamic mode switching in the interactive control plugin are as follows: In decision mode, the input sequence is composed of "task definition information + visual representation of the current video frame + decision suffix", and forward inference is performed once to extract the hidden state at the end of the KV Cache sequence. The probability distribution of the next word morpheme is obtained through language modeling head mapping; Trigger determination: Outputting a flag for the environmental state represented by the current video frame based on the language modeling head calculation model. Predicted probability When satisfied When the current environment state meets the triggering condition, the determination mode is terminated, and the process switches to the generation mode. The preset trigger threshold; Decoding scheduling and state switching: In generation mode, a generation suffix is ​​appended to the basic context to start the autoregressive decoding process and generate the corresponding blind assistance instructions. When decoding is completed, the temporary state generated in the generation mode phase is released and the system is restored to the basic context state. If the triggering condition is not met, the current KV Cache remains unchanged, and the incremental processing flow of the next video frame is directly entered; if the trigger is successful, the next judgment cycle is entered after the output is completed, thus forming a continuous real-time interactive closed loop.

7. An electronic device, characterized in that: The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the streaming video active interactive control method for assistive visually impaired scenarios as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that... The storage medium stores at least one instruction or at least one program segment. When the at least one instruction or at least one program segment is executed, it implements the streaming video active interactive control method for assistive visually impaired scenarios as described in any one of claims 1-6.

Images