Intelligent agent multi-modal barrier-free interaction method and device, intelligent terminal and medium

Abstract

The application discloses an agent multi-modal barrier-free interaction method and device, an intelligent terminal and a medium, relates to the technical field of AI human-computer interaction, and comprises the following steps: when it is detected that a user enters the sensing range of the intelligent terminal, collecting user biological feature information; comparing the collected biological feature information with prestored user identity features to determine the current user identity; calling the ability feature parameters suitable for the current user from a preset barrier-free user portrait library; dynamically reconstructing the presentation mode of the interface elements of the intelligent terminal and switching to the operation mode suitable for the current user; acquiring multi-modal heterogeneous input signals of the user in the adaptive operation mode, analyzing the user operation intention, E, responding to the analyzed user operation intention to execute corresponding services. The application can adapt to users of various ages, various physical or cognitive abilities through dynamically adaptive interaction modes, AI scene sensing and natural interaction, and provide better and more convenient interaction services for each user.

Description

Technical Field

[0001] This invention relates to the field of AI human-computer interaction technology, and in particular to a method, device, smart terminal, and storage medium for multimodal barrier-free interaction of intelligent agents based on a smart terminal. Background Technology

[0002] With the development of artificial intelligence and the continuous improvement of people's living standards, the use of various smart terminals such as smart TVs is becoming more and more widespread, and smart TVs have become an indispensable leisure and entertainment tool in people's lives.

[0003] As the information and entertainment center of the home, television's traditional interaction methods (such as remote controls and complex menus) pose usage barriers for many users: 1) For the elderly: declining eyesight makes it difficult to see small fonts and complex interfaces; hearing loss affects audio content reception; and the complex operating logic of smart TV remote controls presents learning difficulties. 2) For visually impaired users: the graphical user interface (GUI) that relies entirely on visual information makes it impossible for them to use the television independently. 3) For hearing-impaired users: audio information from television programs and system prompts cannot be effectively received. 4) For users with mobility impairments: fine motor operation of remote control buttons or gesture interaction is difficult. 5) For children: limited literacy makes it impossible to understand text menus; and their logical thinking abilities are not yet fully developed, making it difficult to navigate complex interfaces.

[0004] Existing solutions are mostly fragmented auxiliary functions, lacking a universal intelligent interaction system that is user-centric and deeply integrated with AI.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method, device, smart terminal and storage medium for multimodal barrier-free interaction of intelligent agents based on smart terminals, addressing the deficiencies of the prior art. The present invention provides a TV intelligent agent multimodal barrier-free interaction system and method for universal access; through dynamically adaptable interaction modalities, AI context perception and natural interaction, it can adapt to users of various ages and with different physical or cognitive abilities, providing better and more convenient interactive services for each user.

[0007] The technical solution adopted by this invention to solve the problem is as follows: A method for multimodal barrier-free interaction of intelligent agents based on smart terminals, comprising: A. When a user is detected entering the sensing range of the smart terminal, the user's biometric information is collected; the collected biometric information is compared with the pre-stored user identity features to determine the current user's identity; the smart terminal is awakened by triggering methods such as voice wake-up, gesture wake-up, and / or external auxiliary devices. B. Based on the determined current user identity, retrieve the capability feature parameters that are suitable for the current user from the preset accessibility user profile database; C. Based on the aforementioned capability characteristic parameters, dynamically reconstruct the presentation mode of the smart terminal interface elements and switch to an operation mode adapted to the current user; enable multimodal guidance feedback, and open corresponding alternative perception channels according to the user's capability characteristics, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. D. Acquire multimodal heterogeneous input signals from the user in the adapted operation mode, the input signals including voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; fuse and understand the multimodal input signals to obtain the user's operation intention; E. Response parsing the user's operation intent to execute the corresponding business; and perform perception enhancement processing on media content: real-time analysis of video images to generate visual narration audio, recognition of audio signals to generate dialogue subtitles and environmental sound effects text descriptions, and output through established alternative perception channels to compensate for the user's missing perception information.

[0008] The aforementioned intelligent agent multimodal barrier-free interaction method based on intelligent terminals further includes, after step E: F. Record the data of this interaction process, iteratively optimize the user's corresponding recognition model based on successful / failed interaction samples, and update the user preference configuration in the accessibility user profile database.

[0009] The aforementioned intelligent agent multimodal barrier-free interaction method based on intelligent terminals further includes the following steps before step A: Pre-collect specific ability characteristics of different users, and set corresponding ability characteristic parameters for users with different specific ability characteristics; The specific ability characteristics of different users, along with their corresponding ability characteristic parameters, including barrier type, perception channel preference, and interface parameters, are stored in the accessibility user profile database.

[0010] The aforementioned intelligent agent multimodal barrier-free interaction method based on intelligent terminals, wherein the step of collecting user biometric information when a user is detected entering the sensing range of the intelligent terminal includes: The identity of the current user can be determined seamlessly by using facial recognition via a camera or voiceprint recognition via a voice perception module with a microphone array. The voice perception module uses a microphone array and incorporates an optimized recognition model for the unclear speech of the elderly and the voices of children.

[0011] The aforementioned intelligent agent multimodal barrier-free interaction method based on intelligent terminals, wherein the steps of dynamically reconstructing the presentation mode of intelligent terminal interface elements according to the capability feature parameters and switching to an operation mode adapted to the current user; enabling multimodal guided feedback and opening corresponding alternative perception channels according to the user's capability characteristics, including interface voice broadcast reading, global real-time subtitles, visual cues, and outputting guided interaction prompts, further include: When the current user is a visually impaired user, the control engine shuts down visual rendering that consumes more energy than the predetermined energy value, switches to a pure audio user interface, and converts the menu structure and option status into a voice broadcast sequence. When the current user is a hearing-impaired user, the control engine overlays a real-time subtitle layer on the video layer and converts the system prompts into flashing or color changes at the edge of the screen; and / or converts system prompts, call reminders, and operation confirmation information into vibration and tactile signals through communication with the remote control or smart bracelet. When the current user is an elderly person, the control engine simplifies the interface hierarchy and enlarges the click hotspot to a predetermined value; When the current user is someone who has difficulty controlling fine motor skills in their upper limbs, head tracking is used to select the target instead of an input device; the interaction parameters are dynamically adjusted through software algorithms: the click hotspot is enlarged and the cursor movement speed is reduced to a preset speed; The virtual assistant greets you with both voice and text and provides multimodal guidance.

[0012] The aforementioned intelligent agent multimodal barrier-free interaction method based on intelligent terminals, wherein the step of acquiring multimodal heterogeneous input signals of the user in an adapted operation mode, the input signals including voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; and fusing and understanding the multimodal input signals to parse the user's operation intention, further includes: By using a weighted fusion algorithm, the screen space coordinates of the user's gesture location are correlated and parsed with the speech semantics to obtain the user's operation intention, which is then used to achieve fine-grained control.

[0013] The aforementioned intelligent terminal-based multimodal barrier-free interaction method for intelligent agents further includes the following steps: An audio activity detector is pre-deployed in the audio decoding layer of the smart terminal to analyze the loudness and spectral characteristics of the audio stream of film and television content in real time. Two types of time windows are identified and marked: an absolutely silent window where the audio loudness meets the first preset condition, and a relatively low-activity window where the speech pauses or background music meet the second preset condition. Each detected window is stamped with a timestamp and confidence label to build a dynamic, insertable time slot queue. The audio activity detection module performs real-time analysis of the audio content to identify natural auditory gaps in film and television narratives; A navigation voice buffer is established to temporarily store the interface information to be broadcast in a queue without immediately triggering voice output. The natural auditory gap is used as the transmission time slot for navigation voice by an intelligent scheduler. When the natural auditory gap is reached, the navigation text to be broadcast is extracted from the navigation voice buffer and transmitted. The navigation text is also subjected to semantic compression and speed-changing processing to shorten the duration of a single voice message. Before the navigation voice broadcast, a tactile pre-prompt is issued via the user's remote control or smart bracelet at a predetermined time, using cross-modal attention guidance effect to redirect the user's attention to the system navigation; for urgent information that must be responded to immediately, channel switching is achieved through loudness contrast and fade-in / fade-out technology.

[0014] A smart terminal-based intelligent agent multimodal barrier-free interaction device, wherein the device includes: The multimodal perception module is used to collect the user's biometric information when a user is detected entering the perception range of the smart terminal; compare the collected biometric information with pre-stored user identity features to determine the current user's identity; the smart terminal is awakened by triggering methods such as voice wake-up, gesture wake-up and / or wake-up by external auxiliary devices; The capability feature parameter retrieval module is used to retrieve capability feature parameters that are suitable for the current user from a preset accessibility user profile database based on the determined current user identity. The adaptation mode switching module is used to dynamically reconstruct the presentation mode of the smart terminal interface elements according to the capability feature parameters, and switch to the operation mode adapted to the current user; enable multimodal guidance feedback, and open the corresponding alternative perception channels according to the user's capability features, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. A multimodal interaction module is used to acquire multimodal heterogeneous input signals from the user in the adapted operation mode. The input signals include voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices. The module fuses and understands the multimodal input signals to parse the user's operation intention. The multi-module response and execution module is used to respond to the parsed user operation intent and execute the corresponding business; and to perform perception enhancement processing on media content: real-time analysis of video images to generate visual narration audio, recognition of audio signals to generate dialogue subtitles and environmental sound effects text descriptions, and output through established alternative perception channels to compensate for the user's missing perception information. The data optimization module records data from the interaction process, iteratively optimizes the user's recognition model based on successful / failed interaction samples, and updates the user preference configuration in the accessibility user profile database.

[0015] A smart terminal includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs comprising the method for performing any one of the methods.

[0016] A computer-readable storage medium wherein, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described herein.

[0017] The present invention provides a method, device, smart terminal, and storage medium for multimodal barrier-free interaction of intelligent agents based on smart terminals. Compared with the prior art, the present invention has the following significant advantages: 1) This invention can achieve truly inclusive functionality, fundamentally solving the problem of special groups using smart TVs, embodying the inclusive design philosophy of designing for people, and has profound social value.

[0018] 2) With the dynamic adaptation and fusion function of interactive modal, this invention is no longer a fixed "one mode", but dynamically selects the optimal combination of interaction according to the user's identity and ability, realizing a barrier-free experience of "a thousand people, a thousand faces".

[0019] 3) Achieving natural and proactive intelligence: This invention proactively provides information enhancement (such as visual narration and ambient sound captions) through AI perception and content understanding, going beyond the scope of passive response and providing a deeper level of access services.

[0020] 4) Reduce learning costs and user anxiety. In the specific implementation of this invention, users do not need to learn complicated operations and can interact with the TV in the most natural and instinctive way (speaking, gestures), which greatly improves ease of use and user confidence.

[0021] 5) It has systematic and integrated functions. This invention integrates scattered accessibility functions into a collaborative intelligent system, providing a unified and seamless terminal experience. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the intelligent agent multimodal barrier-free interaction method based on a smart terminal provided in Embodiment 1 of the present invention.

[0023] Figure 2 The present invention provides a schematic diagram of an embodiment of a smart agent multimodal barrier-free interaction device based on a smart terminal.

[0024] Figure 3 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0026] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0027] Existing smart terminals, such as smart TVs, which serve as home information and entertainment centers, generally adopt a single interaction paradigm of "remote control + graphical interface." This paradigm presupposes that users possess normal visual, auditory, and fine motor skills. However, this assumption is severely mismatched with the actual capabilities of diverse user groups, specifically manifesting in inconvenience in the following application scenarios: For example, elderly people with declining vision and visually impaired users cannot effectively perceive small fonts and complex menus in the graphical interface; elderly people with hearing loss and hearing-impaired users cannot effectively receive audio content and system prompts; users with mobility impairments who have difficulty moving their upper limbs face physical difficulties in finely operating the remote control or performing complex gestures; and children, with their limited literacy and underdeveloped logical thinking abilities, find it difficult to navigate complex text-based and hierarchical interfaces.

[0028] Existing solutions only provide scattered auxiliary functions, such as simple font magnification or monochrome mode. They lack a universal intelligent interaction system that is centered on individual user capabilities and can dynamically adjust the interaction mode according to the user's perception and operation characteristics. This results in inconvenience for specific groups of users to operate smart TVs, and the inability to adapt to specific user groups and access digital TV services equally, creating a significant digital divide.

[0029] To address this, this application proposes a method for multimodal barrier-free interaction of intelligent agents based on smart terminals, such as... Figure 1 As shown in Embodiment 1 of the present invention, a method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals includes the following steps: S100. When a user is detected entering the sensing range of the smart terminal, the user's biometric information is collected; the collected biometric information is compared with the pre-stored user identity features to determine the current user's identity; the smart terminal receives a wake-up trigger via voice wake-up, gesture wake-up, and / or external auxiliary device wake-up. In this embodiment of the invention, the smart terminal is described using a smart TV as an example. In a specific implementation of the invention, a multimodal perception layer is pre-installed on the smart TV, including: Visual perception modules, such as cameras installed on smart TVs, are used for gesture recognition, lip reading, and user identification and basic status (such as whether they are in front of the screen).

[0030] Voice perception modules, such as high-performance microphone arrays installed on smart TVs, support far-field voice wake-up and recognition, and optimize recognition models for the slurred speech of the elderly and the voices of children.

[0031] Auxiliary input module: compatible with external switches, head trackers, eye trackers and other special input devices.

[0032] In addition, before implementing this invention, it is necessary to build an accessible user profile database to store the specific needs and preferences of different users, such as user A: visually impaired, prefers voice interaction and extra-large fonts, and user B: hearing impaired, requires real-time subtitles.

[0033] In this invention, the intelligent terminal sensing range refers to the spatial area centered on the intelligent terminal (such as a smart TV or smart screen) and equipped with various sensors, such as the aforementioned visual sensing module and voice sensing module, capable of effectively detecting the user's presence. Specifically, through the coordinated work of devices such as cameras, microphone arrays, human infrared sensors, and UWB ultra-wideband positioning, the terminal can automatically sense the user's presence and trigger subsequent interaction processes when the user approaches, completing the pre-processing of "sensing-wake-interaction" without the user's active operation. For example, the front-facing camera of a smart TV can detect users within a range of several meters in front, and the microphone array can pick up sound activity in the room.

[0034] In this embodiment, biometric information refers to a set of physiological and behavioral characteristic data used to uniquely identify or describe an individual user. Specifically, it can be obtained through methods such as facial images captured by a camera, iris features, and voiceprint features captured by a microphone. This provides the ability to perform contactless and non-invasive identification and verification of user identity, completing the identity determination without disturbing the user. For example, facial images can be considered a typical form of biometric information.

[0035] In this embodiment, the user identity features refer to identity template data that is pre-registered and stored in the system for comparison and matching with real-time collected biometric information. It is composed of facial and voiceprint feature templates registered and entered by the user upon first use, and has the ability to compare similarity with real-time collected data to determine the user's identity. This supports a mechanism for identity determination that allows for one-time registration and multiple seamless identifications.

[0036] Voice wake-up, gesture wake-up, and external assistive device wake-up refer to three independent yet combinable methods for triggering smart terminal wake-up. Voice wake-up involves the terminal's microphone continuously listening for a specific wake word, activating interaction when the user speaks the wake word. Gesture wake-up involves the terminal's camera recognizing a specific gesture (such as waving) and activating interaction. External assistive device wake-up involves the user actively triggering wake-up via a connected assistive device (such as a barrier-free button, wheelchair control lever, or wearable device). These three methods complement each other, covering the interactive capabilities of different user groups, and can select the most suitable wake-up method based on the user's physical condition and preferences.

[0037] In specific implementation of step S100, when a user is detected within the smart TV's sensing range (e.g., within 8 meters), the system begins collecting the user's biometric information. In this embodiment, the system no longer relies on the user manually turning on the power or searching for the remote control. Instead, the smart terminal's sensors actively sense the user's entry into the sensing range, collect their biometric information, and compare it with pre-stored identity features to determine the current user's identity. Then, based on the user's characteristics, a suitable wake-up method is selected to activate the terminal. This step lays the foundation for subsequent personalized accessibility adaptation, because only after determining who is using the device can the system retrieve the corresponding capability feature parameters for targeted adaptation.

[0038] In one embodiment, the smart TV of this embodiment can continuously monitor the sensing range through a camera and a human infrared sensor. When someone is detected entering, it automatically collects their facial image as biometric information, compares it with a pre-stored user identity feature database, and after confirming the identity, wakes the user up by voice, gesture or external auxiliary device according to the user's preset preference.

[0039] For example, when an elderly man, Mr. Wang, approaches the smart TV in the living room to watch the news, the smart TV's front-facing camera and infrared sensor detect that Mr. Wang has entered the sensing range. It automatically initiates facial recognition, comparing the captured image of Mr. Wang's face with the system's pre-stored family member identity feature database. If a match is found, the current user is confirmed to be Mr. Wang, as the database records information such as Mr. Wang being elderly, having poor eyesight, and experiencing hearing loss. Since Mr. Wang's preset wake-up preference is voice wake-up, the smart TV in this embodiment enters voice listening mode. Mr. Wang doesn't need to search for the remote control; he can simply say "Hello, TV" to wake up the TV.

[0040] As can be seen, through step S100, this invention achieves a paradigm shift from user-operated terminal to terminal-initiated user-sensing interaction. Biometric-based identity recognition enables the system to know who the user is and what their characteristics are before interaction begins, providing a precise identity prerequisite for subsequent personalized and accessible adaptation. Furthermore, the combination of multiple wake-up methods in this invention's embodiments, such as voice, gestures, and external assistive devices, covers the differences in interaction capabilities among different user groups, including the elderly, visually impaired, hearing impaired, and those with mobility impairments. This allows users to start the terminal in the most natural and effortless way, fundamentally reducing the usage barriers posed by traditional remote controls and complex menus, and achieving a seamless and universally accessible interaction entry point.

[0041] S200: Based on the determined current user identity, retrieve the capability feature parameters that are suitable for the current user from the preset accessibility user profile database; The accessibility user profile database in this embodiment refers to a data management database that centrally stores accessibility-related characteristics and preferences on an individual user basis. It can be built and maintained through ability assessments during initial user registration and continuous learning and updates during subsequent interactions. It has the capability to create multi-dimensional ability descriptions for each user, including visual, auditory, cognitive, and motor skills, providing the system with precise information on the accessibility support each user requires. For example, the profile database can store structured information such as: Mr. Wang: visual acuity 0.3, moderate hearing loss, preference for voice interaction, need for large fonts and high contrast, etc.

[0042] The capability feature parameters in this embodiment refer to a set of structured parameters extracted from an accessible user profile database to guide smart terminals in interface reconstruction and interaction adaptation. By quantifying user capability descriptions, such as visual acuity, hearing ability, motor fine motor skills, cognitive level, and literacy level, into configuration parameters that the system can directly access, these parameters have the ability to dynamically adjust interface presentation, operation modes, and multimodal feedback strategies. This allows the same system to present different interactive forms based on the differentiated capabilities of different users. For example, visual acuity parameters can drive the setting of font size and contrast, while hearing parameters can drive the activation of subtitles and visual cues.

[0043] Step S200 addresses the challenge of transforming a determined user identity into an executable adaptation strategy. Preferably, based on the current user identity determined in step S100, the corresponding ability characteristic parameters for that user are retrieved from a pre-defined accessibility user profile database. These parameters provide a quantitative description of the user's multi-dimensional abilities, including visual, auditory, cognitive, and motor skills, offering a unified adaptation basis for subsequent interface reconstruction in S300, input understanding in S400, and perception enhancement in S500. Step S200 achieves a crucial connection between identity recognition and ability adaptation, enabling accessibility support to evolve from a one-size-fits-all general assistive function to personalized adaptation tailored to each individual user.

[0044] For example, after identifying the current user as Mr. Wang, the system in this embodiment of the invention automatically retrieves Mr. Wang's ability characteristic parameters from the accessibility user profile database, including: visual parameters—e.g., visual acuity of 0.3, requiring font size to be enlarged by 1.8 times and high contrast mode; hearing parameters—e.g., moderate hearing loss, requiring volume enhancement and global subtitles to be enabled; motor parameters—e.g., average fine motor skills, requiring increased touch area and simplified operation path; and cognitive parameters—e.g., weak understanding of complex menus, requiring voice reading aloud. These parameters will serve as the direct basis for subsequent interface reconstruction and interaction adaptation.

[0045] Step S200 establishes a rapid mapping mechanism from identity to intelligent parameters. Centralized management of the accessibility user profile database enables the system to accurately grasp the differentiated capability needs of each user, while the structured expression of capability characteristic parameters allows these needs to be directly and efficiently invoked by the system. This mechanism addresses the problems of fragmented, generic, and unsuitable individual differences in traditional solutions, achieving precise and personalized accessibility support and providing tailored interactive support for users with different capability levels.

[0046] S300. Based on the capability characteristic parameters, dynamically reconstruct the presentation mode of the smart terminal interface elements and switch to the operation mode adapted to the current user; enable multimodal guidance feedback, open the corresponding alternative perception channel according to the user's capability characteristics, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. The dynamic reconstruction of interface element presentation in this step refers to adjusting the visual presentation of the smart terminal interface in real time based on user capability characteristics. By dynamically modifying attributes such as font size, color contrast, icon size, layout density, and touch area size of interface elements, it has the ability to present a differentiated visual appearance to different users for the same interface, compensating for users' visual, motor, and other capability deficiencies. For example, for users with poor vision, the font and icons can be enlarged and the color contrast increased; for users with mobility impairments, the clickable area can be increased and the number of controls requiring fine-grained operation can be reduced.

[0047] The "adapting to the current user's operating mode" step in this section refers to the technical mechanism that dynamically selects and switches between multiple preset operating modes based on the user's ability characteristics. It can utilize preset operating modes such as voice-priority mode, simplified touch mode, and external assistive device mode, automatically matching the optimal operating method based on the user's actual interaction capabilities, enabling the user to complete operations in a way that best suits their abilities. For example, it switches visually impaired users to an operating mode primarily based on voice broadcasts and voice commands, while switching users with mobility impairments to an operating mode primarily based on simplified buttons or external assistive devices.

[0048] The multimodal guidance feedback in this step refers to an interactive mechanism that outputs guidance and feedback information to the user simultaneously or alternately through multiple sensory channels. By integrating various feedback formats such as voice broadcasts, subtitles, and visual cues, it has the ability to flexibly switch and combine between different sensory channels, ensuring that guidance information is effectively conveyed through at least one user-perceptible channel. For example, the system provides operation guidance through voice broadcasts, synchronized on-screen subtitles, and visual highlighting of the focus position.

[0049] The alternative sensory channel in this embodiment refers to an alternative information transmission path used to compensate for a user's lack of a certain sensory ability. Based on the theory of sensory substitution, it converts information from the missing sensory channel to other sensory channels that the user can still use, enabling visually impaired users to obtain information through hearing and hearing-impaired users to obtain information through vision, thus achieving barrier-free information acquisition. For example, converting visual interface information into voice broadcasts to replace seeing, and converting audio content into subtitles to replace hearing, can compensate for a user's lack of a certain sensory ability.

[0050] In this embodiment, interface voice broadcasting, global real-time subtitles, and visual cues refer to three specific alternative perception channel implementations. Interface voice broadcasting reads aloud the menus, buttons, text, and other content on the interface to the user using speech synthesis technology; global real-time subtitles convert audio information such as system prompts and voice content into on-screen text in real time; and visual cues convey prompts to the user through visual means such as flashing, highlighting, color changes, and icon animations. These three features have the ability to provide targeted compensation for different user skill deficiencies, forming a multimodal feedback system that covers major accessibility needs.

[0051] Step S300 addresses the challenge of adapting the interface and feedback to the user's capabilities. Building upon the capability feature parameters retrieved in S200, this step adapts the interface in two dimensions: visual adaptation and auditory adaptation. Visual adaptation dynamically reconstructs the presentation of interface elements based on visual and motion capabilities, switching to an operation mode suitable for the current user. Auditory adaptation enables multimodal guided feedback, activating corresponding alternative sensory channels based on the user's capabilities, such as interface voice broadcasts, real-time global subtitles, and visual cues, outputting guided interactive prompts. Step S300 transforms abstract capability parameters into concrete interface presentation and feedback strategies, shifting the focus from user-adapted interface to interface-adapted user.

[0052] For example, after the system retrieves Mr. Wang's ability characteristic parameters, step S300 begins execution: the font size of interface elements is automatically enlarged by 1.8 times, a high-contrast color scheme is switched, the icon spacing is increased, and the touch area is expanded; the operation mode is switched to voice priority + simplified touch mode, and the menu hierarchy is simplified. At the same time, the control system enables multimodal guided feedback—through the interface voice broadcast reading function, the TV broadcasts the name of the menu item every time Mr. Wang moves the focus to a menu item; since Mr. Wang's hearing is impaired, in this embodiment of the invention, the system will also simultaneously turn on global real-time subtitles, displaying the content of the voice broadcast in large subtitles at the bottom of the screen, supplemented by visual highlighting of the current focus position, outputting the prompt that you are currently in the news section and can enter by pressing the confirmation button or saying open.

[0053] As can be seen, through step S300, the interface presentation and interactive feedback are proactively adapted to the user's capabilities, completely changing the traditional fixed interface and user-adaptive interaction mode. This invention enables users with limited vision or motor skills to clearly identify and easily operate the interface by dynamically reconstructing interface elements; multimodal guided feedback provides alternative information acquisition paths for visually and hearing impaired users through alternative sensory channels. This dual adaptation of interface reconstruction and sensory channel substitution truly enables smart terminals to achieve user-capability-centric, universally accessible interaction, significantly reducing usage barriers for the elderly, disabled, and children.

[0054] S400: Acquire multimodal heterogeneous input signals from the user in the adapted operation mode, the input signals including voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; fuse and understand the multimodal input signals to obtain the user's operation intention; The multimodal heterogeneous input signal in this embodiment refers to a set of user input signals from different sensory sources and with different data formats. It can be composed of various heterogeneous data, such as voice signals (e.g., audio data) collected by a microphone, gesture / lip reading visual recognition signals (e.g., image or video data) collected by a camera, and input signals generated by external auxiliary devices (e.g., discrete control commands). This allows for the simultaneous capture of user intentions from multiple dimensions, overcoming the limitations of single input methods for specific users or scenarios. For example, the same user can express their intentions simultaneously through speech and gestures, with voice and visual signals complementing each other.

[0055] The gesture / lip reading visual recognition signal in this embodiment refers to the visual input form that captures the user's body movements or lip movements through a camera and converts them into understandable semantics. Gesture recognition understands the user's intention by analyzing hand posture and movement trajectory; lip reading infers the language content expressed by analyzing the user's lip movements. Both have the ability to provide alternative input when voice input is unavailable or unreliable, such as in noisy environments or when hearing-impaired users cannot pronounce clearly, thus extending the system's input robustness.

[0056] The fusion understanding in this invention refers to an intelligent processing procedure that aligns, complements, and synthesizes multimodal heterogeneous input signals to form a unified semantic understanding. It uses multimodal fusion models (such as attention-based cross-modal alignment, feature-level fusion, or decision-level fusion techniques) to jointly analyze multiple signals from speech, vision, and assistive devices. It possesses the ability to improve intent recognition accuracy by leveraging the complementarity and redundancy between different modalities, and can accurately interpret the user's true intent even when a single modal signal is missing, ambiguous, or conflicting.

[0057] In this embodiment, the user's operational intent refers to a structured semantic representation obtained after fusion and understanding, characterizing what operation the user wishes to complete through the system. It can parse multimodal input signals into specific operational instructions, such as opening a news channel, increasing the volume, or playing the next episode, and has the capability to be directly consumed and executed by subsequent business logic. It serves as a standardized semantic bridge between user input and system execution.

[0058] Step S400 addresses the challenge of accurately understanding user input intentions under various capability conditions. Building upon the interface adaptation completed in S300, this step acquires multimodal heterogeneous input signals from the user in the adapted operation mode, such as voice, gestures / lip reading, visual input, and external assistive devices. These heterogeneous signals are then fused and analyzed to derive a structured user operation intention. This step overcomes the limitations of a single input channel across different user groups and scenarios through multimodal fusion technology, enabling the system to accurately understand the true intentions of users with varying expressive abilities and operation methods.

[0059] For example, Mr. Wang wants to turn up the volume while watching the news. He says the voice command "louder," and simultaneously makes an upward hand gesture, generating a visual recognition signal. The system in this embodiment simultaneously collects these two heterogeneous signals: the voice recognition module recognizes "louder," and the gesture recognition module recognizes the gesture of "increasing volume." The fusion understanding module aligns and fuses the two signals, determining that their semantics are consistent, thus resolving with high confidence that the user's intention is "to increase the volume." The system then executes the volume increase operation accordingly. When Mr. Wang's voice is soft, resulting in lower confidence in the voice recognition, but the gesture recognition signal is clear, the fusion understanding module in this embodiment can still supplement the semantics with the visual signal to accurately resolve the intention to increase the volume.

[0060] As can be seen, this invention, through the multimodal heterogeneous input and fusion understanding technology in step S400, significantly improves the system's inclusiveness towards different user input capabilities and the accuracy of intent recognition. For users with difficulty in speech expression, such as hearing-impaired users and users with speech impairments, gesture and lip-reading visual recognition provide alternative input channels; for users with mobility impairments, external assistive devices provide reliable input; and for ordinary users, multimodal fusion further improves the robustness and accuracy of recognition. This multi-channel input and fusion understanding mechanism makes the operation of smart terminals no longer an exclusive capability of a certain type of user, truly achieving the goal of universally accessible and inclusive interaction.

[0061] S500 responds to the user's operational intent obtained through parsing and executes the corresponding business; and performs perception enhancement processing on media content: real-time analysis of video images generates visual narration audio, audio signal recognition generates dialogue subtitles and environmental sound effect text descriptions, and outputs through established alternative perception channels to compensate for the user's missing perception information.

[0062] The perception enhancement processing in this step refers to the intelligent analysis of media content and the generation of supplementary perceptual information. Specifically, it can generate supplementary information corresponding to the user's missing perceptual channels through real-time semantic analysis of video images and audio signals, such as generating on-screen narration for visually impaired users and subtitles for hearing impaired users. It has the ability to convert media content into a form that adapts to the user's perceptual capabilities, thus achieving comprehensive accessibility to content.

[0063] The visual narration audio in this embodiment refers to the spoken description audio automatically generated through real-time analysis of video footage. Specifically, it uses computer vision technology to identify elements in the video footage such as scenes, people, actions, and key information like text, facial expressions, and scene transitions, and then uses speech synthesis technology to convert them into natural language narration audio. This provides visually impaired users with the ability to describe the content of the footage, compensating for the visual information they cannot obtain. For example, it can automatically narrate when news footage changes, such as "The scene has switched to the studio, and the host is broadcasting."

[0064] The dialogue subtitles in this embodiment refer to text subtitles generated and displayed in real time through speech recognition of audio signals. They convert dialogues, narration, and other content in audio into text using automatic speech recognition (ASR) technology, enabling them to present the speech content in audio to hearing-impaired users and compensating for the speech information they cannot obtain through hearing.

[0065] The environmental sound effect text description in this embodiment refers to the identification and conversion of non-speech environmental sounds in the audio signal into textual descriptions. Specifically, it can analyze environmental sound effects in the audio, such as applause, thunder, doorbell, laughter, etc., through audio event recognition technology, and describe these sound effects in textual form. This has the ability to convey auditory information in addition to speech to hearing-impaired users, enabling them to fully understand the auditory picture of media content.

[0066] Step S500 is the business execution and perception enhancement layer of the entire method, designed to address how to execute user intentions and compensate for missing user perceptual information. In this step, on one hand, it responds to the user's operational intentions parsed by S400 by executing corresponding business operations, such as switching channels, adjusting volume, or playing programs. On the other hand, during media content presentation, it performs perception enhancement processing—real-time analysis of video footage to generate visual narration audio, allowing users to hear instead of see, serving visually impaired users; and recognition of audio signals to generate dialogue subtitles and environmental sound effects text descriptions, allowing users to see instead of hear, serving hearing-impaired users. This is output through the alternative perceptual channel established by S300, thereby compensating for the user's missing perceptual information. This step ensures that media content acquisition is no longer limited to the user's single perceptual ability.

[0067] For example, Mr. Wang, who has poor eyesight, continues to watch the television news. In terms of execution, the system has already adjusted the volume according to his intention to increase it. Regarding perceptual enhancement, this invention performs real-time analysis of the news footage. When news headlines and data charts appear on the screen, visual narration audio is automatically played, and the screen displays a headline: "Urban Traffic Improvement Plan," with relevant data presented in chart form. Simultaneously, considering Mr. Wang's hearing loss, the system in this embodiment performs speech recognition on the news audio, generating dialogue subtitles displayed below the screen, and generating text descriptions for ambient sound effects (such as applause from the audience). This enhanced information is output through the previously established "voice broadcast + global subtitle" alternative perceptual channel for Mr. Wang, enabling him to fully access the news content.

[0068] For Xiao Li, a hearing-impaired user, the system in this embodiment of the invention focuses on generating subtitles: generating dialogue subtitles in real time for program dialogues, and generating text descriptions for environmental sound effects such as doorbell sounds and explosion sounds, so that Xiao Li can fully understand the program content even though he cannot hear the sound.

[0069] As can be seen, through step S500, this embodiment of the invention extends accessibility support from the "interaction level" to the "content level," achieving enhanced perception and alternative delivery of media content. Visual narration audio enables visually impaired users to understand the visuals, while dialogue subtitles and environmental sound effects text descriptions enable hearing-impaired users to understand the sounds. Both are accurately output through the established alternative perception channel, fundamentally compensating for the perceptual deficiencies experienced by users during media content consumption. This mechanism makes the smart terminal not only an operable device but also an understandable information medium, truly achieving universal information access.

[0070] In a further embodiment of the present invention, the intelligent agent multimodal barrier-free interaction method based on a smart terminal further includes, after step S500: S600: Record the data of this interaction process, iteratively optimize the user's corresponding recognition model based on successful / failed interaction samples, and update the user preference configuration in the accessibility user profile database.

[0071] The interaction process data in this embodiment refers to the set of operational data that records the entire interaction process between the user and the smart terminal. Specifically, it can record the user's input signals, the system's recognition results, the executed operations, and the user's subsequent feedback, such as whether corrections were made, whether the interaction was repeated, or whether it was interrupted. This data has the ability to completely reconstruct an interaction process and can provide original training material for subsequent model optimization and profile updates. For example, it can record the complete chain of "user voice command → recognition result → action execution → user manual correction".

[0072] In this embodiment, the successful / failed interaction samples refer to training samples formed after labeling the interaction process data. Specifically, automatic or semi-automatic labeling can be performed by judging whether an interaction achieves the user's expected goal (success) or fails to achieve it (failure). This has the ability to drive iterative optimization of the model as supervised learning training data, enabling the recognition model to continuously approach the user's true expression habits through continuous use. For example, if the user's voice command is correctly recognized and executed, and the user does not correct it again, the interaction is marked as a successful sample; otherwise, it is marked as a failed sample.

[0073] The recognition model in this embodiment refers to a machine learning model used to recognize and understand user input signals such as biometrics, voice, gestures, and lip reading. Through continuous sample training and parameter updates, it has the ability to continuously adapt to changes in the specific user's expression habits and characteristics, and can gradually improve the recognition accuracy for that user with increased usage. For example, a speech recognition model that is progressively optimized based on a user's accent characteristics.

[0074] The user preference configuration refers to a data set that records the personalized preference settings that users gradually form during use. By mining users' habitual choices and behavioral patterns from interaction process data, such as commonly used operation paths, preferred volume ranges, and preferred wake-up methods, it has the ability to dynamically update to better meet users' actual needs, enabling the accessibility user profile database to continuously evolve from "initial assessment" to "real preferences".

[0075] Step S600 addresses the challenge of enabling the system to better understand users through repeated use. After a complete interaction cycle is completed in steps S100 to S500, this step records the interaction data, iteratively optimizes the user recognition model based on successful / failed interaction samples, and updates the user preference configuration in the accessibility user profile database. This step forms a closed loop of "interaction → recording → learning → optimization," transforming the system from a static, one-time configuration tool into an intelligent agent that continuously evolves with use, achieving continuous self-optimization for accessibility adaptation.

[0076] For example, in one interaction, Mr. Wang said "Open the news broadcast" with a regional accent. The system initially failed to recognize it accurately, mistaking it for "Open the news exhibition." Mr. Wang then manually corrected himself to the correct program. The system in this embodiment of the invention records this interaction as a "failure sample" and incorporates the correspondence between Mr. Wang's dialect pronunciation and the correct command into the training data, iteratively optimizing the speech recognition model corresponding to Mr. Wang. After accumulating many similar samples, the system's speech recognition model gradually learns and adapts to Mr. Wang's accent characteristics, significantly improving subsequent recognition accuracy. Simultaneously, the system discovers from Mr. Wang's interaction data that he habitually watches the news at 7 PM and prefers a medium-to-high volume. Therefore, it automatically updates Mr. Wang's user preference configuration in the profile database, allowing for direct news recommendations and pre-adjustment to his preferred volume the next time he uses the system.

[0077] As can be seen, through the S600 steps, this solution achieves closed-loop self-learning and continuous evolution of the accessibility interaction system. Iterative optimization of the recognition model based on successful / failed interaction samples enables the system to gradually adapt to each user's unique expression habits, such as dialects, accents, and habitual gestures, with recognition accuracy continuously improving with use. The dynamic updating of the accessibility user profile database allows the adaptation strategy to evolve from initial static evaluation to reflect the user's actual preferences, achieving an experience upgrade where "the more you use it, the better it understands you; the more you use it, the smoother it becomes." This closed-loop learning mechanism distinguishes this solution from fragmented, static traditional assistive functions, constructing a universally accessible intelligent interaction system centered on user capabilities, deeply integrated with AI, and capable of continuous self-optimization.

[0078] Optionally, the intelligent agent multimodal barrier-free interaction method based on intelligent terminals described in this invention further includes the following steps before specific implementation: S101. Pre-collect specific ability characteristics of different users and set corresponding ability characteristic parameters for users with different specific ability characteristics; store the specific ability characteristics of different users and their corresponding ability characteristic parameters, including obstacle type, perception channel preference and interface parameters, in the accessibility user profile database.

[0079] The specific capability features in this embodiment refer to personalized descriptive information used to characterize a user's perception, cognition, and operational abilities. These features can be obtained through user self-assessment questionnaires, professional ability assessments, behavioral observation and collection, or historical interaction data analysis. They cover multiple dimensions, including the user's visual ability, auditory ability, language ability, motor control ability, and cognitive comprehension ability. They objectively reflect the specific capability limitations and strengths a user faces when using a smart terminal, providing factual basis for subsequent capability feature parameter settings. For example, a user's "visual acuity 0.2" can be obtained through vision testing, and "moderate hearing loss in the left ear" can be obtained through audiological assessment.

[0080] The disability type refers to a standardized classification label for the category to which a user's specific ability characteristics belong. By categorizing user ability limitations into standard categories such as visual impairment, hearing impairment, speech impairment, physical / motor impairment, cognitive impairment, and age-related combined impairment, it enables the system to quickly identify the specific accessibility support needed by the user. It can serve as a primary classification dimension for organizing and managing user characteristics in an accessibility user profile database. For example, elderly users are often labeled with a combined disability type of "visual decline + hearing loss."

[0081] The aforementioned sensory channel preference refers to the user's preferred settings regarding which sensory channels they prefer to receive information through. By collecting user preferences for different sensory channels such as voice, subtitles, visual cues, and vibration during registration or use, the system can be guided to prioritize information delivery through the user's preferred channels, thus matching the information delivery method with the user's sensory advantages. For example, a visually impaired user might prefer receiving information via voice broadcast, while a hearing-impaired user might prefer receiving information via subtitles and visual cues.

[0082] The interface parameters refer to the set of specific configuration items used to control the visual presentation of the smart terminal interface. By setting specific values ​​for interface attributes such as font size, color contrast, icon size, layout density, touch area size, and animation speed, it has the ability to dynamically reconstruct the presentation of interface elements, enabling the visual form of the interface to accurately adapt to the user's vision and movement capabilities.

[0083] The accessibility user profile library in this embodiment refers to a data management library that centrally stores accessibility-related characteristics and preferences on an individual user basis. Specifically, it can be constructed by pre-collecting specific ability characteristics of users, setting corresponding ability characteristic parameters, and storing them in a structured manner. It has the ability to maintain a multi-dimensional accessibility profile for each user, including disability type, perception channel preference, and interface parameters, and can provide unified data support for the system to retrieve adaptation parameters according to user identity during runtime.

[0084] This step S101, the data preparation layer of the entire invention method, aims to address the problem of how to pre-establish an accessible user profile library so that the system can quickly retrieve adaptation parameters based on user identity during runtime. This step occurs before the real-time interaction process from S100 to S600 and is a preparatory step for the system initialization and user registration phases. Specifically, firstly, specific ability characteristics of different users are pre-collected—for each user, their multi-dimensional ability limitations and strengths in vision, hearing, language, movement, and cognition are collected through self-assessment, evaluation, or historical data; secondly, corresponding ability characteristic parameters are set for users with different specific ability characteristics—the collected ability characteristics are mapped into structured configuration parameters that can be directly called by the system; finally, the specific ability characteristics of different users and their corresponding ability characteristic parameters, including obstacle type, perception channel preference, and interface parameters, are uniformly stored in the accessible user profile library, forming a complete accessible profile organized by user identity.

[0085] In one embodiment, when a user uses a smart terminal for the first time, the present invention will guide the user to complete a capability assessment process and collect the user's specific capability characteristics; based on the collection results, the corresponding capability characteristic parameters will be automatically set for the user; subsequently, these characteristics and parameters, together with the obstacle type, perception channel preference and interface parameters, will be stored in the accessibility user profile library to complete the establishment of the user's initial profile.

[0086] For example, a family of four—Mr. Wang, Mrs. Wang, Mr. and Mrs. Li, and a visually impaired visitor, Mr. Zhao—need to use the same smart TV. Upon first use, the system of this embodiment of the invention collects their respective capability characteristics: For Mr. Wang, an elderly person with 0.3 vision and moderate hearing loss, the system collects specific ability characteristics such as vision decline, hearing loss, and decreased fine motor skills. Based on these collected specific ability characteristics, this invention sets ability characteristic parameters—font size increased by 1.8 times, high contrast enabled, volume enhancement enabled, touch area enlarged, and operation path simplified; the impairment type is labeled as: age-related complex impairment; the perception channel preference is set to: voice broadcast priority + global subtitle assistance; and the interface parameters are set to large font, high contrast, and large icons.

[0087] For Grandma Wang, an elderly person with relatively good eyesight but declining hearing, the system settings in this embodiment of the invention include: font size magnified by 1.2 times, volume enhancement enabled, and global subtitles enabled; the impairment type is hearing impairment; the perception channel preference is subtitle priority; and the interface parameters are standard font size with subtitle bar.

[0088] For Mr. and Mrs. Li, who are middle-aged and young adults with no significant impairments, the system in this embodiment of the invention is set to standard ability characteristic parameters, the impairment type is no significant impairment, and standard interface parameters are used.

[0089] For Xiao Zhao, a visitor who is completely visually impaired, the system sets the following capability characteristics parameters: full-time interface voice broadcasting, disabled visual-dependent operations, and voice command priority mode; the disability type is visual impairment; the perception channel preference is pure voice broadcasting; and the interface parameters are set to voice navigation mode.

[0090] All of the aforementioned users' specific ability characteristics, ability characteristic parameters, obstacle types, perception channel preferences, and interface parameters are structurally stored in the accessibility user profile database. Subsequently, when any user enters the perception range of the smart terminal, the system can accurately obtain the corresponding adaptation parameters for that user through identity recognition in S100 → profile retrieval in S200.

[0091] As can be seen, this invention, through the preliminary step S101, pre-constructs a structured, user-capability-centric barrier-free user profile library, providing a solid data foundation for the entire real-time interaction process. First, by pre-collecting specific capability characteristics and setting corresponding capability characteristic parameters, the system gains a quantifiable and actionable understanding of each user's capability status, avoiding delays and inaccuracies in runtime assessments. Second, by incorporating barrier types, perception channel preferences, and interface parameters into the unified storage of the profile library, the profile information covers three key dimensions: "what barriers the user has, what perception methods they prefer, and what interface presentation they need," providing a complete and consistent adaptation basis for interface reconstruction in S300, input understanding in S400, and perception enhancement in S500. Finally, the centralized management of the profile library enables the same terminal to simultaneously serve multiple users with significantly different capabilities (such as the elderly and visually impaired visitors), and allows for rapid and accurate adaptation switching when users switch, truly achieving the universal intelligent interaction goal of "one terminal, multiple forms, usable by everyone."

[0092] Optionally, the intelligent agent multimodal barrier-free interaction method based on intelligent terminal of the present invention further includes step S100: performing facial recognition through a camera or performing voiceprint recognition through a voice perception module with a microphone array to seamlessly determine the identity of the current user; wherein the voice perception module adopts a microphone array and has a built-in optimized recognition model for the unclear speech of the elderly and the voices of children.

[0093] In this embodiment of the invention, facial recognition and / or voiceprint recognition can be performed using a camera and a voice perception module with a microphone array to seamlessly identify the current user. The voice perception module employs a high-performance microphone array and incorporates optimized recognition models for the unclear speech of the elderly and children's voices. When an elderly user with mild Parkinson's disease says "play opera" in a trembling and unclear voice, the system of this embodiment can accurately recognize their intention and will not be misinterpreted by the standard voice recognition model as playing something else. This significantly improves the accuracy of recognizing the unclear speech of the elderly, such as unclear pronunciation, slow speech rate, and heavy regional accents; and the high pitch and incomplete pronunciation of children's voices. Through targeted acoustic model training, the error rate of voice recognition caused by the physiological characteristics of specific groups is reduced, avoiding user frustration caused by repeated recognition failures and ensuring the usability of the voice interaction channel for users of all ages.

[0094] In a further embodiment of the present invention, the intelligent agent multimodal barrier-free interaction method based on a smart terminal, wherein step S300 further includes: S301. When the current user is a visually impaired user, the control engine shuts down visual rendering that consumes more energy than the predetermined energy value, switches to a pure audio user interface, and converts the menu structure and option status into a voice broadcast sequence. Step S301 is a dedicated interface switching operation for visually impaired users. In practice, once the system confirms the current user is visually impaired through identity verification, given that visual rendering is no longer meaningful to the user but continues to consume system resources, the system controls the rendering engine to shut down visual rendering that consumes more energy than a predetermined value, such as high-definition graphics rendering, animation effects, and high refresh rate displays; the entire interface is switched to a pure audio user interface; simultaneously, the menu structure and option states, originally presented as visual graphics, are converted into a sequence of voice announcements. That is, the menu hierarchy, current options, and executable operations are reorganized into sequentially read audio content, and the interface information is completely conveyed to the visually impaired user through the voice channel.

[0095] In this embodiment, the reason for disabling high-energy-consuming visual rendering is that visually impaired users cannot benefit from visual rendering. Disabling visual rendering that consumes more energy than a predetermined value, such as 4K high-definition rendering, complex animations, and high-brightness backlighting, eliminates useless resource overhead for the user, achieving a balance between accessibility and energy efficiency. On the other hand, it reduces hardware load and heat generation, indirectly improving the terminal's operational stability and the response speed of other sensory channels such as voice.

[0096] In this embodiment, the pure audio user interface refers to an interface form that carries and presents interactive information entirely through audio, and can complete all operations without relying on any visual elements. It is a complete and equivalent reconstruction of the visual interface in the auditory channel.

[0097] Converting a menu structure into a voice broadcast sequence refers to reorganizing the parallel or hierarchical menu items and current option statuses (such as "selected" or "selected") on the visual interface into a broadcast sequence that can be read aloud sequentially by a speech synthesis engine, according to a preset order. For example, "Main Menu: Channel - Settings - Applications" can be converted into the voice sequence "The main menu has three items: first, Channel; second, Settings; third, Applications; currently in the first, Channel."

[0098] As can be seen, this step not only provides visually impaired users with a fully usable pure audio interface, but also optimizes resources by turning off unnecessary high-energy-consuming visual rendering. This not only ensures the independent use ability of visually impaired users, but also reduces the power consumption and load of the terminal, demonstrating the dual technical effects of accessibility design and energy efficiency optimization.

[0099] S302. When the current user is a hearing-impaired user, the control engine overlays a real-time subtitle layer on the video layer and converts the system prompts into flashing or color changes at the edge of the screen; and / or converts system prompts, incoming call reminders, and operation confirmation information into vibration tactile signals through communication with the remote control or smart bracelet. In this embodiment of the invention, overlaying a real-time subtitle layer onto a video layer means overlaying an independent, configurable subtitle rendering layer on top of the video screen, and converting the speech content in the audio into text in real time after speech recognition and displaying it synchronously, so that hearing-impaired users can understand the dialogue content that they could only hear before.

[0100] The conversion of system prompts into screen edge flashing or color changes refers to transforming system prompts that were originally presented only in sound form, such as operation success prompts and error warnings, into visual signals at the screen edge. Different prompt types are distinguished by differentiating the flashing frequency and color changes, thereby achieving the visual transmission of prompt information.

[0101] The communication linkage with remote control or smart bracelet to convert into vibration tactile signals refers to the use of short-range communication methods such as Bluetooth to enable the terminal to link with personal devices with vibration motors, such as remote control or smart bracelet, to convert system prompts, incoming call reminders, operation confirmation information and other events into specific vibration modes. For example, different vibration durations, frequencies and rhythms correspond to different event types, so that hearing-impaired users can perceive reminders in time through touch even when their eyes are away from the screen.

[0102] Step S302 is a dedicated sensory substitution operation for hearing-impaired users. In specific implementation, after the system confirms the current user is hearing-impaired through identity recognition, since hearing-impaired users cannot receive information through hearing, the system in this embodiment compensates through two alternative paths: visual and tactile. On one hand, the control engine overlays a real-time subtitle layer on top of the video layer, converting audio content such as dialogue and narration into synchronously displayed text, and converting system prompts into visual cues such as screen edge flashing or color changes. On the other hand, through communication with external devices such as remote controls or smart bracelets, system prompts, call reminders, and operation confirmation information can be converted into vibrational tactile signals, transmitting information originally dependent on hearing to hearing-impaired users through the tactile channel.

[0103] As can be seen, through this step S302, the present invention constructs a dual alternative sensory channel for hearing-impaired users, namely "subtitles + visual cues" and "vibration and touch," enabling them to effectively obtain information that originally relied on hearing in scenarios such as watching programs, receiving incoming calls, and confirming operations. In particular, by linking with the vibration of the wearable device, it compensates for the limitation of visual substitution failing when the user's gaze leaves the screen, achieving full-scenario coverage of information transmission and significantly improving the independence of hearing-impaired users in using smart terminals and the completeness of information acquisition.

[0104] S303. When the current user is an elderly person, the control engine simplifies the interface hierarchy and enlarges the click hotspot to a predetermined value. Step S303 involves simplifying and adapting the interface for elderly users. Once the user is confirmed to be elderly through identity verification, considering the common characteristics of elderly individuals such as declining vision, reduced cognitive comprehension, and weakened fine motor skills, the system control engine of this invention adapts the interface from two dimensions: structural simplification and operational error tolerance. On the one hand, it simplifies the interface hierarchy, reduces menu nesting depth and the number of function entry points, and prioritizes frequently used functions, allowing elderly users to reach their target functions without complex navigation. On the other hand, it enlarges the clickable hotspot—the effective area on the interface that can respond to clicks—to a predetermined value, reducing the precision requirements for clicks, so that elderly users can successfully trigger the operation even if the click position is slightly off.

[0105] As can be seen, step S303 lowers the barrier to entry for elderly users by improving both the interface structure and operational error tolerance. Simplifying the interface hierarchy reduces the cognitive burden and navigation complexity for elderly users; enlarging the clickable hotspot to a predetermined value lowers the requirements for fine motor skills, significantly reducing accidental touches and missed clicks, enabling elderly users to complete operations more easily and accurately.

[0106] S304. When the current user is a user with difficulty in controlling fine motor skills of the upper limbs, head tracking is used to select the target instead of the input device; the interaction parameters are dynamically adjusted through software algorithms: the click hotspot is enlarged and the cursor movement speed is reduced to the preset movement speed; Step S304 involves input method substitution and interaction parameter optimization for users with difficulty in fine motor control of the upper limbs. When the system identifies that the current user has difficulty in fine motor control of the upper limbs, such as hand tremors or motor dysfunction, this embodiment of the invention adapts from two dimensions: input substitution and parameter softening. On the one hand, head tracking technology is used to replace traditional input devices, such as remote controls and touch screens, for target selection. That is, the user's head movement trajectory is tracked in real time by a camera, and the head orientation or movement is mapped to the cursor position and selection operation on the screen, so that the user can complete the target selection without relying on fine hand operation. On the other hand, the interaction parameters are dynamically adjusted through software algorithms—the click hotspot is enlarged to reduce the selection accuracy requirement, and the cursor movement speed is reduced to a preset movement speed to adapt to the user's limited motor control ability and avoid the cursor being unable to stop on the target due to moving too fast.

[0107] In this way, through this step, the present invention opens up an alternative input channel of head tracking for users who have difficulty controlling hand movements, freeing them from dependence on fine hand operations; at the same time, by enlarging the click hotspot and reducing the cursor movement speed, the precision and speed requirements of the interaction are softened at the software level. The combination of these two aspects enables such users to accurately and stably complete target selection and operations in a way that is under their control, significantly improving the feasibility and comfort of operation.

[0108] S305, the virtual assistant greets you with both voice and text and provides multimodal guidance.

[0109] This step utilizes a virtual assistant to provide multimodal greetings and guidance. Specifically, when the user completes identity verification and enters the adapted interactive environment, the system of this embodiment can activate the virtual assistant to greet the user in both voice and text, such as simultaneously broadcasting and displaying "Welcome back, Grandpa Wang" on the screen, ensuring that the greeting information is conveyed through at least one perceptible channel. Subsequently, the virtual assistant provides multimodal guidance, that is, it guides the user to complete subsequent operations synchronously or alternately through multiple perceptual channels such as voice broadcasting, on-screen subtitles, and visual highlighting. For example, it broadcasts the currently available operation options, visually highlights the focus position, and synchronously displays operation instructions with subtitles.

[0110] In this way, by using a virtual assistant to greet users and provide multimodal guidance through both voice and text in this step, the differentiated needs of users with different perceptual abilities are taken into account—visually impaired users can rely on voice, hearing impaired users can rely on subtitles, and the elderly can receive both visual and auditory prompts simultaneously. This multimodal guidance reduces the user's sense of confusion after entering the interactive state, enabling users to clearly know where they are, what they can do, and how to operate, thus improving the coherence and user-friendliness of the entire accessible interaction process.

[0111] In a further embodiment of the present invention, the intelligent agent multimodal barrier-free interaction method based on intelligent terminal can further enhance the fusion and understanding of multimodal input signals by: using a weighted fusion algorithm to correlate and parse the screen space coordinates of the user's gesture pointing position with the speech semantics, thereby obtaining the user's operation intention for refined control.

[0112] In this embodiment of the invention, in order to solve the ambiguity problem of single-modal input, a weighted fusion algorithm is used to associate and parse the screen space coordinates of the gesture pointing position with the speech semantics such as "this" and "here", so as to achieve fine control.

[0113] This invention breaks the limitations of single input methods through a multimodal input fusion mechanism, specifically a multimodal intent fusion engine. Specifically, the multimodal intent fusion engine simultaneously listens to voice commands (such as playing news), gesture trajectories (such as waving to the left), and assistive device signals (such as head tracker cursor position and lip reading features), and uses a weighted fusion algorithm to analyze user intent. For example, if an elderly user points to a small window in the lower right corner of the screen and says, "This... louder," the multimodal intent fusion engine in this embodiment of the invention combines gesture trajectory location (such as the lower right corner area) with voice keywords (such as "louder"), accurately identifying it as increasing the volume of the application in the lower right corner window, rather than simply increasing the main TV volume. This allows for finer control and provides a better user experience. This fusion mechanism allows users to utilize their most reliable operational abilities; for example, users who have lost fine hand control can use head tracking, and aphasic users can interact with the system using gestures.

[0114] The present invention will be further described in detail below through another specific application embodiment: The intelligent agent multimodal barrier-free interaction method based on a smart terminal provided in this specific application embodiment 2 includes the following steps: S1: Seamless Recognition and Personalized Wake-up Steps. The current user's identity can be seamlessly determined through facial recognition or voiceprint recognition via camera; then, based on the confirmed current user identity, the corresponding accessibility profile is automatically loaded.

[0115] In this embodiment of the invention, after identifying the current user's identity, the smart TV can be woken up in a variety of ways, such as by voice activation, by waving at the camera, or by pressing an external switch to activate the auxiliary device.

[0116] S2: Dynamic Interface Adaptation and Multimodal Bootstrapping Steps: The TV interface is immediately switched to the adaptation mode through the interface rendering engine. For example, the "voice navigation mode" is activated for visually impaired users, and the entire interface operation is driven and broadcast by voice; for hearing impaired users, global subtitles are enabled and enlarged by default.

[0117] In this embodiment of the invention, a virtual assistant greets and guides the user using both voice and text: "Hello, what do you need me to do? You can say I want to watch TV or listen to the news."

[0118] S3: Multimodal command reception and fusion understanding steps: Users can issue commands in their preferred way: for example, by issuing a voice command: "Play CCTV-1." They can also use gestures: make a "swipe left" gesture to switch channels.

[0119] It can also be achieved through combined interactions. For example, an elderly person might point to a certain area of ​​the screen and say, "This... louder." In this embodiment of the invention, the system can understand that the intention is to adjust the volume of the specified area.

[0120] S4: Intelligent Content Understanding and Enhanced Presentation Steps: In this step, for the video content: the system automatically analyzes the screen and generates and broadcasts visual narration for visually impaired users, such as: in the screen, a woman wearing red clothes is running in the park.

[0121] For audio content: For hearing-impaired users, in addition to providing dialogue subtitles, non-voice information such as background music and environmental sound effects are also described in text, such as when tense music plays.

[0122] S5: Continuous learning and optimization steps; In this embodiment of the invention, the system also records successful and failed cases of user interaction and continuously optimizes the recognition model and interaction strategy for that user, such as learning and adapting to the unique local accent of an elderly person.

[0123] For example, in a scenario for visually impaired users: 1) Mr. Wang, who is visually impaired, walks into the living room, and the TV camera recognizes his identity (corresponding to step S1 above). 2) The system in this embodiment immediately switches the smart TV to voice navigation mode. The screen dims to save power, and a voice announcement is made: "Hello Mr. Wang, voice navigation is now enabled. Currently, on the main interface, there are three options: TV series, news, and music" (corresponding to step S2 above); 3) Mr. Wang says: "Listen to the news" (corresponding to step S3 above); 4) The system in this embodiment starts playing the news broadcast. At the same time, the content understanding engine analyzes the news footage in real time and inserts key visual information through voice: "Behind the anchor is a scene of a meeting, and the screen displays a headline about steady economic growth..." (corresponding to step S4 above), enabling Mr. Wang to obtain information almost equivalent to that of a sighted person.

[0124] For scenarios involving users with mobility impairments: 1) For example, Ms. Li, who has difficulty moving her upper limbs, uses a remote control equipped with a head tracker. She controls the screen cursor by making slight head movements and confirms the selection by clicking a large switch (corresponding to steps S1 and S3 above). 2) The system maintains the standard layout of its interface but enlarges the clickable area and provides a slow cursor movement speed (corresponding to step S2 above). 3) Ms. Li successfully selects and plays her favorite opera program.

[0125] In this embodiment of the invention, the user profile database can be initially set up with the assistance of an administrator, such as a family member. All biometric data is processed locally, fully protecting user privacy. This embodiment of the invention encourages users to adjust the interface in real time through simple interactions, such as indicating that the text is too small; the system will learn and save this preference.

[0126] In a further optimized embodiment of the present invention, the intelligent agent multimodal barrier-free interaction method based on a smart terminal includes the following steps: S910: Deploy an audio activity detector in advance in the audio decoding layer of the smart terminal to analyze the loudness and spectral characteristics of the audio stream of film and television content in real time, identify and mark two types of time windows as: an absolutely silent window where the audio loudness meets the first preset condition, and a relatively low-activity window where the speech pause or background music meets the second preset condition. Add a timestamp and confidence label to each detected window to build a dynamic insertable time slot queue. In this embodiment, an audio activity detector (i.e., an audio activity detection module) can be pre-deployed at the television audio decoding layer to analyze the loudness and spectral characteristics of the audio stream of film and television content in real time, and identify and mark two types of time windows: The first is an absolutely silent window, where the judgment condition is set to the section where the audio loudness is below -40dB and lasts for more than 300 milliseconds, which usually corresponds to scene switching or plot pauses; Second, a relatively low-activity window, the judgment condition is set to the segment where the voice pause interval exceeds 500 milliseconds, or the background music is in a gradually decreasing / low volume state.

[0127] The smart TV system of this invention assigns a timestamp and confidence label to each detected window, thereby constructing a dynamic, insertable time slot queue.

[0128] The S920 audio activity detection module performs real-time analysis of the content audio to identify natural auditory gaps in film and television narratives. An audio activity detection module, also known as an audio activity detector, is a signal processing component used to monitor and analyze the sound activity state in audio signals in real time. It can perform real-time analysis of audio content frame by frame or segment by segment using technologies such as Voice Activity Detection (VAD), audio event classification, and energy threshold analysis. It has the ability to distinguish between "active periods with speech / key sound effects" and "silent / low-activity periods without important sounds" in audio, providing accurate basis for determining the timing of inserting narration audio.

[0129] The audio content in this embodiment refers to the collection of sound signals that carry information in media content such as movies, TV shows, news, and live broadcasts. It includes various sound components such as dialogue, narration, environmental sound effects, and background music, and has the function of conveying narrative information in movies and TV shows. It is the core object processed for subtitle generation for hearing-impaired users and narration insertion for visually impaired users.

[0130] The natural auditory gaps in this embodiment refer to brief periods of silence or low activity that naturally exist in the audio content and do not contain important speech content or key sound effects. These gaps mostly exist within film and television narratives, such as pauses between character dialogues, transitions between scenes, and empty shots without dialogue. They possess the ability to provide a natural window for narration without interrupting the original audio narrative, thus avoiding overlap and interference between the narration and the original film / television soundtrack.

[0131] This step in the embodiment is a pre-analysis stage in the visual narration audio generation process, designed to address the issue of when to insert narration audio without conflicting with the original film / TV soundtrack. When the smart TV system of this invention generates visual narration audio for visually impaired users, the narration needs to be inserted into the film / TV audio. However, if the narration and the original dialogue or key sound effects are played simultaneously, they can easily interfere with each other, making it difficult for the user to hear clearly. Therefore, this embodiment of the invention uses an audio activity detection module to perform real-time analysis of the content audio. Through voice activity detection and audio event classification, it identifies naturally occurring auditory gaps in the film / TV narrative—that is, silent / low-activity periods without important dialogue, narration, or key sound effects. These identified natural auditory gaps are the safe insertion points for subsequent narration audio.

[0132] For example, in a film or television drama, a scene might depict a dialogue between two main characters, followed by a transition to an empty landscape shot with soothing background music, before returning to the dialogue after a few seconds. The audio activity detection module of this embodiment analyzes this audio segment in real time: identifying the dialogue segment as active and the empty landscape segment as a relatively inactive window (judgment criteria set to a pause in speech exceeding 500 milliseconds or a section where the background music is gradually decreasing / low volume). The system of this embodiment identifies this empty landscape segment as a "natural auditory gap," facilitating the insertion of visual narration audio within this gap in subsequent steps, such as announcing, "The scene transitions to a lakeside at dusk, with the setting sun reflecting on the water." Because the narration is precisely timed to coincide with the dialogue pauses, visually impaired users can clearly hear the original dialogue while simultaneously receiving the visual narration without interference.

[0133] As can be seen, the real-time analysis of the audio content by the audio activity detection module in step S920 accurately identifies natural auditory gaps in the film and television narrative, providing a safe and conflict-free window for inserting narration audio. This fundamentally avoids the problem of overlapping and mutual interference between narration and the original film and television soundtrack, which prevents users from hearing clearly, in traditional solutions. It enables visual narration audio to supplement the missing visual information of visually impaired users without interruption or interference, significantly improving the quality of perceptual enhancement processing and the naturalness and fluency of the user experience.

[0134] S930. Establish a navigation voice buffer, temporarily store the interface information to be broadcast in the queue, and do not immediately trigger voice output. Use the natural auditory gap as the transmission time slot of the navigation voice through the intelligent scheduler. When the natural auditory gap is available, extract the navigation text to be broadcast from the navigation voice buffer and transmit it. Perform semantic compression and speed-changing processing on the navigation text to shorten the time occupied by a single voice message. In this embodiment of the invention, a navigation voice buffer is established to temporarily store the interface information to be broadcast, such as "Currently selected: Historical Drama category, volume increased to 30%", in a queue without immediately triggering voice output. A scheduler, based on the timing markers of the insertable time slot queue, uses a shortest wait-first algorithm to preload speech synthesis data 200 milliseconds in advance when an upcoming relatively inactive window is detected, ensuring immediate broadcast upon window arrival. If a high-priority emergency alert is detected, such as low battery or network disconnection, a forced insertion mechanism is triggered. This involves automatically reducing the content audio volume (e.g., limiting it to 20% of the original volume) and increasing the navigation voice volume (e.g., gain to 120% of the standard volume) to achieve instantaneous preemption. Simultaneously, a 0.5-second fade-in / fade-out transition is superimposed on the content audio track to avoid abrupt auditory interruptions.

[0135] In addition, in this embodiment of the invention, the navigation text is semantically simplified by removing redundant modifiers. The complete description, such as "Your current location is the third row, second column of the main menu interface, and the corresponding item is the Movie Channel," is compressed into key information units, such as "Main Menu - Movie Channel - Focus," with a compression ratio controlled within 40%. Simultaneously, a temporal compression algorithm is used to increase the standard broadcast speed from 150 words per minute to 220 words per minute, compressing the duration of a single navigation voice message to within 1.2 seconds to accommodate the short dialogue breaks common in film and television content. This shortens the duration of a single voice message and improves the efficiency of navigation voice broadcasting.

[0136] S940: Before the navigation voice broadcast, a tactile pre-prompt is issued through the remote control held by the user or the smart bracelet worn by the user at a predetermined time, and the user's attention is turned to the system navigation by using the cross-modal attention guidance effect; for urgent information that must be responded to immediately, channel switching is achieved by loudness contrast and fade-in / fade-out technology.

[0137] In a specific embodiment of the present invention, a predetermined time, such as 150 milliseconds, is used before the actual broadcast of the navigation voice. A specific pattern of short vibrations, such as two short and one long pulse, is emitted via the user's remote control or smart bracelet. This utilizes the independence of the tactile channel to pre-establish the user's attention and prepare them for receiving voice information. For televisions supporting multi-channel output, the navigation voice is fixedly mapped to the center channel or virtually positioned directly in front of the user using a head-related transfer function (HRTF) algorithm, while the content audio remains in a stereo field. This utilizes the principle of spatial auditory separation to reduce mutual masking between the two.

[0138] As can be seen from the above, the optimized scheme of this invention achieves dynamic reuse of auditory resources through a four-stage collaborative mechanism of "detection-waiting-compression-prompt". The audio activity detection module analyzes the content audio in real time to accurately identify natural auditory gaps in film and television narratives; the intelligent scheduler utilizes these natural auditory gaps as transmission slots for navigation voice, fundamentally avoiding congestion in the auditory channel; semantic compression and speed-up processing shorten the duration of a single voice message, improving the utilization rate of brief gaps; and tactile pre-prompts utilize cross-modal attention guidance effects to ensure that users can immediately shift their attention to system navigation when auditory gaps occur, without occupying auditory bandwidth. In this embodiment, for urgent information requiring immediate feedback, loudness contrast and fade-in / fade-out techniques are used to achieve rapid channel switching. This refined time-domain allocation strategy enables visually impaired users to obtain clear and timely interactive feedback without interrupting the continuity of the film and television plot, achieving time-division multiplexing and harmonious coexistence of content consumption and interface navigation on a single auditory channel.

[0139] Exemplary device like Figure 2 As shown, this embodiment of the invention provides a smart agent multimodal barrier-free interaction device based on a smart terminal, comprising: The multimodal perception module 310 is used to collect user biometric information when a user is detected entering the perception range of the smart terminal; compare the collected biometric information with pre-stored user identity features to determine the current user's identity; the smart terminal is awakened by triggering methods such as voice wake-up, gesture wake-up and / or wake-up by external auxiliary devices; The capability feature parameter retrieval module 320 is used to retrieve capability feature parameters adapted to the current user from a preset accessibility user profile database based on the determined current user identity. The adaptation mode switching module 330 is used to dynamically reconstruct the presentation mode of the smart terminal interface elements according to the capability feature parameters, switch to the operation mode adapted to the current user; enable multimodal guidance feedback, open the corresponding alternative perception channel according to the user's capability features, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. The multimodal interaction module 340 is used to acquire multimodal heterogeneous input signals from the user in the adapted operation mode, the input signals including voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; and to fuse and understand the multimodal input signals to obtain the user's operation intention. The multi-module response execution module 350 is used to respond to the parsed user operation intent and execute the corresponding business; and to perform perception enhancement processing on media content: real-time analysis of video images to generate visual narration audio, recognition of audio signals to generate dialogue subtitles and environmental sound effects text descriptions, and output through established alternative perception channels to compensate for the user's missing perception information. The data optimization module 360 ​​is used to record the data of this interaction process, iteratively optimize the user's corresponding recognition model based on successful / failed interaction samples, and update the user preference configuration in the accessibility user profile library, as described above.

[0140] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 3As shown, the intelligent terminal includes a processor, memory, network interface, display screen, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal barrier-free interaction method based on the intelligent terminal. The database of the intelligent terminal stores the multimodal barrier-free interaction program based on the intelligent terminal.

[0141] Those skilled in the art will understand that Figure 3 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one embodiment, a smart terminal is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: A. When a user is detected entering the sensing range of the smart terminal, the user's biometric information is collected; the collected biometric information is compared with the pre-stored user identity features to determine the current user's identity; the smart terminal is awakened by triggering methods such as voice wake-up, gesture wake-up, and / or external auxiliary devices. B. Based on the determined current user identity, retrieve the capability feature parameters that are suitable for the current user from the preset accessibility user profile database; C. Based on the aforementioned capability characteristic parameters, dynamically reconstruct the presentation mode of the smart terminal interface elements and switch to an operation mode adapted to the current user; enable multimodal guidance feedback, and open corresponding alternative perception channels according to the user's capability characteristics, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. D. Acquire multimodal heterogeneous input signals from the user in the adapted operation mode, the input signals including voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; fuse and understand the multimodal input signals to obtain the user's operation intention; E. Execute the corresponding business based on the user's operation intent obtained from the response parsing; and perform perception enhancement processing on the media content: analyze the video screen in real time to generate visual narration audio, identify the audio signal to generate dialogue subtitles and environmental sound effect text descriptions, and output through the established alternative perception channels to compensate for the user's missing perception information, as described above.

[0143] The step E is followed by: F. Record the data of this interaction process, iteratively optimize the user's corresponding recognition model based on successful / failed interaction samples, and update the user preference configuration in the accessibility user profile database.

[0144] The procedure preceding step A includes: Pre-collect specific ability characteristics of different users, and set corresponding ability characteristic parameters for users with different specific ability characteristics; The specific ability characteristics of different users, along with their corresponding ability characteristic parameters, including barrier type, perception channel preference, and interface parameters, are stored in the accessibility user profile database.

[0145] The aforementioned intelligent agent multimodal barrier-free interaction method based on intelligent terminals, wherein the step of collecting user biometric information when a user is detected entering the sensing range of the intelligent terminal includes: The identity of the current user can be determined seamlessly by using facial recognition via a camera or voiceprint recognition via a voice perception module with a microphone array. The voice perception module uses a microphone array and incorporates an optimized recognition model for the unclear speech of the elderly and the voices of children.

[0146] The steps of dynamically reconstructing the presentation mode of the smart terminal interface elements based on the capability feature parameters and switching to an operation mode adapted to the current user; enabling multimodal guidance feedback and opening corresponding alternative perception channels based on the user's capability features, including interface voice broadcast reading, global real-time subtitles, visual cues, and outputting guidance interaction prompts, further include: When the current user is a visually impaired user, the control engine shuts down visual rendering that consumes more energy than the predetermined energy value, switches to a pure audio user interface, and converts the menu structure and option status into a voice broadcast sequence. When the current user is a hearing-impaired user, the control engine overlays a real-time subtitle layer on the video layer and converts the system prompts into flashing or color changes at the edge of the screen; and / or converts system prompts, call reminders, and operation confirmation information into vibration and tactile signals through communication with the remote control or smart bracelet. When the current user is an elderly person, the control engine simplifies the interface hierarchy and enlarges the click hotspot to a predetermined value; When the current user is someone who has difficulty controlling fine motor skills in their upper limbs, head tracking is used to select the target instead of an input device; the interaction parameters are dynamically adjusted through software algorithms: the click hotspot is enlarged and the cursor movement speed is reduced to a preset speed; The virtual assistant greets you with both voice and text and provides multimodal guidance.

[0147] The step of acquiring multimodal heterogeneous input signals from the user in the adapted operation mode, wherein the input signals include voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; and fusing and understanding the multimodal input signals to parse the user's operation intention, further includes: By using a weighted fusion algorithm, the screen space coordinates of the user's gesture location are correlated and parsed with the speech semantics to obtain the user's operation intention, which is then used to achieve fine-grained control.

[0148] It also includes the following steps: An audio activity detector is pre-deployed in the audio decoding layer of the smart terminal to analyze the loudness and spectral characteristics of the audio stream of film and television content in real time. Two types of time windows are identified and marked: an absolutely silent window where the audio loudness meets the first preset condition, and a relatively low-activity window where the speech pauses or background music meet the second preset condition. Each detected window is stamped with a timestamp and confidence label to build a dynamic, insertable time slot queue. The audio activity detection module performs real-time analysis of the audio content to identify natural auditory gaps in film and television narratives; A navigation voice buffer is established to temporarily store the interface information to be broadcast in a queue without immediately triggering voice output. The natural auditory gap is used as the transmission time slot for navigation voice by an intelligent scheduler. When the natural auditory gap is reached, the navigation text to be broadcast is extracted from the navigation voice buffer and transmitted. The navigation text is also subjected to semantic compression and speed-changing processing to shorten the duration of a single voice message. Before the navigation voice broadcast, a tactile pre-prompt is issued via the remote control held by the user or the smart bracelet worn by the user, using the cross-modal attention guidance effect to redirect the user's attention to the system navigation; for urgent information that must be responded to immediately, channel switching is achieved through loudness contrast and fade-in / fade-out technology, as described above.

[0149] In other embodiments, this application proposes a computer-readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the above-described method.

[0150] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals, characterized in that, include: A. When a user is detected entering the sensing range of the smart terminal, the user's biometric information is collected; the collected biometric information is compared with the pre-stored user identity features to determine the current user's identity; the smart terminal is awakened by triggering methods such as voice wake-up, gesture wake-up, and / or external auxiliary devices. B. Based on the determined current user identity, retrieve the capability feature parameters that are suitable for the current user from the preset accessibility user profile database; C. Based on the aforementioned capability characteristic parameters, dynamically reconstruct the presentation mode of the smart terminal interface elements and switch to an operation mode adapted to the current user; enable multimodal guidance feedback, and open corresponding alternative perception channels according to the user's capability characteristics, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. D. Acquire multimodal heterogeneous input signals from the user in the adapted operation mode, the input signals including voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices; fuse and understand the multimodal input signals to obtain the user's operation intention; E. Execute the corresponding business logic based on the user's operation intent obtained from the response parsing; It also performs perceptual enhancement processing on media content: real-time analysis of video footage generates visual narration audio, audio signal recognition generates dialogue subtitles and environmental sound effect text descriptions, and outputs them through established alternative perceptual channels to compensate for the user's missing perceptual information.

2. The method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals according to claim 1, characterized in that, Step E is followed by: F. Record the data of this interaction process, iteratively optimize the user's corresponding recognition model based on successful / failed interaction samples, and update the user preference configuration in the accessibility user profile database.

3. The method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals according to claim 1, characterized in that, Before step A, the following also applies: Pre-collect specific ability characteristics of different users, and set corresponding ability characteristic parameters for users with different specific ability characteristics; The specific ability characteristics of different users, along with their corresponding ability characteristic parameters, including barrier type, perception channel preference, and interface parameters, are stored in the accessibility user profile database.

4. The method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals according to claim 1, characterized in that, The step of collecting user biometric information when a user is detected entering the sensing range of the smart terminal includes: The identity of the current user can be determined seamlessly by using facial recognition via a camera or voiceprint recognition via a voice perception module with a microphone array. The voice perception module uses a microphone array and incorporates an optimized recognition model for the unclear speech of the elderly and the voices of children.

5. The intelligent agent multimodal barrier-free interaction method based on intelligent terminal according to claim 1, characterized in that, The steps of dynamically reconstructing the presentation mode of the smart terminal interface elements based on the capability characteristic parameters and switching to an operation mode adapted to the current user; enabling multimodal guidance feedback and opening corresponding alternative perception channels based on the user's capability characteristics, including interface voice broadcasting, global real-time subtitles, visual cues, and outputting guidance interaction prompts, further include: When the current user is a visually impaired user, the control engine shuts down visual rendering that consumes more energy than the predetermined energy value, switches to a pure audio user interface, and converts the menu structure and option status into a voice broadcast sequence. When the current user is a hearing-impaired user, the control engine overlays a real-time subtitle layer on the video layer and converts the system prompts into flashing or color changes at the edge of the screen; and / or converts system prompts, call reminders, and operation confirmation information into vibration and tactile signals through communication with the remote control or smart bracelet. When the current user is an elderly person, the control engine simplifies the interface hierarchy and enlarges the click hotspot to a predetermined value; When the current user is someone who has difficulty controlling fine motor skills in their upper limbs, head tracking is used to select the target instead of an input device; the interaction parameters are dynamically adjusted through software algorithms: the click hotspot is enlarged and the cursor movement speed is reduced to a preset speed; The virtual assistant greets you with both voice and text and provides multimodal guidance.

6. The method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals according to claim 1, characterized in that, The acquisition of multimodal heterogeneous input signals from the user in the adapted operation mode includes voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices. The steps for fusing and understanding multimodal input signals to derive the user's operational intent also include: By using a weighted fusion algorithm, the screen space coordinates of the user's gesture location are correlated and parsed with the speech semantics to obtain the user's operation intention, which is then used to achieve fine-grained control.

7. The method for multimodal barrier-free interaction of intelligent agents based on intelligent terminals according to claim 1, characterized in that, It also includes the following steps: An audio activity detector is pre-deployed in the audio decoding layer of the smart terminal to analyze the loudness and spectral characteristics of the audio stream of film and television content in real time. Two types of time windows are identified and marked: an absolutely silent window where the audio loudness meets the first preset condition, and a relatively low-activity window where the speech pauses or background music meet the second preset condition. Each detected window is stamped with a timestamp and confidence label to build a dynamic, insertable time slot queue. The audio activity detection module performs real-time analysis of the audio content to identify natural auditory gaps in film and television narratives; A navigation voice buffer is set up to temporarily store the interface information to be broadcast in a queue without immediately triggering voice output. The natural auditory gap is used as the transmission time slot for navigation voice by the intelligent scheduler. When the natural auditory gap is reached, the navigation text to be broadcast is extracted from the navigation voice buffer and transmitted. Furthermore, semantic compression and speed adjustment are performed on the navigation text to shorten the duration of a single voice message; Before the navigation voice broadcast, a tactile pre-prompt is issued via the user's remote control or smart bracelet at a predetermined time, using cross-modal attention guidance effect to redirect the user's attention to the system navigation; for urgent information that must be responded to immediately, channel switching is achieved through loudness contrast and fade-in / fade-out technology.

8. A multimodal barrier-free interaction device for intelligent agents based on a smart terminal, characterized in that, The device includes: The multimodal perception module is used to collect the user's biometric information when a user is detected entering the perception range of the smart terminal; compare the collected biometric information with pre-stored user identity features to determine the current user's identity; the smart terminal is awakened by triggering methods such as voice wake-up, gesture wake-up and / or wake-up by external auxiliary devices; The capability feature parameter retrieval module is used to retrieve capability feature parameters that are suitable for the current user from a preset accessibility user profile database based on the determined current user identity. The adaptation mode switching module is used to dynamically reconstruct the presentation mode of the smart terminal interface elements according to the capability feature parameters, and switch to the operation mode adapted to the current user; enable multimodal guidance feedback, and open the corresponding alternative perception channels according to the user's capability features, including interface voice broadcast reading, global real-time subtitles, visual prompts, and output guidance interaction prompts. A multimodal interaction module is used to acquire multimodal heterogeneous input signals from the user in the adapted operation mode. The input signals include voice signals, gesture / lip reading visual recognition signals, and / or input signals from external auxiliary devices. The module fuses and understands the multimodal input signals to parse the user's operation intention. The multi-module response and execution module is used to respond to the parsed user operation intent and execute the corresponding business; and to perform perception enhancement processing on media content: real-time analysis of video images to generate visual narration audio, recognition of audio signals to generate dialogue subtitles and environmental sound effects text descriptions, and output through established alternative perception channels to compensate for the user's missing perception information. The data optimization module records data from the interaction process, iteratively optimizes the user's recognition model based on successful / failed interaction samples, and updates the user preference configuration in the accessibility user profile database.

9. A smart terminal, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include methods for performing any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-7.

Images