Multi-mode barrier-free interaction system for safe medication management of visually impaired patients

Abstract

The invention relates to the technical field of intelligent medical treatment, and discloses a multi-mode barrier-free interaction system for safe medication management of visually impaired patients, and the system comprises a central control and decision module, a multi-mode interaction interface module, a state feedback and fusion module, and a safety and emergency module. In the medication management process, touch sense, auditory sense and voice multi-mode interaction is scheduled through the central control and decision module, and user operation is guided; performing dose verification and incompatibility judgment based on a safety rule engine; the state feedback and fusion module encodes a system state into a cross-modal signal and provides consistent feedback; the safety and emergency module is used for monitoring abnormal operation, starting a fusing mechanism to lock an interface and sending out an alarm; the whole system follows a closed-loop principle of guidance-operation-confirmation-feedback, and a visual function is replaced by multi-mode collaboration. According to the invention, the accuracy and operation reliability of medication management are improved, and the safety of the whole process is enhanced.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, specifically to a multimodal barrier-free interactive system for the safe medication management of visually impaired patients. Background Technology

[0002] Smart healthcare is achieved by creating a regional medical information platform for health records and utilizing the most advanced Internet of Things (IoT) technology to enable interaction between patients, medical staff, medical institutions, and medical equipment, gradually achieving informatization.

[0003] Currently, in the process of independent medication management for visually impaired patients, traditional assistive methods are limited in function and cannot provide closed-loop safety assurance through multimodal collaborative mechanisms. When patients make mistakes in drug selection and dosage settings during the operation, it is difficult to detect and correct them in time, which may lead to the safety risk of inaccurate medication.

[0004] Therefore, a multimodal barrier-free interactive system for safe medication management of visually impaired patients is proposed to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal barrier-free interactive system for the safe medication management of visually impaired patients. This system solves the problems mentioned in the background section, such as the inability to provide closed-loop safety confirmation through multimodal collaborative mechanisms and the resulting safety risks of inaccurate medication administration.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal barrier-free interactive system for safe medication management of visually impaired patients, comprising: The central control and decision-making module, as the core of the system, is responsible for scheduling various interaction modes, processing user input, executing medication logic, and activating corresponding feedback channels. Its built-in safety rule engine enables the execution of dose verification, incompatibility judgment, and operation circuit breaking. The multimodal interaction interface module includes a tactile interaction submodule, an auditory interaction submodule, and a voice interaction submodule. The tactile interaction submodule provides physical operation feedback and navigation, the auditory interaction submodule broadcasts private voice commands, and the voice interaction submodule recognizes user voice input. The status feedback and fusion module receives instructions from the central control module and synchronously encodes the operation status and system status into cross-modal feedback signals, ensuring that users can perceive the system status through multiple senses. The safety and emergency module monitors user behavior. When abnormal operations are detected, including continuous incorrect input and attempts to obtain excessive medication, the circuit breaker mechanism is immediately activated, the physical operation interface is locked, and warnings are issued via voice and strong vibration. At the same time, the network assistance process can be initiated.

[0007] Preferably, the central control and decision-making module serves as the system core, responsible for scheduling various interaction modes, processing user input, executing medication logic, and activating corresponding feedback channels. Its built-in safety rule engine enables dose verification, incompatibility judgment, and operation circuit breaking. The specific process of dynamically selecting the dominant interaction mode based on the operation stage is as follows: During the onboarding phase, the auditory submodule is used first to issue voice commands, supplemented by the tactile submodule to provide vibration cues, in order to guide the user's attention. During the operation phase, users are allowed to perform physical operations through the tactile submodule and issue voice commands through the voice submodule. The system receives and processes the input in real time. During the confirmation phase, for critical operations including medication dispensing, the system mandates secondary confirmation, and the secondary confirmation must use an interaction modality different from the initial operation. During the feedback phase, the coordination status feedback and fusion module generate consistent multimodal feedback to notify the user of the operation results; The module also automatically triggers timed tasks based on the medication plan to ensure the timeliness and accuracy of medication administration.

[0008] Preferably, the specific implementation of the guidance phase includes: When the system is activated, including when the preset medication time is reached, the central control module triggers a timed task to play a medication reminder voice through the auditory submodule, which includes the name of the medicine and the medication time. Meanwhile, the tactile submodule generates intermittent pulse vibrations on the physical buttons of the corresponding medicine compartment. The vibration mode is adjustable, including gentle vibrations for regular reminders and strong vibrations for emergency tasks. The auditory submodule uses directional bone conduction speakers and headphones to broadcast voice messages, preventing information leakage and ensuring privacy. The haptic submodule's vibration cues are synchronized with voice commands, forming multi-sensory guidance to help users quickly locate the operating area.

[0009] Preferably, the specific implementation of the operation phase includes: Following the instructions, users perform physical operations through the tactile sub-module, including pressing Braille buttons to select the medicine compartment and rotating the tactile dial to set the dosage. Each rotation of the tactile dial corresponds to a specific dose unit, and the system calculates the accuracy of the set dose using a formula: ; in, For the total set dose, The number of grids the user rotates. Each grid corresponds to a dose unit. When the dose unit corresponding to each grid is 5 mg, it provides a clear tactile click and slight vibration, providing precise physical feedback. Users can also issue voice commands through the voice submodule, including "select antihypertensive drugs" and "set dosage 25 mg", and the voice recognition unit will analyze the commands in real time; During operation, the auditory submodule can play the current operation status for auxiliary confirmation, including announcing "Antihypertensive drug compartment selected" and "Current dose 20 mg"; The central control module monitors the operation process in real time to ensure that the input conforms to the medication logic.

[0010] Preferably, the specific implementation of the confirmation phase includes: For critical operations such as medication dispensing, the system will force a secondary confirmation process after the user completes the first operation. The second confirmation must use an interaction modality different from the first operation. For example, if the first operation is pressing a tactile button, the second confirmation is the voice command "confirm medication" and pressing a tactile confirmation button, but the modality must be different from the first operation. The confirmation options are dynamically generated by the central control module and the confirmation prompts are broadcast through the auditory submodule, including "Please say 'confirm' and press the confirmation button"; After the user completes the secondary confirmation, the security rule engine performs a final verification. If the confirmation is successful and the verification passes, the process continues. If the confirmation fails or the timeout occurs, the security mechanism is triggered. This design reduces the probability of accidental operation and improves safety.

[0011] Preferably, the specific implementation of the feedback phase includes: After any operation is completed, the status feedback and fusion module generates a consistent feedback signal across modes according to the instructions of the central control module; For the "operation successful" status, it is encoded as a combination of a specific sound effect played through the auditory submodule, a short confirmation vibration generated through the tactile submodule, and a "operation completed" voice broadcast. For "operation failure" and "warning" states, the encoding is a combination of warning sound effects, strong vibration, and error description voice. Feedback signals are synchronized in time and consistent in content, ensuring clear perception for the user; The module also supports customized feedback, including user-configurable vibration modes and sound effects to enhance the user experience.

[0012] Preferably, the tactile interaction submodule of the multimodal interactive interface module includes a physical Braille key array, a rotatable tactile dial, a 3D tactile dot matrix display screen, and a multi-mode vibration motor. A physical Braille button array is arranged on the system interface, with Braille markings engraved on the surface of the buttons to provide tactile navigation and operation; A rotatable tactile dial allows for dosage setting, with each rotation producing a click and vibration feedback. The dial resistance is adjustable to suit different users. The 3D haptic dot matrix display can dynamically refresh Braille and simple graphics, including displaying drug information and operation results, and provide visual alternative feedback through dot matrix changes; Multi-mode vibration motors are integrated into buttons and handheld devices, and can generate a variety of vibration modes, including pulse vibration for guidance, continuous vibration for warning, and short vibration for confirmation, to encode different system states.

[0013] Preferably, the auditory interaction submodule of the multimodal interaction interface module includes a directional bone conduction speaker and headphones, as well as an audio processing unit; Bone conduction devices enable the private playback of voice commands and status prompts, with sound transmitted directly through the skull, avoiding environmental noise interference and information leakage; The audio processing unit supports speech synthesis and noise reduction to ensure speech clarity; The submodule also includes an adaptive volume function, which adjusts the volume according to the ambient noise level, including lowering the volume in a quiet environment and raising the volume in a noisy environment; The voice content includes drug details, operating procedures, and safety reminders, using concise and clear language to meet the needs of visually impaired users.

[0014] Preferably, the voice interaction submodule of the multimodal interaction interface module includes a microphone array and a speech recognition and synthesis unit; The microphone array enables the acquisition of user voice input, supports far-field speech recognition and noise suppression, and improves recognition accuracy; The speech recognition unit parses user commands, including querying drug information and confirming operations, and uses natural language processing technology to adapt to colloquial expressions; The speech synthesis unit generates speech feedback, including broadcasting operation results and system status, and the speech speed is adjustable; The submodule also includes a voice wake-up function, which allows users to activate the system with specific commands, including "medicine box", reducing the burden of physical operation.

[0015] Preferably, the safety and emergency module monitors user operation behavior. When abnormal operations are detected, including continuous incorrect input and attempts to obtain excessive medication, a circuit breaker mechanism is immediately activated to lock the physical operation interface. Warnings are issued via voice and strong vibration, and a network assistance process can be initiated simultaneously. Define abnormal operation types, including continuous incorrect input, attempted overdose of medication, excessive operation frequency, and violation of drug incompatibilities; Monitor user action flow and calculate action risk scores in real time to assess the degree of anomaly: ; in, Risk score, This represents the number of erroneous operations. For operating frequency, and The contribution of errors and frequencies is adjusted by setting preset weighting coefficients. The system calculates operational deviations in real time, and triggers rules when the deviation exceeds a preset threshold. When the circuit breaker mechanism is activated, the physical operating interface is immediately locked, including disabling buttons and dials, to prevent further errors. At the same time, the system plays a serious warning voice message, including "Safety warning, operation is prohibited", through the auditory submodule, and generates strong and continuous vibrations through the tactile submodule until the user perceives them. The module also supports online assistance, automatically sending help requests to preset contacts and remote service centers to request manual intervention; The system records all abnormal events and operation logs, enabling subsequent analysis and optimization, and improving long-term security.

[0016] Compared with existing technologies, this invention provides a multimodal barrier-free interactive system for the safe medication management of visually impaired patients, which has the following beneficial effects: 1. In this invention, the central control and decision-making module schedules the multimodal interaction interface module, following the closed-loop principle of guidance-operation-confirmation-feedback. This enables real-time monitoring of drug selection and dosage setting instructions during user operation. Furthermore, the status feedback and fusion module synchronously encodes the operation status into cross-modal feedback signals for immediate prompting and correction. This avoids inaccurate medication use due to missing information and difficulty in confirmation, thereby improving the accuracy and reliability of medication management.

[0017] 2. In this invention, a mandatory secondary confirmation process is adopted. For critical operations such as medication dispensing, users are required to use an interaction mode different from the first operation for verification. This design, combined with the safety rule engine built into the central control and decision-making module, verifies dosage and incompatibilities, which can intercept erroneous operations at the system process level, reduce the medication risk caused by unfriendly operation and lack of error prevention mechanism, and enhance the safety of the entire process.

[0018] 3. In this invention, the bone conduction speaker in the multimodal interaction interface module provides private auditory feedback, and the multimodal vibration motor provides tactile cues, dynamically adapting to different environmental needs to avoid privacy leaks and information interference. At the same time, the security and emergency module monitors abnormal operation behavior in real time and immediately activates the circuit breaker mechanism when the rules are triggered, locking the physical operation interface and issuing a multimodal warning, preventing the continuation of erroneous operations and ensuring the safety and independence of medication management for users in various scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the architecture of a multimodal barrier-free interactive system for safe medication management of visually impaired patients according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 The specific implementation of a multimodal barrier-free interactive system for safe medication management of visually impaired patients is as follows, including: The central control and decision-making module, as the core of the system, is responsible for scheduling various interaction modes, processing user input, executing medication logic, and activating corresponding feedback channels. Its built-in safety rule engine enables the execution of dose verification, incompatibility judgment, and operation circuit breaking. The multimodal interaction interface module includes a tactile interaction submodule, an auditory interaction submodule, and a voice interaction submodule. The tactile interaction submodule provides physical operation feedback and navigation, the auditory interaction submodule broadcasts private voice commands, and the voice interaction submodule recognizes user voice input. The status feedback and fusion module receives instructions from the central control module and synchronously encodes the operation status and system status into cross-modal feedback signals, ensuring that users can perceive the system status through multiple senses. The safety and emergency module monitors user behavior. When abnormal operations are detected, including continuous incorrect input and attempts to obtain excessive medication, the circuit breaker mechanism is immediately activated, the physical operation interface is locked, and warnings are issued via voice and strong vibration. At the same time, the network assistance process can be initiated.

[0022] The central control and decision-making module, as the core of the system, is responsible for scheduling various interaction modes, processing user input, executing medication logic, and activating corresponding feedback channels. Its built-in safety rule engine performs dose verification, incompatibility checks, and operation circuit breakers. The specific process of dynamically selecting the dominant interaction mode based on the operation stage is as follows: During the onboarding phase, the auditory submodule is used first to issue voice commands, supplemented by the tactile submodule to provide vibration cues, in order to guide the user's attention. During the operation phase, users are allowed to perform physical operations through the tactile submodule and issue voice commands through the voice submodule. The system receives and processes the input in real time. During the confirmation phase, for critical operations including medication dispensing, the system mandates secondary confirmation, and the secondary confirmation must use an interaction modality different from the initial operation. During the feedback phase, the coordination status feedback and fusion module generate consistent multimodal feedback to notify the user of the operation results; The module also automatically triggers scheduled tasks based on the medication plan to ensure the timeliness and accuracy of medication administration; Dynamically selecting the dominant interaction mode is achieved by calculating modal priority scores: ; in, Modal priority score, For operational complexity, For user preference historical data, and These are weighting coefficients to balance complexity and user habits; The specific steps include: the system first calculates the operation complexity based on the preset number of steps for the current operation to be performed and the data collected by the environmental noise sensor. Simultaneously, the system retrieves the user's historical interaction records from local storage, analyzes the frequency of their selection of tactile, auditory, and voice modalities, and generates user preference data. The central control module loads weighting coefficients according to a preset strategy. and Substitute into the formula to calculate the current situation The system pre-stores priority thresholds for each interaction modality. When the calculated value... When the value exceeds the activation threshold of a certain mode, the central control module instructs that mode to become the dominant interaction mode in the next operation phase.

[0023] The specific implementation of the bootstrapping phase includes: When the system is activated, including when the preset medication time is reached, the central control module triggers a timed task to play a medication reminder voice through the auditory submodule, which includes the name of the medicine and the medication time. Meanwhile, the tactile submodule generates intermittent pulse vibrations on the physical buttons of the corresponding medicine compartment. The vibration mode is adjustable, including gentle vibrations for regular reminders and strong vibrations for emergency tasks. The auditory submodule uses directional bone conduction speakers and headphones to broadcast voice messages, preventing information leakage and ensuring privacy. The haptic submodule's vibration cues are synchronized with voice commands, forming multi-sensory guidance to help users quickly locate the operating area; Vibration intensity is adaptively adjusted based on user distance, calculated using a formula: ; in, The actual vibration intensity Basic strength value, For reference distance, The user's current distance; The specific steps include: the system uses infrared and ultrasonic distance sensors integrated into the device to monitor the distance between the user's hands and body and the operating interface in real time, as... The central control module reads the preset basic vibration intensity for the current alert type from the configuration file. and optimal operating reference distance Substituting the above parameters into the formula, the actual vibration intensity to be output can be calculated in real time. The control circuit in the tactile submodule is based on the calculated... The value dynamically adjusts the current and voltage signals driving the vibration motor, thereby achieving adaptive adjustment of vibration intensity according to the user's distance, ensuring that the user can perceive tactile cues at different distances.

[0024] The specific implementation of the operation phase includes: Following the instructions, users perform physical operations through the tactile sub-module, including pressing Braille buttons to select the medicine compartment and rotating the tactile dial to set the dosage. Each rotation of the tactile dial corresponds to a specific dose unit, and the system calculates the accuracy of the set dose using a formula: ; in, For the total set dose, The number of grids the user rotates. Each grid corresponds to a dose unit. When the dose unit corresponding to each grid is 5 mg, it provides a clear tactile click and slight vibration, providing precise physical feedback. The value is dynamically retrieved from the built-in drug information database and loaded into the drive unit of the haptic interaction submodule by the central control and decision-making module based on the preset specifications of the currently selected drug. For a drug with a specification of 25 mg per tablet, the system can set... =5 mg, meaning the dose increases or decreases by 5 mg for each rotation. Users can also issue voice commands through the voice submodule, including "select antihypertensive drugs" and "set dosage 25 mg", and the voice recognition unit will analyze the commands in real time; During operation, the auditory submodule can play the current operation status for auxiliary confirmation, including announcing "Antihypertensive drug compartment selected" and "Current dose 20 mg"; The central control module monitors the operation process in real time to ensure that the input conforms to the medication logic.

[0025] The specific implementation of the confirmation phase includes: For critical operations such as medication dispensing, the system will force a secondary confirmation process after the user completes the first operation. The second confirmation must use an interaction modality different from the first operation. For example, if the first operation is pressing a tactile button, the second confirmation is the voice command "confirm medication" and pressing a tactile confirmation button, but the modality must be different from the first operation. The confirmation options are dynamically generated by the central control module and the confirmation prompts are broadcast through the auditory submodule, including "Please say 'confirm' and press the confirmation button"; After the user completes the secondary confirmation, the security rule engine performs a final verification. If the confirmation is successful and the verification passes, the process continues. If the confirmation fails or the timeout occurs, the security mechanism is triggered. This design reduces the probability of accidental operation and improves safety; Confirmation timeout period is adaptively calculated based on operational risk level: ; in, This is the actual timeout time. Based on the timeout value, Risk level, For adjustment coefficients; Risk level The safety rule engine determines the risk level based on multi-layered rule mapping: First, a base risk value is assigned based on the inherent risk levels marked in the drug database, including high-risk, medium-risk, and routine. Second, a weighted correction is made based on the percentage deviation between the dosage set in this operation and the standard dosage. Finally, fine-tuning is performed by referring to the user's average error rate in similar operations recently, generating a final risk value between 0 and 1. Value, adjustment coefficient This is an empirical constant used to control the impact of risk level on timeout duration. The specific steps include: when secondary confirmation is required, the safety rule engine queries and calculates the risk level of the current operation from a predefined risk mapping table based on the drug type, dosage, and the user's historical error rate. The central control module calls the preset basic timeout value. and adjustment coefficient Substituting the above parameters into the formula, the actual timeout allowed for this confirmation process can be calculated. System startup countdown, when the user... If confirmation is not completed within the time limit, it is determined that the confirmation has timed out, and the security mechanism is immediately triggered to lock the current operation and issue a warning.

[0026] The specific implementation of the feedback phase includes: After any operation is completed, the status feedback and fusion module generates a consistent feedback signal across modes according to the instructions of the central control module; For the "operation successful" status, it is encoded as a combination of a specific sound effect played through the auditory submodule, a short confirmation vibration generated through the tactile submodule, and a "operation completed" voice broadcast. For "operation failure" and "warning" states, the encoding is a combination of warning sound effects, strong vibration, and error description voice. Feedback signals are synchronized in time and consistent in content, ensuring clear perception for the user; The module also supports customized feedback, including user-configurable vibration modes and sound effects to enhance the user experience; Feedback consistency is evaluated by calculating the synchronization score of the multimodal signals:

[0027] in, For synchronized scores, For the number of signals, and These are the trigger times for the i-th auditory and tactile signals, respectively. Maximum allowable delay; The specific steps include: after generating a set of multimodal feedback instructions, the status feedback and fusion module records the system timestamp of each instruction being sent to the corresponding hardware submodule, which serves as the trigger time for each signal. and The module reads the maximum allowed time delay for this type of feedback from the preset configuration. The collected time data and Substitute into the formula to calculate the synchronization score for this feedback. The central control module has a synchronization threshold; when the calculated threshold is reached... If the value is below this threshold, the feedback synchronization is deemed poor. The module will record this anomaly and compensate for it by fine-tuning the timing of the instruction transmission and inserting a small delay when generating the feedback instruction next time, in order to optimize the simultaneity of the multimodal signals.

[0028] The tactile interaction sub-module of the multimodal interaction interface module includes a physical Braille button array, a rotatable tactile dial, a 3D tactile dot matrix display screen, and a multi-mode vibration motor. A physical Braille button array is arranged on the system interface, with Braille markings engraved on the surface of the buttons to provide tactile navigation and operation; A rotatable tactile dial allows for dosage setting, with each rotation producing a click and vibration feedback. The dial resistance is adjustable to suit different users. The 3D haptic dot matrix display can dynamically refresh Braille and simple graphics, including displaying drug information and operation results, and provide visual alternative feedback through dot matrix changes; Multi-mode vibration motors are integrated into buttons and handheld devices and can generate multiple vibration modes, including pulse vibration for guidance, continuous vibration for warning, and short vibration for confirmation, to encode different system states; The torque resistance of the tactile dial is adaptively adjusted based on the user's operating force, calculated using a formula: ; in, This is the actual torque value. Based on the base torque, The operational force applied to the user, This is the sensitivity coefficient; The specific steps include: integrating a pressure sensor on the rotating axis of the tactile dial to detect the torque and tangential force applied by the user when rotating the dial in real time, and using the measured value as the user's operating force. The central control module retrieves the corresponding basic torque value from memory based on the currently set task. and preset sensitivity coefficient ,Will , and Substitute the values ​​into the formula to calculate the required resistance torque value in real time. The stepper motor and resistance generator in the haptic submodule are based on the calculated... The output torque is dynamically adjusted so that when the user applies greater force, the resistance of the dial increases proportionally and appropriately, providing a clearer sense of stage and control feedback, and preventing over-rotation.

[0029] The auditory interaction submodule of the multimodal interaction interface module includes a directional bone conduction speaker and headphones, as well as an audio processing unit; Bone conduction devices enable the private playback of voice commands and status prompts, with sound transmitted directly through the skull, avoiding environmental noise interference and information leakage; The audio processing unit supports speech synthesis and noise reduction to ensure speech clarity; The submodule also includes an adaptive volume function, which adjusts the volume according to the ambient noise level, including lowering the volume in a quiet environment and raising the volume in a noisy environment; The voice content includes drug details, operating procedures, and safety reminders, using concise and clear language to meet the needs of visually impaired users.

[0030] The voice interaction submodule of the multimodal interaction interface module includes a microphone array and a speech recognition and synthesis unit; The microphone array enables the acquisition of user voice input, supports far-field speech recognition and noise suppression, and improves recognition accuracy; The speech recognition unit parses user commands, including querying drug information and confirming operations, and uses natural language processing technology to adapt to colloquial expressions; The speech synthesis unit generates speech feedback, including broadcasting operation results and system status, and the speech speed is adjustable; The submodule also includes a voice wake-up function, which allows users to activate the system with specific commands, including "medicine box", reducing the burden of physical operation; Speech recognition accuracy is optimized through environmental signal-to-noise ratio: ; in, For actual recognition accuracy, To achieve the maximum theoretical accuracy, For environmental noise levels, The strength of the voice signal; Maximum theoretical accuracy The method for obtaining the accuracy is as follows: Before leaving the factory and during system initialization, the speech recognition engine is tested in a standard quiet experimental environment using a standard test speech set that covers the system's preset command vocabulary. The average recognition accuracy obtained is the device's accuracy under the current firmware version. The value is stored in the device's non-volatile memory and is estimated in real time. When the value is lower than the system's preset reliability threshold, active noise cancellation gain enhancement is triggered, and the user is prompted by the auditory submodule to "The environment is noisy, please move closer to the microphone and repeat the command"; The specific steps include: In standby and working states, the voice interaction submodule continuously collects ambient sound through the microphone array, separates the ambient noise components through the signal processing unit, and calculates its average energy value as... When user voice activity is detected, the average energy value of that segment of the voice signal is calculated as... The system reads the maximum theoretical accuracy of the current recognition model under ideal quiet conditions from the performance parameter library. , will be acquired in real time and and Substitute these values ​​into the formula to estimate the expected recognition accuracy under the current environment. ,when If the value is below the preset usable threshold, the speech recognition unit automatically switches to a more noise-resistant recognition model and triggers the auditory submodule to play a prompt that says "The environment is noisy, please move closer to speak," thereby dynamically optimizing the recognition performance.

[0031] The status feedback and fusion module receives instructions from the central control module and synchronously encodes the operational and system states into cross-modal feedback signals, ensuring that the user perceives the system state through multiple senses. This process includes: Define feedback status types, including operation success, operation failure, system warning, and process guidance; Receive instructions from the central control module, which include the current status type and associated parameters that need to be fed back; Based on the state type, the corresponding multimodal feedback signal combination is called from the preset encoding rule library. The combination includes at least one tactile code, one auditory code, and one voice broadcast text. Generate cross-modal feedback signals: For tactile signals, the multi-mode vibration motor in the tactile interaction submodule is driven to generate a vibration mode that matches the state, including a short single vibration to indicate successful operation and a strong continuous vibration to indicate a warning. For auditory signals, the auditory interaction submodule is driven to play predefined sound effects that match the state; For voice signals, the auditory interaction submodule and the voice interaction submodule are driven to synthesize and play the corresponding broadcast text; Execute multimodal signal synchronous output and ensure time alignment of tactile, auditory and voice signals at the user's perception level through unified timing control; After the feedback is completed, the module sends a feedback completion signal to the central control module and enters standby mode.

[0032] The safety and emergency module monitors user behavior. When abnormal operations are detected, including continuous incorrect input and attempts to obtain excessive medication, the circuit breaker mechanism is immediately activated, locking the physical interface and issuing warnings via voice and strong vibration. Simultaneously, the process of initiating a network emergency call can be initiated, including: Define abnormal operation types, including continuous incorrect input, attempted overdose of medication, excessive operation frequency, and violation of drug incompatibilities; Monitor user action flow and calculate action risk scores in real time to assess the degree of anomaly: ; in, Risk score, This represents the number of erroneous operations. For operating frequency, and The contribution of errors and frequencies is adjusted by setting preset weighting coefficients. Weighting coefficient and The settings are based on the system's trade-offs regarding security: Focusing on the severity of operational errors, the value is set based on the probability of various errors leading to adverse consequences in historical data analysis, with higher weights assigned to errors that directly lead to overdose risk. This focuses on abnormal operation rhythms, aiming to reduce high-frequency invalid inputs caused by user panic and tentative operations. A typical initialization configuration is as follows: =0.7, =0.3, and authorized caregivers are allowed to adjust it within a certain range through the management interface to suit the habits of specific users. The risk score R is calculated cumulatively on the basis of the current medication task. The system calculates operational deviations in real time, and triggers rules when the deviation exceeds a preset threshold. When the circuit breaker mechanism is activated, the physical operating interface is immediately locked, including disabling buttons and dials, to prevent further errors. At the same time, the system plays a serious warning voice message, including "Safety warning, operation is prohibited", through the auditory submodule, and generates strong and continuous vibrations through the tactile submodule until the user perceives them. The module also supports online assistance, automatically sending help requests to preset contacts and remote service centers to request manual intervention; The system records all abnormal events and operation logs, enabling subsequent analysis and optimization, and improving security.

[0033] The operation steps of a multimodal barrier-free interactive system for safe medication management of visually impaired patients are as follows: Step 1: System Activation and Multimodal Booting When the system starts, the central control and decision-making module is automatically activated according to the preset medication plan. First, this module schedules the auditory interaction submodule in the multimodal interaction interface module to play private voice commands through directional bone conduction speakers and headphones, including "It's time to take your medication at 8:00 AM, please prepare to take your blood pressure medication." At the same time, the tactile interaction submodule generates intermittent pulse vibrations on the physical buttons of the corresponding medication compartment. The vibration mode is dynamically adjusted by the central control module according to the urgency of the task. The core of the guidance phase is to form a multisensory guidance path through the synchronous output of auditory and tactile modalities, helping visually impaired users quickly locate the operation area and avoid relying on visual information. The central control module monitors environmental factors in real time and uses the status feedback and fusion module to ensure the consistency of the guidance signal, including adaptive calculation of vibration intensity based on user distance to enhance the effectiveness of the prompts.

[0034] Step 2: User interaction and modal switching: After the initial setup, the user enters the operation phase. The central control and decision-making module dynamically selects the dominant interaction mode based on the complexity of the operation: the user can select the medication compartment through the physical Braille button array of the tactile interaction submodule, and rotate the tactile dial to set the dosage, with each rotation accompanied by a tactile "click" and vibration feedback; and issue voice commands through the microphone array of the voice interaction submodule, including "set dosage 25 mg". During the operation, the central control module processes the input data in real time, including calculating the dosage accuracy using formulas and comparing the deviation with the required dosage. When the deviation exceeds the limit, the system triggers a reset prompt. Modality switching is based on priority scores. When the environment is noisy, the tactile mode is prioritized to reduce interference. The operation phase emphasizes active user participation. The central control module uses a safety rule engine to initially verify the compliance of the operation and prevent obvious errors.

[0035] Step 3: Mandatory Secondary Confirmation and Security Verification For critical operations such as medication dispensing, the system forces a confirmation phase. The central control module requires the user to use a different interaction modality than the initial operation for secondary confirmation. For example, if the initial operation involves pressing a tactile button, the secondary confirmation involves the voice command "Confirm Medication Dispensing" and pressing the tactile confirmation button. The confirmation options are dynamically generated by the central control module and announced through the auditory interaction submodule. After the user responds, the safety rule engine performs deep verification, including dosage verification, incompatibility judgment, and operation history analysis. The verification process adaptively calculates the timeout based on the risk level, including extending the confirmation window for high-risk drugs. If the verification passes, the system continues to execute; if it fails or times out, the circuit breaker mechanism of the safety and emergency module is triggered. This step reduces the probability of misoperation through heterogeneous modal confirmation and rule engine dual protection.

[0036] Step 4: Multimodal state feedback and closed-loop control: After the operation is completed, the status feedback and fusion module generates a consistent cross-modal feedback signal according to the instructions of the central control module. This includes: for the "operation successful" status, it is encoded as a combination of a specific sound effect, a short confirmation vibration, and a "medication retrieved successfully" voice broadcast; for the "operation failed" status, it is encoded as a warning sound effect, a strong vibration, and an error description. The feedback signal ensures time consistency through synchronization score calculation. When the delay exceeds the limit, the timing is automatically adjusted to form a closed-loop control in the feedback phase. Users can clearly perceive the results based on multimodal signals and query the status through the voice interaction submodule when necessary. The central control module records the operation log to optimize subsequent interactions and improve the system's adaptability.

[0037] Step 5: Anomaly Monitoring and Emergency Circuit Breaker Handling: Throughout the process, the safety and emergency module monitors the user's operation flow in real time, detecting abnormal behaviors including continuous incorrect input, excessive medication dispensing, and excessive operation frequency. The monitoring is based on risk score calculation, including a weighted assessment combining the number of errors and frequency. Once an abnormality triggers a rule, the module immediately activates the circuit breaker mechanism: locking the physical operation interface, playing a warning voice through the auditory submodule, and generating a strong vibration through the tactile submodule. At the same time, the module can initiate a network help process, automatically sending help information to preset contacts. Emergency handling ensures rapid intervention before the risk escalates, protecting user safety. The system analyzes logs afterward to optimize rule thresholds.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal barrier-free interactive system for safe medication management of visually impaired patients, characterized in that, include: The central control and decision-making module, as the core of the system, is responsible for scheduling various interaction modes, processing user input, executing medication logic, and activating corresponding feedback channels. Its built-in safety rule engine enables the execution of dose verification, incompatibility judgment, and operation circuit breaking. The multimodal interaction interface module includes a tactile interaction submodule, an auditory interaction submodule, and a voice interaction submodule. The tactile interaction submodule provides physical operation feedback and navigation, the auditory interaction submodule broadcasts private voice commands, and the voice interaction submodule recognizes user voice input. The status feedback and fusion module receives instructions from the central control module and synchronously encodes the operation status and system status into cross-modal feedback signals, ensuring that users can perceive the system status through multiple senses. The safety and emergency module monitors user behavior. When abnormal operations are detected, including continuous incorrect input and attempts to obtain excessive medication, the circuit breaker mechanism is immediately activated, the physical operation interface is locked, and warnings are issued via voice and strong vibration. At the same time, the network assistance process can be initiated.

2. The multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 1, characterized in that, The central control and decision-making module, as the core of the system, is responsible for scheduling various interaction modes, processing user input, executing medication logic, and activating corresponding feedback channels. Its built-in safety rule engine enables dose verification, incompatibility judgment, and operation circuit breaking. The specific process of dynamically selecting the dominant interaction mode based on the operation stage is as follows: During the onboarding phase, the auditory submodule is used first to issue voice commands, supplemented by the tactile submodule to provide vibration cues, in order to guide the user's attention. During the operation phase, users are allowed to perform physical operations through the tactile submodule and issue voice commands through the voice submodule. The system receives and processes the input in real time. During the confirmation phase, for critical operations including medication dispensing, the system mandates secondary confirmation, and the secondary confirmation must use an interaction modality different from the initial operation. During the feedback phase, the coordination status feedback and fusion module generate consistent multimodal feedback to notify the user of the operation results; The module also automatically triggers timed tasks based on the medication plan to ensure the timeliness and accuracy of medication administration.

3. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 2, characterized in that, The specific implementation of the bootstrapping phase includes: When the system is activated, including when the preset medication time is reached, the central control module triggers a timed task to play a medication reminder voice through the auditory submodule, which includes the name of the medicine and the medication time. Meanwhile, the tactile submodule generates intermittent pulse vibrations on the physical buttons of the corresponding medicine compartment. The vibration mode is adjustable, including gentle vibrations for regular reminders and strong vibrations for emergency tasks. The auditory submodule uses directional bone conduction speakers and headphones to broadcast voice messages, preventing information leakage and ensuring privacy. The haptic submodule's vibration cues are synchronized with voice commands, forming multi-sensory guidance to help users quickly locate the operating area.

4. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 2, characterized in that, The specific implementation of the operation phase includes: Following the instructions, users perform physical operations through the tactile sub-module, including pressing Braille buttons to select the medicine compartment and rotating the tactile dial to set the dosage. Each rotation of the tactile dial corresponds to a specific dose unit, and the system calculates the accuracy of the set dose using a formula: ； in, For the total set dose, The number of grids the user rotates. Each grid corresponds to a dose unit. When the dose unit corresponding to each grid is 5 mg, it provides a clear tactile click and slight vibration, providing precise physical feedback. Users can also issue voice commands through the voice submodule, including "select antihypertensive drugs" and "set dosage 25 mg", and the voice recognition unit will analyze the commands in real time; During operation, the auditory submodule can play the current operation status for auxiliary confirmation, including announcing "Antihypertensive drug compartment selected" and "Current dose 20 mg"; The central control module monitors the operation process in real time to ensure that the input conforms to the medication logic.

5. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 2, characterized in that, The specific implementation of the confirmation phase includes: For critical operations such as medication dispensing, the system will force a secondary confirmation process after the user completes the first operation. Secondary confirmation must use an interaction modality different from the first operation. For example, if the first operation is pressing a tactile button, then the secondary confirmation is the voice command "confirm medication" and pressing a tactile confirmation button, but the modality must be different from the first operation. The confirmation options are dynamically generated by the central control module and the confirmation prompts are broadcast through the auditory submodule, including "Please say 'confirm' and press the confirmation button"; After the user completes the secondary confirmation, the security rule engine performs a final verification. If the confirmation is successful and the verification passes, the process continues. If the confirmation fails or a timeout occurs, the security mechanism is triggered.

6. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 2, characterized in that, The specific implementation of the feedback phase includes: After any operation is completed, the status feedback and fusion module generates a consistent feedback signal across modes according to the instructions of the central control module; For the "operation successful" status, it is encoded as a combination of a specific sound effect played through the auditory submodule, a short confirmation vibration generated through the tactile submodule, and a "operation completed" voice broadcast; For "operation failure" and "warning" states, the encoding is a combination of warning sound effects, strong vibration, and error description voice. Feedback signals are synchronized in time and consistent in content, ensuring clear perception for the user; The module also supports customized feedback, including user-configurable vibration modes and sound effects.

7. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 1, characterized in that, The tactile interaction submodule of the multimodal interaction interface module includes a physical Braille button array, a rotatable tactile dial, a 3D tactile dot matrix display screen, and a multi-mode vibration motor. A physical Braille button array is arranged on the system interface, with Braille markings engraved on the surface of the buttons to provide tactile navigation and operation; A rotatable tactile dial allows for dosage setting, with each rotation producing a click and vibration feedback. The dial resistance is adjustable to suit different users. The 3D haptic dot matrix display can dynamically refresh Braille and simple graphics, including displaying drug information and operation results, and provide visual alternative feedback through dot matrix changes; Multi-mode vibration motors are integrated into buttons and handheld devices, and can generate a variety of vibration modes, including pulse vibration for guidance, continuous vibration for warning, and short vibration for confirmation, to encode different system states.

8. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 1, characterized in that, The auditory interaction submodule of the multimodal interaction interface module includes a directional bone conduction speaker and headphones, as well as an audio processing unit; Bone conduction devices enable the private playback of voice commands and status prompts, with sound transmitted directly through the skull, avoiding environmental noise interference and information leakage; The audio processing unit supports speech synthesis and noise reduction to ensure speech clarity; The submodule also includes an adaptive volume function, which adjusts the volume according to the ambient noise level, including lowering the volume in a quiet environment and raising the volume in a noisy environment; The voice content includes drug details, operating procedures, and safety reminders, using concise and clear language to meet the needs of visually impaired users.

9. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 1, characterized in that, The voice interaction submodule of the multimodal interaction interface module includes a microphone array and a speech recognition and synthesis unit; The microphone array enables the acquisition of user voice input, supports far-field speech recognition and noise suppression, and improves recognition accuracy; The speech recognition unit parses user commands, including querying drug information and confirming operations, and uses natural language processing technology to adapt to colloquial expressions; The speech synthesis unit generates speech feedback, including broadcasting operation results and system status, and the speech speed is adjustable; The submodule also includes a voice wake-up function, which allows users to activate the system with specific commands, including "medicine box", reducing the burden of physical operation.

10. A multimodal barrier-free interactive system for safe medication management of visually impaired patients according to claim 1, characterized in that, The safety and emergency module monitors user behavior. When abnormal operations are detected, including continuous incorrect input and attempts to obtain excessive medication, a circuit breaker mechanism is immediately activated, locking the physical interface and issuing warnings via voice and strong vibration. Simultaneously, the process of initiating a network assistance procedure includes: Define abnormal operation types, including continuous incorrect input, attempted overdose of medication, excessive operation frequency, and violation of drug incompatibilities; Monitor user action flow and calculate action risk scores in real time to assess the degree of anomaly: ； in, Risk score, This represents the number of erroneous operations. For operating frequency, and Preset weighting coefficients; The system calculates operational deviations in real time, and triggers rules when the deviation exceeds a preset threshold. When the circuit breaker mechanism is activated, the physical operating interface is immediately locked, including disabling buttons and dials, to prevent further errors. At the same time, a serious warning voice is played through the auditory submodule, including "Safety warning, operation is prohibited", and a strong and continuous vibration is generated through the tactile submodule until the user perceives it; The module also supports online assistance, automatically sending help requests to preset contacts and remote service centers to request manual intervention.

Images