A medical care-patient bidirectional communication translation method based on multi-modal interaction

Abstract

This invention belongs to the field of medical wristband technology, specifically disclosing a translation method for bidirectional communication between medical staff and patients based on multimodal interaction, including the following steps: Step S1, the multimodal interaction unit receives multimodal input signals; Step S2, the multimodal interaction unit fuses the multimodal input signals to generate comprehensive input information; Step S3, the comprehensive input information is identified and translated based on a medical-specific translation engine; Step S4, the context-aware module analyzes the dialogue content, identifies the scenario, and dynamically adjusts the translation results; Step S5, the adjusted translation results are synchronously output bidirectionally and broadcast via voice. This invention achieves three core breakthroughs in clinical cross-language communication through a translation mechanism combining multimodal signal fusion and context awareness: enabling real-time, accurate, and private barrier-free communication between doctors and patients, reducing the risk of medical errors due to language barriers, improving consultation efficiency and patient satisfaction, and reducing the risk of misdiagnosis.

Description

Technical Field

[0001] This invention belongs to the field of medical wristband technology, and in particular relates to a translation method for two-way communication between medical staff and patients based on multimodal interaction. Background Technology

[0002] Against the backdrop of globalization and an aging population, healthcare institutions face increasing challenges from language or communication barriers with patients. These barriers primarily originate from two main groups: first, foreign language patients, namely expatriates or ethnic minority patients who are not familiar with the local mainstream language; and second, patients with hearing or speech impairments due to illness (such as aphasia after stroke, laryngeal surgery, or neurological diseases), congenital factors, or acquired factors.

[0003] Currently, the main approaches to resolving such communication problems in clinical practice include: Human translation: Relies on professional medical translators or patients' family members. This method has significant drawbacks: translators are not always available, especially in emergency rooms or at night; it is costly; family translators may deliver inaccurate information due to emotional fluctuations or a lack of medical knowledge, especially when conveying bad news or complex treatment plans, which can easily lead to misunderstandings and disputes.

[0004] Simple tools: using pen and paper, notes or charts printed with common phrases. This method is extremely inefficient, unable to handle dynamic and complex doctor-patient dialogues; it is difficult to express precise medical concepts (such as the nature and degree of pain); and it cannot guarantee that the patient fully understands, with information flowing only one way and lacking an effective feedback mechanism.

[0005] General-purpose translation software, such as translation apps on smartphones, while convenient, suffers from significantly insufficient accuracy in recognizing and translating medical terminology (such as the drug name "levofloxacin hydrochloride," the anatomical location "distal radius," and the symptom description "intermittent claudication"). Furthermore, general-purpose software lacks understanding of medical scenarios and cannot distinguish between different language styles used in "consultation" and "preoperative information dissemination." In noisy hospital environments, mobile phone microphones have poor sound pickup, and the transmission of sensitive health information involving patient privacy on public networks and in the cloud poses serious data security risks.

[0006] In conclusion, existing technologies cannot meet the comprehensive requirements of professionalism, real-time performance, accuracy, robustness (anti-interference ability), and privacy in medical scenarios.

[0007] Therefore, the inventors are dedicated to designing a communication and translation method for medical staff and patients to solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to provide a translation method for two-way communication between medical staff and patients based on multimodal interaction, which can realize real-time, accurate, private and barrier-free communication between doctors and patients, significantly reduce the risk of medical errors caused by language barriers, and improve consultation efficiency and patient satisfaction.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A translation method for two-way communication between medical staff and patients based on multimodal interaction includes the following steps: Step S1, Multimodal Input Signal Reception: The multimodal interaction unit receives multimodal input signals from medical staff or patients, wherein the multimodal input signals include at least two of the following: voice signals, touch signals, and bio-signals; Step S2, Multimodal signal fusion and integrated information generation: The multimodal interaction unit fuses the multimodal input signals to generate integrated input information; Step S3, Medical Scenario-Specific Translation: Based on a medical-specific translation engine, the comprehensive input information is identified and translated to generate translation results; Step S4, Context Awareness and Dynamic Adjustment of Expression: The context awareness module analyzes the dialogue content, identifies the context, and dynamically adjusts the translation results accordingly; Step S5, Synchronous Two-Way Output and Broadcast: The adjusted translation results are output synchronously in both directions and broadcast synchronously.

[0010] As an improvement to the medical-patient bidirectional communication translation method based on multimodal interaction of the present invention, step S2 "multimodal interaction unit fuses the multimodal input signal" specifically includes: determining whether the clarity or confidence of the speech signal is lower than a preset threshold; If the content exceeds the preset threshold, then the speech-to-text content will be the primary information. If the value is below the preset threshold, the preset symptom, need icon, or phrase selected by the user via the touchscreen will be used as the dominant information.

[0011] As an improvement to the medical-patient bidirectional communication translation method based on multimodal interaction of the present invention, step S2 "the multimodal interaction unit fuses the multimodal input signal" further includes: When the biosignals indicate that the wearer is in a preset stress state, the system automatically tags the interaction.

[0012] As an improvement to the multimodal interactive bidirectional communication translation method for medical staff and patients in this invention, the "medical-specific translation engine" in step S3 refers to: using a neural network model trained on cross-language medical dialogue corpus to identify and translate medical terms, drug names, and symptom descriptions.

[0013] As an improvement to the multimodal interaction-based bidirectional communication translation method for medical staff and patients of this invention, step S4, "the context-aware module analyzes the dialogue content, identifies the scenario, and dynamically adjusts the translation result accordingly," includes: a. Identify medical scenario types by analyzing keywords in the comprehensive input information or translation results, including consultation, informed consent, preoperative instructions, and daily care; b. Based on the scenario type, call the corresponding terminology database and usage norms to adjust the formality and content structure of the translation results.

[0014] As an improvement to the medical-patient bidirectional communication translation method based on multimodal interaction of the present invention, between step S2 and step S3, a step S2a, sensitivity judgment and processing path selection is also included: the privacy protection unit performs sensitivity judgment on the comprehensive input information; a. If sensitive information is involved, it will be marked, and all subsequent calculations will be performed on the wristband or its paired local device, and the data will not be uploaded; b. If the information is non-sensitive, it will be encrypted and then sent to the cloud server for processing, allowing subsequent steps to access the cloud server.

[0015] As an improvement to the multimodal interactive bidirectional communication translation method for medical staff and patients in this invention, the "bidirectional output" in step S5 specifically refers to visual display, that is, the translation results are displayed simultaneously on two display surfaces of the bidirectional split-screen device with opposite text directions, so that medical staff and patients facing each other can read at the same time; the "voice broadcast" in step S5 specifically refers to broadcasting the translation results to medical staff and patients respectively through headphones or speakers to avoid interference.

[0016] As an improvement to the multimodal interaction-based two-way communication translation method for medical staff and patients of the present invention, the multimodal interaction-based two-way communication translation method for medical staff and patients is executed by a wearable smart bracelet.

[0017] As an improvement to the multimodal interaction-based bidirectional communication translation method for medical staff and patients of this invention, the wearable smart bracelet includes a bracelet body. Adjustment straps and fixing straps are respectively provided at both ends of the bracelet body. The adjustment straps and fixing straps are movably fastened together. The bracelet body is equipped with a bidirectional split-screen display, an information acquisition module for collecting user input signals, and an information output module for outputting information to the user. The bidirectional split-screen display includes a first display surface for medical staff to view and a second display surface for patients to view. The first display surface and the second display surface are arranged opposite to each other to simultaneously display information to both medical staff and patients. Both the information acquisition module and the information output module are electrically connected to a main control circuit board disposed within the bracelet body.

[0018] As an improvement to the multimodal interactive bidirectional communication translation method for medical staff and patients of the present invention, the information acquisition module includes a medical-grade microphone array disposed on the inner wall of the wristband body and at least one biosensor disposed on the inner side of the wristband body. The multiple pickup holes of the medical-grade microphone array are disposed around the bidirectional split-screen display on the wristband body. The detection end of the biosensor is exposed on the side of the wristband body close to the wrist. The information output module includes a speaker disposed in the wristband body, and the sound outlet of the speaker is located on the side wall of the wristband body.

[0019] Compared with existing technologies, this invention provides a multimodal interactive translation method for two-way communication between medical staff and patients. Through a translation mechanism combining multimodal signal fusion and context awareness, it achieves three core breakthroughs in clinical cross-language communication: First, by utilizing the redundancy and complementarity of voice, touch, and biosignals, it can reliably capture key information even in noisy environments or when patients have difficulty expressing themselves, significantly improving the robustness and error tolerance of communication. Second, through a medical-specific translation engine and dynamic adaptation based on context awareness, it ensures the accuracy and contextual relevance of translations from complex medical terminology to everyday nursing language, making professional information transmission both precise and compliant with medical ethics. Third, with privacy protection as a design premise, it adopts a local-first hybrid computing mode. While ensuring patient data security, it constructs a real-time, intuitive, and respectful information transmission loop through a two-way split-screen and synchronous broadcasting system, achieving real-time, accurate, and private barrier-free communication between doctors and patients. This significantly reduces the risk of medical errors caused by language barriers, improves consultation efficiency and patient satisfaction, reduces the risk of misdiagnosis, and strengthens the foundation of trust and cooperation between doctors and patients. Attached Figure Description

[0020] Figure 1 This is a flowchart of the translation method for two-way communication between medical staff and patients based on multimodal interaction, as described in this invention. Figure 2 This is a perspective view of the wearable smart bracelet of the present invention in its unfolded state; Figure 3 yes Figure 2 Enlarged view of point A in the middle; Figure 4 This is another perspective view of the wearable smart bracelet of the present invention in its unfolded state; Figure 5 yes Figure 4 Enlarged view of point B in the middle; Figure 6 yes Figure 4 Enlarged view of point C in the middle; Figure 7 This is a cross-sectional view of the wearable smart bracelet of the present invention; Figure 8 yes Figure 7Enlarged view of point D in the middle; Figure 9 This is a three-dimensional enlarged view of the connecting buckle in this invention; Figure 10 This is a schematic diagram of the wearable smart bracelet of the present invention in the wearing and fastening state; Figure 11 This is an enlarged cross-sectional view of the single connecting buckle on the fixing belt and the single adjusting hole on the adjusting belt in the present invention.

[0021] Illustration: 1. Bracelet body; 11. Button; 12. Sound outlet; 13. Sound pickup hole; 14. Biosensor; 2. Adjustment strap; 21. Strap hole; 22. Anti-slip groove; 23. Adjustment hole; 24. Groove; 3. Fixing strap; 4. Two-way split-screen display; 41. First display surface; 42. Second display surface; 5. Main control circuit board; 6. Battery; 7. Connecting buckle; 71. Fastening groove. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The drawings are for reference and illustration only and do not constitute a limitation on the scope of protection of the present invention.

[0023] See Figure 1 A translation method for two-way communication between medical staff and patients based on multimodal interaction includes the following steps: Step S1, Multimode input signal reception: The wearable smart bracelet's multimodal interaction unit receives multimodal input signals from medical staff or patients, including at least two of the following: voice signals (via microphone), touch signals (user taps preset symptom, need icons or phrases on the bracelet's touchscreen), and bio-signals (sensing physiological state via sensors such as heart rate). The signals are collected by the bracelet's built-in sensors. Step S2, Multimodal signal fusion and integrated information generation: The multimodal interaction unit performs real-time fusion processing on the multimodal input signals locally on the device to generate structured comprehensive input information. The principle is as follows: a. Identify the dominant information source: Determine whether the clarity or confidence level of the speech signal exceeds a preset threshold. If the value exceeds the threshold, the speech-to-text content will be used as the primary information. If the threshold is not met (e.g., noisy environment, weak patient), a redundant channel is activated, using the preset symptom, demand icon or phrase selected by the user via touchscreen as the primary information. If there is no touch input at this time, a prompt "Information unclear, request repeat" is generated. b. Integrated biological context: In parallel, real-time analysis of biological signals (such as abnormally high heart rate). When biological signals indicate that the wearer is in a preset stress state (such as abnormally high heart rate, increased sweat gland secretion), the system automatically labels the interaction with a context label of "high stress" or "potential emergency".

[0024] c. Generate structured information: Bind the above-mentioned dominant information (from voice or touch) with biological context labels, encapsulate it into a comprehensive input information containing content, source type and emotion / state markers, and output it to the next stage.

[0025] Step S2a, Sensitivity Assessment and Processing Path Selection: The privacy protection unit performs sensitivity assessments on comprehensive input information (based on keyword recognition, such as patient identity and medical history): a. If sensitive information such as personal identity or detailed medical history is involved, it will be identified as sensitive information and marked as "local processing". All subsequent calculations will be performed on the wristband or its paired local device (such as a mobile phone), and the data will not be uploaded. b. If the information is general inquiry, daily care, or other non-sensitive information, it will be classified as non-sensitive information. After the information is encrypted, the encrypted integrated input information will be sent to the cloud server for processing, allowing subsequent steps to call the cloud server to obtain more powerful computing capabilities. Step S3, Medical Scenario-Specific Translation: Based on a medical-specific translation engine (local or cloud-based, the "medical ASR engine" and the "medical NMT engine" together constitute the core technology and workflow of the "medical-specific translation engine"), comprehensive input information is identified and translated to generate translation results. The medical-specific translation engine refers to the use of a neural network model trained on cross-language medical dialogue corpus to identify and translate medical terms, drug names, and symptom descriptions to ensure accurate translation results. A medical-specific translation engine was invoked: a. The high-precision automatic speech recognition module converts speech signals (if any) into text; b. The medical terminology enhanced neural machine translation module translates text (or phrases from touch input). This module is trained on a massive cross-lingual medical dialogue corpus to ensure accurate translation of medical terms, drug names, body parts, and symptom descriptions; Step S4: Contextual Awareness and Dynamic Adjustment of Expression The context-aware module analyzes the dialogue content, identifies the context (such as medical consultation, informed consent, etc.), and dynamically adjusts the translation results accordingly. a. Identify medical scenario types by analyzing keywords in the comprehensive input information or translation results, including consultation, informed consent, preoperative instructions, and daily care; b. Based on the scenario type, call the corresponding terminology database and usage norms (e.g., informed consent scenarios require formality and detail) to adjust the formality and content structure of the translation results; Step S5: Synchronous bidirectional output and broadcasting: The adjusted translation results will be output synchronously in both directions, and will be read aloud synchronously. a. Visual display: The translation results are displayed simultaneously on two display surfaces of a two-way split-screen device (such as a wristband) with opposite text directions (e.g., Chinese on the medical staff's end and English on the patient's end), so that medical staff and patients facing each other can read at the same time; b. Voice broadcast: Translation results are broadcast separately to medical staff and patients via headphones or a speaker (such as the speaker on the wristband), avoiding interference. Broadcasts can be set to be triggered on demand and recorded in encrypted storage on the device to ensure privacy.

[0026] Reference Figures 2 to 11 The aforementioned multimodal interaction-based two-way communication translation method between medical staff and patients is executed by a wearable smart bracelet. The wearable smart bracelet includes a bracelet body 1, an adjustment strap 2, and a fixing strap 3. The adjustment strap 2 and the fixing strap 3 are respectively located at both ends of the bracelet body 1, and the adjustment strap 2 and the fixing strap 3 are movably fastened together. The bracelet body 1 is provided with a two-way split-screen display 4, an information acquisition module, and an information output module. The information acquisition module is used to acquire user input signals, and the information output module is used to output information to the user. The two-way split-screen display 4 includes a first display surface 41 and a second display surface 42. The first display surface 41 is for medical staff to view, and the second display surface 42 is for patients to view. The first display surface 41 and the second display surface 42 are arranged opposite to each other to simultaneously display information to medical staff and patients. The information acquisition module and the information output module are both electrically connected to the main control circuit board 5 located in the bracelet body 1.

[0027] Reference Figure 2 , Figure 4 and Figure 7The top of the wristband body 1 is hollowed out, and the wristband body 1 is generally rectangular. Two buttons 11 (e.g., power switch and recording button) are provided on the side wall of the wristband body 1. When the entire wristband is in the unfolded state, the wristband body 1 is arranged along the length direction of the adjustment strap 2 and the fixing strap 3. Multiple arc-shaped anti-slip grooves 22 are spaced apart on the side of the adjustment strap 2 and the fixing strap 3 away from the wrist. Specifically, the anti-slip grooves 22 on the adjustment strap 2 are arc-shaped along the width direction of the adjustment strap 2, and the anti-slip grooves 22 on the fixing strap 3 are arc-shaped along the width direction of the fixing strap 3. The width of the band 3 is arc-shaped. The anti-slip groove 22 on the adjusting band 2 and the anti-slip groove 22 on the fixing band 3 have opposite bending directions. The adjusting band 2 and the fixing band 3 have multiple elliptical grooves 24 spaced apart along their respective lengths on the side close to the wrist. A cylindrical adjusting hole 23 passes through the groove 24 on the upper part of the adjusting band 2. A long strip-shaped strap hole 21 passes through the end of the adjusting band 2 away from the main body 1 of the bracelet. The strap hole 21 allows the free end of the fixing band 3 to pass through. All the grooves 24 on the adjusting band 2 are located in the strap hole. The 21 is movably connected to the main body 1 of the bracelet, and the fixing band 3 and the adjusting band 2 are movably connected by two connecting buckles 7. The two connecting buckles 7 are spaced apart along the length of the fixing band 3. The two connecting buckles 7 are staggered from the grooves 24 on the fixing band 3. The distance between the two connecting buckles 7 is the same as the distance between two adjacent adjusting holes 23 on the adjusting band 2, so that the two connecting buckles 7 can be fastened to any two adjusting holes 23. In this embodiment, the two connecting buckles 7 have the same structure. The cross-section of each connecting buckle 7 is I-shaped. One end of each connecting buckle 7 is embedded in the fixing band 3, and the other end of the connecting buckle 7 protrudes from the side of the fixing band 3 near the wrist, so that the end of the connecting buckle 7 near the wrist can move through any adjusting hole 23. In order to prevent the connecting buckle 7 from detaching from the fixing band 3, an annular fastening groove 71 is provided on the end face of the part of the connecting buckle 7 embedded in the fixing band 3 near the wrist. The two connecting buckles 7 and the fixing band 3 are fixedly connected by injection molding so that the injection plastic can enter the fastening groove 71, making the connection between the connecting buckle 7 and the fixing band 3 more secure.

[0028] Reference Figure 2 , Figure 4 and Figure 7The bidirectional split-screen display 4 is elongated and covers the top of the wristband body 1 to form an inner cavity. The bidirectional split-screen display 4 is set along the length direction of the adjustment band 2 and the fixing band 3 in the unfolded state. The bidirectional split-screen display 4 is a flexible or bendable display screen so that the screen can cover the entire upper surface of the wristband body 1. In this embodiment, the bidirectional split-screen display 4 is a flexible AMOLED display screen or a Micro-LED display screen. These displays have high brightness (>800 nits, ensuring indoor and outdoor visibility), high contrast and low power consumption characteristics. Under software control, the screen of the bidirectional split-screen display 4 can be divided into two display areas (i.e., the first display surface 41 and the second display surface 42). The text direction of the two display areas is opposite. In this way, when medical staff and patients face each other, they can each see the clear and easy-to-read text content facing them. In this embodiment, the first display surface 41 is composed of half of the bidirectional split-screen display 4 near the fixing band 3, and the second display surface 42 is composed of the other half of the bidirectional split-screen display 4 near the adjustment band 2.

[0029] Reference Figure 2 , Figure 4 and Figure 7The main control circuit board 5 is housed within the main body 1 of the wristband. A battery 6 is also housed within the main body 1. The main control circuit board 5 is located between the two-way split-screen display 4 and the battery 6, and the battery 6 is electrically connected to the main control circuit board 5. The main control circuit board 5 integrates a core processor, memory, storage chip, power management chip, and wireless communication module (such as Wi-Fi / Bluetooth). The information acquisition module includes a medical-grade microphone array mounted on the inner wall of the main body 1 and at least one biosensor 14 mounted on the inner side of the main body 1. Multiple pickup holes 13 of the medical-grade microphone array are arranged around the two-way split-screen display 4 on the main body 1, effectively focusing and picking up the voices of medical staff or patients while suppressing environmental noise from other directions (such as instrument sounds and conversations in the ward). This is crucial for noisy medical environments. The medical-grade microphone array consists of multiple microphone units arranged in a linear or circular array. The detection end of the biosensor 14 is exposed on the side of the wristband body 1 that is close to the wrist, to ensure that the detection end of the biosensor 14 can maintain stable and close contact with the skin when the wristband is worn, thereby obtaining accurate signals. The biosensor 14 is an optical heart rate sensor, a skin temperature sensor, or a bioimpedance sensor. Among them, the optical heart rate sensor is used to monitor heart rate, blood oxygen saturation, etc. The information output module includes a speaker set in the wristband body 1. The sound outlet 12 of the speaker is located on the side wall of the wristband body 1. This design can utilize the skin surface of the wrist to form a natural resonance cavity, making the voice broadcast sound clearer and also having a certain directionality, reducing interference to irrelevant people around and protecting communication privacy. All the above-mentioned peripheral devices (such as microphone, speaker, sensor, screen) are ultimately connected to the main control circuit board 5 through cables or connectors, and the main control circuit board 5 provides unified power supply, control and data processing.

[0030] The wearable smart bracelet of this invention no longer relies solely on voice, but intelligently integrates multiple information sources (multimodal) such as voice, touch selection, and biosignals. Through an intelligent decision center, it selects the most appropriate method to achieve accurate and stable two-way communication under various interferences. The entire process can be broken down into three main stages: multimodal input and perception, intelligent context processing and fusion translation, and two-way output and feedback. This bracelet integrates the following functions: (1) Dual-engine speech recognition for medical scenarios: It includes a high-precision automatic speech recognition module and a medical terminology enhanced neural machine translation module. This engine has been trained on a massive amount of cross-language medical dialogue data and can accurately identify and translate medical terms, drug names, body parts and descriptions of common symptoms, ensuring the accuracy of professional terminology translation.

[0031] (2) Multimodal interaction module: It supports input methods that combine voice, touch screen selection (built-in common symptoms, demand icons and phrase library) and biosignals (such as sensing the patient's pain stress response through an integrated heart rate sensor), providing redundant input channels in noisy environments or when the patient's speech is unclear, thus improving robustness.

[0032] (3) Context Awareness and Privacy Protection Module: The built-in algorithm identifies dialogue scenarios (such as consultation, informed consent, preoperative instructions, and daily care) and dynamically adjusts translation strategies and terminology. The device adopts a hybrid local and cloud computing mode, sensitive information can be processed on the device, and dialogue records can be uploaded as needed after encryption, ensuring patient privacy and data security.

[0033] (4) Two-way split-screen display and voice broadcast system: The translation results are displayed in clear large characters on the two-way split-screen display screen 4, which are read by medical staff and patients respectively, and voice broadcast is carried out simultaneously to ensure that the information is transmitted accurately in both directions.

[0034] This wearable smart bracelet enables real-time, accurate, and private two-way communication between doctors and patients, significantly reducing the risk of medical errors caused by language barriers and improving consultation efficiency and patient satisfaction. Specifically designed for medical environments, it features high professionalism, flexible interaction methods, adaptability to complex clinical scenarios, and a focus on data security, thus possessing broad prospects for clinical application.

[0035] The main control circuit board 5 of this wearable smart bracelet is deeply adapted to medical scenarios, featuring a built-in standardized medical terminology database and specialist communication templates (such as those for cardiology and pediatrics). It can quickly access professional translations and supports seamless integration with hospital information systems (HIS) to ensure information accuracy. Simultaneously, encrypted transmission technology ensures data security and prevents patient privacy leaks. In emergency situations, the bracelet is equipped with a one-button call function, allowing patients to quickly contact medical staff and avoid delays in treatment due to communication difficulties.

[0036] This invention significantly reduces the risk of medical errors caused by language barriers and improves consultation efficiency and patient satisfaction through the synergistic optimization of hardware structure and software algorithms. Its strong professionalism, flexible interaction, adaptability to complex clinical scenarios, and emphasis on data security make it widely applicable in various scenarios such as emergency rooms, outpatient clinics, operating rooms, and rehabilitation departments, providing an innovative solution to the challenges of doctor-patient communication.

[0037] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Therefore, any equivalent variations made in accordance with the scope of the patent application of the present invention are still within the scope of the present invention.

Claims

1. A translation method for two-way communication between medical staff and patients based on multimodal interaction, characterized in that, Includes the following steps: Step S1, Multimodal Input Signal Reception: The multimodal interaction unit receives multimodal input signals from medical staff or patients, wherein the multimodal input signals include at least two of the following: voice signals, touch signals, and bio-signals; Step S2, Multimodal signal fusion and integrated information generation: The multimodal interaction unit fuses the multimodal input signals to generate integrated input information; Step S3, Medical Scenario-Specific Translation: Based on a medical-specific translation engine, the comprehensive input information is identified and translated to generate translation results; Step S4, Context Awareness and Dynamic Adjustment of Expression: The context awareness module analyzes the dialogue content, identifies the context, and dynamically adjusts the translation results accordingly; Step S5, Synchronous Two-Way Output and Broadcast: The adjusted translation results are output synchronously in both directions and broadcast synchronously.

2. The translation method for bidirectional communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, The step S2 "multimodal interaction unit fuses the multimodal input signal" specifically includes: determining whether the clarity or confidence of the voice signal is lower than a preset threshold; If the content exceeds the preset threshold, then the speech-to-text content will be the primary information. If the value is below the preset threshold, the preset symptom, need icon, or phrase selected by the user via the touchscreen will be used as the dominant information.

3. The translation method for two-way communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, The step S2, "the multimodal interaction unit fuses the multimodal input signals," further includes: When the biosignals indicate that the wearer is in a preset stress state, the system automatically tags the interaction.

4. The translation method for bidirectional communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, In step S3, the "medical-specific translation engine" refers to the use of a neural network model trained on cross-linguistic medical dialogue corpus to identify and translate medical terms, drug names, and symptom descriptions.

5. The translation method for two-way communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, Step S4, "The context-aware module analyzes the dialogue content, identifies the scene, and dynamically adjusts the translation results accordingly," includes: a. Identify medical scenario types by analyzing keywords in the comprehensive input information or translation results, including consultation, informed consent, preoperative instructions, and daily care; b. Based on the scenario type, call the corresponding terminology database and usage norms to adjust the formality and content structure of the translation results.

6. The translation method for bidirectional communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, Between steps S2 and S3, there is also a step S2a, sensitivity judgment and processing path selection: the privacy protection unit performs sensitivity judgment on the comprehensive input information; a. If sensitive information is involved, it will be marked, and all subsequent calculations will be performed on the wristband or its paired local device, and the data will not be uploaded; b. If the information is non-sensitive, it will be encrypted and then sent to the cloud server for processing, allowing subsequent steps to access the cloud server.

7. The translation method for bidirectional communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, In step S5, "bidirectional output" specifically refers to visual display, where the translation results are simultaneously displayed on two display surfaces of the bidirectional split-screen device with opposite text directions, so that face-to-face medical staff and patients can read them at the same time. In step S5, "voice broadcasting" specifically refers to broadcasting the translation results to medical staff and patients respectively through headphones or speakers to avoid interference.

8. The translation method for two-way communication between medical staff and patients based on multimodal interaction according to claim 1, characterized in that, The multimodal interaction-based two-way communication translation method between medical staff and patients is executed by a wearable smart bracelet.

9. The translation method for bidirectional communication between medical staff and patients based on multimodal interaction according to claim 8, characterized in that, The wearable smart bracelet includes a bracelet body, with an adjustment strap and a fixing strap at both ends of the bracelet body. The adjustment strap and the fixing strap are movably fastened together. The bracelet body is provided with a two-way split-screen display, an information acquisition module for collecting user input signals, and an information output module for outputting information to the user. The two-way split-screen display includes a first display surface for medical staff to view and a second display surface for patients to view. The first display surface and the second display surface are arranged opposite to each other to simultaneously display information to medical staff and patients. The information acquisition module and the information output module are both electrically connected to a main control circuit board disposed in the bracelet body.

10. The translation method for bidirectional communication between medical staff and patients based on multimodal interaction according to claim 9, characterized in that, The information acquisition module includes a medical-grade microphone array disposed on the inner wall of the wristband body and at least one biosensor disposed on the inner side of the wristband body. Multiple pickup holes of the medical-grade microphone array are disposed around the bidirectional split-screen display on the wristband body. The detection end of the biosensor is exposed on the side of the wristband body that is close to the wrist. The information output module includes a speaker disposed inside the wristband body, and the sound outlet of the speaker is located on the side wall of the wristband body.

Images