Clinical intelligent communication system for old people based on multi-mode biological signal fusion

Abstract

The invention discloses an old people clinical intelligent communication system based on multi-modal biological signal fusion, relates to the technical field of smart medical treatment, and solves the problems of insufficient cognitive disorder adaptation, weak privacy protection and the like in traditional human-computer interaction through a multi-modal biological signal fusion technology. The system is composed of a multi-mode biological signal acquisition module, a dynamic fusion processing module, an interaction feedback module and a privacy protection unit. According to the system designed by the invention, the problem of unclear medical treatment expression of the old people is solved, the medical treatment experience of the old people is obviously improved, and new technical enlightenment is provided for intelligent medical treatment.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and more specifically, to a clinical intelligent communication system for the elderly based on multimodal biological signal fusion. Background Art

[0002] With the intensification of global aging, the doctor-patient communication barriers caused by the degradation of audiovisual functions and the decline of cognitive abilities in elderly patients have become increasingly prominent. Research data shows that the misdiagnosis rate caused by poor communication in the elderly outpatient clinics of tertiary hospitals is as high as 14.2%, and the average consultation time is 62% longer than that of younger patients. The existing technical system has significant bottlenecks in solving the following core problems: Physical limitations of environmental noise interference: There is high-intensity background noise in the medical scenario (such as the noise of electrocardiogram monitors, ventilators, etc.), and its spectral characteristics highly overlap with human speech (mainly concentrated in 300Hz - 4kHz). The measured data shows that the average signal-to-noise ratio (SNR) in ordinary consulting rooms is only -5dB to 3dB, and traditional noise reduction means face two major limitations: Single-channel speech enhancement algorithms (such as spectral subtraction) fail under low signal-to-noise ratio conditions. When the noise power exceeds the speech signal, the speech intelligibility (STOI) drops below 0.45 (IEEE Trans. Biomed. Eng. 2023); Fixed beamforming microphone arrays have source localization offset due to the change of patient position. When the azimuth error exceeds 8°, the speech gain drops by 6dB (ICASSP 2024).

[0003] Semantic ambiguity and complexity of pathological correlation: When elderly patients describe their main symptoms, there are generally ambiguous expressions, and they are unable to express discomfort in a timely manner, unable to convey information to nurses in a timely manner, and receive timely treatment. For example: vague pain description, significant emotional interference, etc. Summary of the Invention

[0004] To solve the above problems, the purpose of the present invention is to provide a clinical intelligent communication system for the elderly based on multimodal biological signal fusion, aiming to solve the translation problem when the elderly see a doctor.

[0005] To achieve the above technical purpose, the present application provides a clinical intelligent communication system for the elderly based on multimodal biological signal fusion, including: A multimodal biological signal acquisition module for real-time acquisition of the voice, electroencephalogram, heart rate variability, and limb movement data of the elderly; A dynamic fusion processing module coupled to the acquisition module, which maps multimodal signals into semantic vectors through an adaptive weight allocation algorithm and generates emotion state labels; The interactive feedback module, based on semantic vectors and emotion tags, outputs tactile, visual, and voice guidance instructions through wearable devices, adapting to the characteristics of cognitive ability degradation in the elderly. The privacy protection unit completes the encryption and anonymization processing of biological signals at the local edge node. Preferably, the multimodal biological signal acquisition module includes: The non-contact millimeter-wave radar array is deployed on the ward ceiling to capture limb micro-movements in the 60GHz frequency band. The flexible epidermal electrode patch is integrated into the smart wristband to synchronously collect skin conductance and three-lead electrocardiogram signals.

[0006] Preferably, the dynamic fusion processing module is also used to eliminate the time deviation of cross-modal data through device fingerprint matching and sub-second clock drift compensation, and output the six-level probability distribution of anxiety, pain, and cognitive confusion based on the pre-trained deep neural network DNN.

[0007] Preferably, the training data for the deep neural network DNN training includes: the speech intonation database in specific scenarios of the elderly, and the physiological signal pattern library associated with chronic diseases.

[0008] Preferably, the interactive feedback module further includes: The multi-channel adaptation unit automatically selects the feedback mode according to the user profile: For those with normal cognition: augmented reality glasses project text prompts. For those with hearing impairments: smart gloves generate Morse code tactile sequences. For those with language impairments: the brain-computer interface analyzes the attention focus and drives speech synthesis.

[0009] Preferably, the system further includes AR glasses, and the display logic of the AR glasses includes: When the detected anxiety level ≥ 4, superimpose dynamic breathing guidance animations and ambient light color temperature adjustment. For Alzheimer's disease patients, preferentially display the virtual image of relatives and historical memory fragments.

[0010] Preferably, the system further includes an emergency intervention module for: When the emotion tag is continuously marked as "severe pain" or "acute disturbance of consciousness" for 5 minutes, automatically trigger an alarm at the nursing station and push the patient's real-time vital signs. Before the medical staff responds, adjust the body position to the preset first aid posture through the smart mattress.

[0011] Preferably, the system further includes: a voice emotion analysis module for calculating the anxiety index to quantify the patient's psychological pressure and guide the doctor to adjust the communication strategy. ; wherein, A I represents the anxiety index; represents the standard deviation of the voice fundamental frequency; represents the mean value of the baseline fundamental frequency; represents the standard deviation of the baseline fundamental frequency; ΔG represents the conductance change rate; G max represents the maximum measurable conductance value of the device.

[0012] Preferably, the system is also used to identify sensor anomalies through Mahalanobis distance: ; wherein, D M is the Mahalanobis distance; x is the current sensor data vector; μ is the historical data mean vector; is the inverse matrix of the covariance matrix, and T is the vector transpose operator.

[0013] The present invention discloses the following technical effects: Through the design of the present invention, the problem that the elderly have unclear expressions when seeking medical treatment is solved, the medical treatment experience of the elderly is improved, and new technical inspiration is provided for intelligent medical treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0015] Figure 1 is a schematic structural diagram of the system described in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some, rather than all, of the embodiments of the present application. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application that is required to be protected, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0017] Such as Figure 1As shown in the figure, the present invention provides an elderly clinical intelligent communication system based on multimodal biometric signal fusion, including: A multimodal biometric signal acquisition module, which is used to obtain the voice, electroencephalogram, heart rate variability and limb movement data of the elderly in real time; A dynamic fusion processing module, coupled to the acquisition module, maps the multimodal signals into semantic vectors through an adaptive weight allocation algorithm, and generates emotion state labels; An interaction feedback module, based on the semantic vector and emotion label, outputs tactile, visual and voice guidance instructions through a wearable device, adapting to the characteristics of the elderly's cognitive ability degradation; A privacy protection unit, which completes the encryption and anonymization processing of biometric signals at the local edge node. The multimodal biometric signal acquisition module includes: A non-contact millimeter wave radar array, deployed on the ward ceiling, capturing limb micro-movements in the 60GHz band; A flexible epidermal electrode patch, integrated into the smart wristband, synchronously acquiring skin conductance and three-lead electrocardiogram signals.

[0018] The dynamic fusion processing module is also used to eliminate the time deviation of cross-modal data through device fingerprint matching and sub-second clock drift compensation, and output a six-level probability distribution of anxiety, pain, and cognitive confusion based on a pre-trained deep neural network DNN.

[0019] The training data for training the deep neural network DNN includes: a voice intonation database of the elderly in specific scenarios, and a physiological signal pattern library associated with chronic diseases.

[0020] The interaction feedback module also includes: A multi-channel adaptation unit, which automatically selects a feedback mode according to the user profile: For those with normal cognition: augmented reality glasses project text prompts; For those with hearing impairments: smart gloves generate Morse code tactile sequences; For those with language impairments: a brain-computer interface analyzes the focus of attention and drives speech synthesis.

[0021] The system is also provided with AR glasses, and the display logic of the AR glasses includes: When the detected anxiety level is ≥4, superimpose a dynamic breathing guidance animation and ambient light color temperature adjustment; For Alzheimer's disease patients, preferentially display virtual images of relatives and historical memory fragments.

[0022] Furthermore, the system provided by the present invention specifically includes the following contents: Housing structure and ergonomic design: The main hardware of this system is a credit card-sized smart name tag (hereinafter referred to as "device"), whose outer shell is cast with medical-grade titanium alloy (Ti-6Al-4VELI), and an anti-corrosion coating is formed on the surface through an anodic oxidation process. The curvature radius of the shell is designed as R = 15.7 mm, and the matching degree with the physiological curvature of the human collarbone area reaches 93.2% (based on the ergonomic verification data of ISO13485:2024). The weight of the device is strictly controlled within the range of 18 ± 0.5 g to avoid causing additional burden on the necks of elderly patients.

[0023] In terms of internal layout, the device is divided into three functional areas: Sensing area: Integrated with 4 groups of MEMS microphone arrays, distributed in a rhombus shape (spacing 8.2 mm), achieving 360° sound source localization, and the azimuth resolution ≤ 3°; Computing area: Equipped with a dual-core ARM Cortex-M7 processor with a main frequency of 240 MHz, and 128 KB SRAM is provided for real-time signal processing; Interaction area: Installed with an eccentric rotor motor (vibration amplitude adjustable from 0.8 G to 1.5 G) and a piezoelectric ceramic speaker (frequency response range 300 Hz to 5 kHz).

[0024] Bio-signal sensing components: A flexible printed circuit board (FPCB) is attached to the back of the device, and the following biosensors are built-in: Triaxial accelerometer (ADXL375): Sampling rate 3200 Hz, dynamic range ±200 g, used to detect limb tremors (indicators of Parkinson's disease) Skin conductance sensor (EDA): Using Ag / AgCl electrodes, contact impedance ≤ 10 kΩ, measuring the skin conductance change rate (Formula 1): (1) (When ΔG > 2.5 μS / s, it is determined as emotional fluctuation) In the formula, ΔG is the skin conductance change rate (unit: μS / s), reflecting the intensity of emotional fluctuation; G(t) represents the skin conductance value at time t (unit: μS); T represents the integration time window (default 5 seconds); t0 represents the current calculation start time point. It is a quantitative index for detecting the sudden change of conductivity when the patient is anxious or in pain.

[0025] Temperature sensor (MLX90614): Non-contact infrared temperature measurement, accuracy ±0.2 °C, spatial resolution 4 × 4 pixels.

[0026] Power supply and communication module: The device is powered by a CR2032 button battery, and the battery life is extended through a dynamic power consumption management algorithm (DPPM). In the typical working mode: ‌Standby state‌: current consumption ≤ 15μA (BLE broadcast mode off); ‌Voice activated state‌: Peak current 12mA (automatically reduced after 5 seconds); Emergency signal transmission: Instantaneous current 80mA (triggering GPS positioning and cellular network communication); The wireless communication module supports dual-mode protocols: ‌Bluetooth 5.2‌: Transmit medical data to family members’ mobile phone APP, speed 2Mbps, bit error rate <10 -6 .

[0027] ‌LoRaWAN‌: Establish a low-power connection with the hospital base station within 1km, and the reception strength indicator (R) is used for indoor positioning, with a positioning error of ≤1.5m (Formula 2): (2) Among them, A is the reference signal strength at 1m (-45dBm), n is the environmental attenuation factor (n=2.8 in the actual hospital corridor); d is the distance between the device and the base station (unit: meter); RSSI is the received signal strength indicator value (unit: dBm); Indicates the environmental calibration offset (unit: dB, generally between -3 and +5). represents the human body density attenuation factor, =0.1~0.6; Indicates the population density of the area, 0 to 4 people / m 2 .

[0028] Technical significance: Sub-meter positioning is achieved through the attenuation characteristics of LoRa signals, which is used for location tracking during emergency rescue. Implementation process: Speech enhancement algorithm implementation: The original speech signal collected by the device microphone is first preprocessed: Adaptive beamforming: Generalized sidelobe canceller (GSC) is used to suppress noise in non-target directions, and array gain is ≥8dB; ‌ Improved Wiener Filtering‌: Perform noise power spectrum estimation in the frequency domain (Equation 3): (3) in, represents the enhanced speech signal of the kth frequency point in the mth frame; Y(k,m) represents the frequency domain representation of the original noisy speech; Indicates the estimated value of the noise power spectrum; k indicates the frequency index (corresponding to the 0-8kHz frequency band); m indicates the time frame index. It is used to separate human voice and equipment noise in the frequency domain and improve the voice signal-to-noise ratio (measured improvement of 12dB).

[0029] Nonlinear spectrum enhancement: Harmonic reconstruction is performed on high-frequency components above 6 kHz to improve the clarity of consonants (PESQ score increased by 0.43).

[0030] Medical semantic understanding model: The LSTM neural network deployed in the edge computing layer has the following characteristics: Input layer: 256-dimensional Mel spectrum features, frame length 25 ms, frame shift 10 ms; Hidden layer: Bidirectional LSTM structure, 128 units in each direction, Dropout rate 0.3; Output layer: Connect to the CRF layer to optimize the label sequence and identify 14 types of medical entities (such as drug names, dosages, times); The model training adopts a transfer learning strategy: Pre-training stage: Learn the basic semantic representation in a general medical corpus (5 million sentences); Fine-tuning stage: Use a real doctor-patient dialogue dataset (200 hours of audio, annotation consistency Kappa = 0.91) to optimize domain adaptability; Bio-signal fusion decision: Multimodal data is spatio-temporally aligned through Kalman filtering (Equation 4): (4) where, represents the state estimation vector at time k (including heart rate, exercise intensity, etc.); represents the state transition matrix (describing the time evolution law of bio-signals); represents the control input matrix (the impact of external intervention on the state); represents the control vector (such as manual parameter adjustment by the doctor); represents the Kalman gain (dynamically balancing the weights of prediction and observation); represents the sensor observation value (actual measurement data); represents the observation matrix (sensor accuracy model). It is used to fuse multimodal bio-signals and eliminate errors caused by sensor noise.

[0031] The decision engine performs three types of operations based on the fusion result: Mode switching: When the voice tremor frequency is detected continuously for 5 minutes automatically enable the simplified communication mode; Alarm trigger: If the body temperature > 38°C and the skin conductance suddenly increases, send a level 3 warning signal to the nurse station; Data recording: All bio-signals are encrypted and stored in the HL7 FHIR format for the electronic medical record system to call.

[0032] Clinical Deployment Specification: Device Initialization Configuration: Medical staff activate the device according to the following process: Identity Binding: Scan the QR code on the patient's wristband to automatically associate the medical record number and allergy history in the HIS system; Voiceprint Registration: Guide the patient to read a standard phrase (such as "feeling dizzy today") to establish a personalized voice template; Environmental Calibration: Start 360° sound field mapping in the center of the consultation room to generate a background noise feature fingerprint library.

[0033] Doctor-Patient Collaboration Workflow: The system realizes a full closed-loop interaction during the consultation process: Doctor's Questioning Stage: The chest badge microphone directionally enhances the doctor's sound source (SNR improvement ≥ 10dB); The accelerometer detects the patient's nodding / shaking head movements (recognition accuracy 98.7%); Patient's Statement Stage: The voice emotion analysis module calculates the anxiety index in real time (Formula 5): (5) Where A I represents the anxiety index (threshold 0.7 triggers intervention); represents the standard deviation of the voice fundamental frequency (reflecting the degree of voice tremor); represents the mean of the baseline fundamental frequency (statistical value under the patient's calm state); represents the standard deviation of the baseline fundamental frequency; ΔG represents the conductance change rate; G max represents the maximum measurable conductance value of the device (50 μS); Clinical Significance: Quantify the patient's psychological stress and guide the doctor to adjust the communication strategy. When A I > 0.7, the chest badge vibrates to prompt the doctor to slow down the speaking speed.

[0034] Doctor's Order Issuance Stage: The system automatically extracts key information to generate a voice summary (such as "Nifedipine 10mg bid") Synchronously push the graphic version guidance to the family member's mobile phone, including the drug appearance photo and taking schematic diagram.

[0035] Emergency Handling Mechanism: When the patient long-presses the device for 3 seconds to trigger SOS: Location Broadcasting: Obtain coordinates through hospital Wi-Fi triangulation and GPS, with the error circle radius ≤ 3m.

[0036] Hierarchical Response: Level 1 Response (Fall Detection): Call the nearest nurse station and play pre-recorded guidance voice Level 2 Response (Cardiac Arrest): Link to AED device navigation and unlock the emergency medicine cabinet Post-event Tracing: Automatically export biometric signals 5 minutes before and after the event and generate a PDF report for analysis by the Medical Quality Management Committee.

[0037] Verification and Optimization Methods: Reliability Testing: Complete the following rigorous tests in the EMC laboratory: Radio Frequency Interference Test: At a field strength of 3 V / m, the voice misrecognition rate < 0.2%; Mechanical Shock Test: After withstanding an impact with a peak acceleration of 50 g, the sensor drift ≤ 0.5% FS; Aging Test: Continuously work for 200 hours in an environment of 70°C / 90% RH, and the performance degradation < 3%.

[0038] Clinical Effectiveness Verification: Conduct a double-blind controlled trial (n = 120) in the hospital: Communication Efficiency Index: The median consultation time using this system decreased from 22.3 minutes to 9.7 minutes (p < 0.001, Wilcoxon test) Misdiagnosis Rate Comparison: The clinical pathway deviation rate of the experimental group was 2.3%, significantly lower than 6.8% of the control group (OR = 0.33, 95% CI 0.17 - 0.64); Technology Acceptance: 89.2% of patients over 80 years old considered the device "easy to use" (SUS score ≥ 80).

[0039] Continuous Optimization Strategy: After the system is deployed, it is iteratively upgraded through the following mechanisms: Federated Learning: Train models with local data from each hospital and encrypt and aggregate global parameter updates Anomaly Detection: Identify sensor anomalies through Mahalanobis distance (Formula 6): (6) where D M is the Mahalanobis distance; x is the current sensor data vector (temperature, acceleration, etc.); μ is the historical data mean vector; is the inverse matrix of the covariance matrix (describing parameter correlation), T is the vector transpose operator. Used to detect sensor drift or faults (alert when the threshold > 3.5), that is, trigger the automatic calibration program when D_M > 3.5.

[0040] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0041] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality of" means two or more, unless otherwise specifically defined.

[0042] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. The clinical intelligent communication system for the elderly based on multimodal bio-signal fusion is characterized by: include: ‌Multimodal biosignal acquisition module‌, used to obtain the elderly’s voice, brain wave, heart rate variability and body movement data in real time; A dynamic fusion processing module, coupled to the acquisition module, maps the multimodal signal into a semantic vector through an adaptive weight allocation algorithm and generates an emotional state label; The interactive feedback module outputs tactile, visual and voice guidance instructions through wearable devices based on the semantic vectors and emotion tags, adapting to the characteristics of cognitive deterioration of the elderly; The privacy protection unit completes the encryption and anonymization of biological signals at the local edge node.

2. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 1 is characterized in that: The multimodal biological signal acquisition module comprises: ‌Non-contact millimeter-wave radar array‌, deployed on the ceiling of the ward, captures micro-movements of the limbs in the 60GHz frequency band; Flexible epidermal electrode patch, integrated into a smart wristband, synchronously collects skin conductance and three-lead ECG signals.

3. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 1 is characterized in that: The dynamic fusion processing module is also used to eliminate the time deviation of cross-modal data through device fingerprint matching and sub-second clock drift compensation, and output a six-level probability distribution of anxiety, pain, and cognitive confusion based on a pre-trained deep neural network DNN.

4. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 3 is characterized in that: The training data used for deep neural network (DNN) training includes: a database of speech and intonation in specific scenarios for the elderly, and a library of physiological signal patterns associated with chronic diseases.

5. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 1 is characterized in that: The interactive feedback module further includes: ‌Multi-channel Adaptation Unit‌, automatically selects feedback mode based on user profile: For people with normal cognition: augmented reality glasses project text prompts; Hearing impaired: Smart gloves generate tactile sequences in Morse code; The speech impaired: Brain-computer interfaces parse attention focus and drive speech synthesis.

6. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 5 is characterized in that: The system also includes AR glasses; The display logic of the AR glasses includes: When the anxiety level is detected to be ≥ level 4, dynamic breathing guidance animation and ambient light color temperature adjustment are superimposed; For Alzheimer's patients, virtual images of relatives and historical memory fragments are displayed preferentially.

7. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 1 is characterized in that: The system also includes an Emergency Intervention Module, which is used to: When the emotion tag is marked as "severe pain" or "acute consciousness disorder" for 5 minutes, the nursing station will automatically trigger an alarm and push the patient's real-time vital signs; Before medical staff respond, the smart mattress can be used to adjust the body position to the preset first aid posture.

8. The clinical intelligent communication system for the elderly based on multimodal bio-signal fusion according to claim 7 is characterized in that: The system also includes: a voice emotion analysis module, which is used to calculate the anxiety index to quantify the patient's psychological stress and guide doctors to adjust communication strategies: ； Among them, A I Indicates anxiety index; Indicates the standard deviation of the fundamental frequency of speech; represents the mean of baseline fundamental frequency; represents the standard deviation of the baseline fundamental frequency; ΔG represents the conductivity change rate; G max Indicates the maximum measurable conductivity value of the device.

9. The clinical intelligent communication system for the elderly based on multimodal biosignal fusion according to claim 8, The system is also used to identify sensor anomalies through Mahalanobis distance: ； in, D M is the Mahalanobis distance; x is the current sensor data vector; μ is the historical data mean vector; is the inverse matrix of the covariance matrix, and T is the vector transpose operator.

Images