Multi-modal data fusion identification response method based on walking-aid crutch umbrella

Abstract

The invention relates to the technical field of intelligent processing of data signals, and provides a multi-modal data fusion recognition response method based on a walking-aid crutch umbrella, which comprises the following steps: S1, configuring a multi-modal perception recognition layer, an intelligent controller, a power module, a communication module, a positioning and posture sensing module, a first-aid medicine bin module, a vital sign detection module and a voice interaction module are integrally arranged on a crutch umbrella body, and taking guidance information of different medicine categories is stored in a storage unit; three-level alarm threshold values of heart rate, oxyhemoglobin saturation and skin electric response signals; a third-level response decision execution scheme corresponding to the third-level alarm threshold; the method comprises the steps of S1, event triggering, S2, data fusion and correlation analysis based on event triggering, S3, motion artifact elimination and physiological feature extraction, and S4, hierarchical response decision and dynamic upgrading and downgrading are executed on multi-modal perception recognition. The problems that an existing walking-aid crutch umbrella depends on active operation of a user, unconscious states cannot be recognized, and response efficiency is affected are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent data signal processing technology, and in particular to a multimodal data fusion recognition and response method based on a walking cane umbrella. Background Technology

[0002] The walking stick umbrella is an outdoor product that combines the functions of a walking stick for support and an umbrella for sun and rain protection. It is widely used by the elderly and people with mobility impairments in their daily travels. With the increasing aging of the population, the safety needs of the elderly when going out are rising, especially for the elderly with chronic diseases such as heart disease and high blood pressure. When they suddenly fall ill outdoors, they often miss the best time for treatment because they cannot get first aid guidance in time or quickly seek help from the outside world.

[0003] Therefore, walking aid umbrellas have emerged. However, current walking aid umbrellas on the market have the following limitations: 1) Single triggering mechanism and high false alarm rate: Most devices rely solely on manual button presses or rudimentary fall detection to trigger alarms, easily leading to false alarms or missed alarms. 2) Isolated information and lack of guidance: Existing devices only send location or simple SOS signals after alarming, failing to inform rescuers of "what illness the user has," resulting in a lack of crucial information for rescue; simultaneously, the devices cannot provide users with corresponding medication guidance. 3) Single response chain: Typically, only point-to-point notification to family members occurs. When family members cannot respond promptly, delays occur, affecting rescue efforts. Furthermore, alarm triggering relies on active user operation, failing to recognize unconscious states, resulting in insufficient rescue information dimensions and impacting response efficiency.

[0004] Therefore, there is an urgent need for an intelligent cane umbrella that can fuse multimodal detection data for collaborative recognition and perform hierarchical response processing, which would have significant practical value. Summary of the Invention

[0005] Therefore, to address the aforementioned issues, this invention proposes a multimodal data fusion identification and response method based on a walking cane umbrella. By fusing multimodal detection data of "effective medication dispensing," "motion artifact elimination," and "vital sign data extraction," the method collaboratively identifies the user's current state and takes corresponding response measures, thereby improving the safety of users when traveling outdoors.

[0006] To solve this technical problem, the present invention adopts the following solution: a multimodal data fusion and recognition response method based on a walking cane umbrella, comprising the following steps: S1. Multimodal perception and recognition layer configuration: The cane umbrella body integrates an intelligent controller, power module, communication module, positioning and attitude perception module, first aid medicine compartment module, vital sign detection module and voice interaction module. The cane umbrella body has an installation cavity inside the handle. The vital sign detection module is set in the grip area of ​​the cane umbrella body handle. The pull-out first aid medicine compartment module is set in the installation cavity of the handle. The positioning and attitude perception module is set in the umbrella shaft of the cane umbrella body. All of these are uniformly coordinated and controlled by the intelligent controller. The emergency medicine storage module includes a medicine storage shell with access control and weighing sensors located at the bottom of each medicine storage unit inside the medicine storage shell, which are used to sense the opening status of the medicine storage and the instantaneous mass change ΔW of each unit's medicine in real time. The vital signs detection module includes a PPG sensor array and a bioimpedance sensor, which are used to synchronously collect the user's heart rate (HR), blood oxygen saturation (SpO2), and skin conductance response (SCR) in a high-frequency sampling mode under trigger conditions. The positioning and attitude perception module includes a positioning unit and a nine-axis inertial measurement unit. The positioning unit and the nine-axis inertial measurement unit detect the user's position information in real time and identify whether the user's posture is abnormal or whether the user has fallen. The intelligent controller includes a storage unit that stores: medication instructions for different drug categories; three-level alarm thresholds for heart rate, blood oxygen saturation, and skin conductance signals; and three-level response execution schemes corresponding to the three-level alarm thresholds. S2. Event-triggered data fusion and correlation analysis: The intelligent controller monitors the access control signal in real time. When the access control is detected to be open, the intelligent controller judges whether the drug retrieval action is valid based on the change in drug inventory. If the drug retrieval is valid, the intelligent controller immediately triggers the vital signs detection module to switch from low power mode to high frequency sampling mode and activates the high-precision output mode of the positioning and attitude perception module. S3. Motion artifact elimination and physiological feature extraction: The intelligent controller synchronously receives high-frequency vital sign data and posture data triggered in step S2, and extracts the correct user vital sign data after real-time motion artifact elimination correction of the received data. S4. Perform hierarchical response decision-making and dynamic upgrade / downgrade for multimodal perception and recognition: The intelligent controller compares the real-time feature data extracted in step S3 with the preset personalized three-level alarm thresholds in the storage unit and performs corresponding three-level response processing. Level 1 response: If the feature value exceeds the Level 1 threshold, the voice interaction module will play the emergency guidance voice matching the drug type Dn, and the communication module will send the first warning information containing the location, drug type Dn and heart rate HRcorr(t) summary to the preset Level 1 contact. Level 2 response: If the characteristic value exceeds the Level 2 threshold, the second stage of emergency guidance will be provided through the voice interaction module, and the location and upgraded distress information containing real-time vital signs data will be sent to the Level 2 contact person and the preset remote medical platform. Level 3 Response: If the feature value exceeds the Level 3 threshold, or if the positioning and posture perception module detects and identifies a user falling and there is no effective posture recovery within a preset time, a local high-decibel voice distress call will be made through the voice interaction module. A direct voice connection with the nearest emergency center will be automatically established through the communication module to synthesize and automatically broadcast the location, user identity, and real-time key vital signs data. At the same time, the highest-level alarm will be sent to all preset contacts and the real-time location will be continuously shared to continuously acquire vital signs data. If the positioning and posture perception module detects that the user has fallen and stood up on their own, the current vital signs data will be reassessed, and the corresponding level of response decision will be executed based on the assessment results.

[0007] Furthermore, it also includes: S5, network outage redundancy safety protection: when the third-level response is initiated, if the communication module detects no network signal, the intelligent controller automatically activates the emergency information display window set on the handle, lights up and displays the preset personal illness information and emergency contact information, and the intelligent controller controls the emergency flashlight set on the handle to emit a light signal in SOS mode; the intelligent controller also controls the voice interaction module to perform local emergency distress voice broadcast.

[0008] Furthermore, in step S2, the valid drug retrieval judgment process involves the intelligent controller monitoring the access control signal in real time. When the access control is detected to be open, the opening time is immediately recorded. And according to the weighing sensor Calculate the drug quality change matrix for each drug storage unit based on the data collected before and after. ;like Any element in satisfies ,in If the standard quality of a single drug is given, N is the preset quantity of drugs to be taken, and δ is the error threshold, then it is determined to be a valid drug taking event, and the corresponding drug type Dn is recorded.

[0009] Furthermore, the emergency medication compartment module includes a medication compartment shell, which contains at least two medication storage units. Each medication storage unit has a weighing sensor at its bottom to detect the amount of medication stored in it. The medication compartment shell is equipped with an access control system. The medication compartment shell is movably mounted on a handle mounting cavity, and can be moved out of the handle mounting cavity to open for medication retrieval or closed and inserted into the handle mounting cavity. The medication compartment shell includes multiple medication storage units, each with a different number, allowing users to take corresponding emergency medications based on different threshold levels reached by their vital signs data.

[0010] Furthermore, one side of the handle has an opening communicating with the mounting cavity. The medicine container housing is movably inserted through the opening of the handle into the mounting cavity. The medicine container housing is provided with an adjustable limiting post. The handle has an adjustment groove along the moving direction of the medicine container housing, which allows the adjustable limiting post to move. The medicine container housing moves along the adjustment groove of the handle via the adjustable limiting post, causing the medicine container housing to move out of the handle mounting cavity to open the corresponding medicine storage unit to take out medicine or close the medicine storage unit to insert into the handle mounting cavity.

[0011] Furthermore, the vital signs detection module includes a single-wavelength PPG sensor, a dual-wavelength PPG sensor, and a microcurrent bioimpedance sensor. The single-wavelength PPG sensor, the dual-wavelength PPG sensor, and the microcurrent bioimpedance sensor are arranged in an array in the grip area of ​​the handle to synchronously and in real time collect the user's heart rate, blood oxygen saturation, and skin conductance response signals.

[0012] Furthermore, in step S3, the motion artifact elimination and correction process involves: the intelligent controller synchronously receiving high-frequency vital sign data and attitude data triggered in step S2; and based on the motion intensity output by the nine-axis inertial measurement unit... , Acceleration data were collected along the x, y, and z axes, with MI representing motion intensity. An adaptive filtering algorithm was used with MI(t) as the reference noise to perform real-time motion artifact removal on the raw heart rate HR(t) and blood oxygen saturation SpO2(t) signals collected by the PPG sensor of the vital signs detection module, resulting in the corrected clean signals of user heart rate HRcorr(t) and blood oxygen saturation SpO2corr(t). Based on the corrected signals, the peak features of heart rate variability (HRV) and skin conductance response (SCR) were extracted.

[0013] Furthermore, the nine-axis inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. The intelligent controller comprehensively determines a fall event through the following steps: Step 1, based on the three-axis accelerometer data, an instantaneous impact exceeding a preset threshold is detected; Step 2, within a time window after the impact is detected, the data from the three-axis gyroscope and the three-axis magnetometer are fused and analyzed to determine whether the human posture changes from an upright state to a non-upright state and continues for more than a preset time; Step 3, the vital signs data after the impact are analyzed simultaneously. If the heart rate signal disappears, the heart rate drops sharply, or the blood oxygen saturation drops by more than a preset percentage in a short period of time, then the fall is finally determined by combining the above steps.

[0014] Furthermore, it also includes an emergency light, which is connected to and controlled by an intelligent controller. When a response alarm is triggered at night, the intelligent controller controls the emergency light to turn on for auxiliary guidance.

[0015] Furthermore, when the intelligent controller executes the second-level and third-level responses, it also establishes a two-way voice channel for paramedics from remote medical platforms or emergency centers to provide assistance and guidance to users.

[0016] By adopting the aforementioned technical solution, the beneficial effects of this invention are as follows: By configuring a multimodal perception and recognition layer on the walking cane umbrella, that is, integrating an emergency medication compartment module on the cane umbrella body, the invention receives and integrates in real time the access control status signal of the emergency medication compartment module, the medication quantity detection information of the weighing sensor, the vital signs data detected by the vital signs detection module, the user location information detected by the positioning and posture perception module, and the user posture information; by using access control detection in conjunction with the weighing sensor to detect the medication retrieval action, the invention integrates the multimodal detection data of "effective medication retrieval," "motion artifact elimination," and "vital signs data extraction" to collaboratively identify the user's current state and take corresponding response measures; the invention identifies the type of medication retrieved based on the reduced medication inventory, compares the currently detected vital signs data with the heart rate, blood oxygen saturation, and skin conductance response signals pre-stored in the storage unit, and compares the user posture information detected by the positioning and posture perception module to determine the user's current alarm threshold level and execute the corresponding level of alarm, thus realizing "effective medication retrieval - voice guidance - posture information - vital signs data - graded response execution decision." The system employs closed-loop multimodal fusion recognition and response decision processing, while timely local voice guidance helps users take medication correctly, avoiding medication errors caused by panic. Combined with vital sign data collected by the vital sign detection module, it achieves tiered response execution decisions. When vital sign data is normal after medication retrieval, only a basic SOS SMS is sent; when vital sign data remains abnormal, the response execution level is automatically escalated. Voice calls are made, improving the accuracy and effectiveness of response execution decisions. This dynamic tiered response execution mechanism enables emergency first aid during the "golden rescue time," ensuring users receive timely and effective assistance. Different numbers are assigned to the medication storage units, allowing users to take corresponding emergency medications based on different threshold levels of vital sign data. This enables rapid execution of the multi-level response mechanism, improving rescue efficiency. The medication compartment shell moves out of the handle mounting cavity via an adjustable limit post on the handle's adjustment slot, opening the corresponding medication storage unit for retrieval or closing the storage unit for insertion into the handle mounting cavity. The compact and reasonable structure allows for effective use of the handle's space without requiring a larger handle. Attached Figure Description

[0017] Figure 1 This is a flowchart of the multimodal recognition response according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the product structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure from another angle in an embodiment of the present invention; Figure 4 This is a partially exploded structural diagram of an embodiment of the present invention; Figure 5 This is a schematic diagram of the opening structure of the medicine compartment shell according to an embodiment of the present invention; Figure 6 This is a cross-sectional structural schematic diagram of an embodiment of the present invention; Figure 7 This is a schematic diagram of the adjustable limiting column structure according to an embodiment of the present invention; The components include: 1. Umbrella pole, 2. Umbrella canopy, 3. Handle, 4. Support foot, 5. Intelligent controller, 6. Power module, 7. Positioning and attitude sensing module, 8. Medicine compartment shell, 9. Vital signs detection module, 10. Voice interaction module, 31. Adjustment slot, 51. Control panel, 52. Display screen, 81. Medicine storage unit, 82. Weighing sensor, 83. Access control, 84. Adjustable limit post, 91. Single-wavelength PPG sensor, 92. Dual-wavelength PPG sensor, 93. Microcurrent bioimpedance sensor, 311. Card slot, 841. Spring, 842. Card block. Detailed Implementation

[0018] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0019] refer to Figures 1-7 The preferred method of the present invention for multimodal data fusion recognition and response based on walking cane umbrella includes the following steps: S1. Multimodal perception and recognition layer configuration: including the cane umbrella body, which includes an umbrella pole 1, an umbrella canopy 2, a handle 3, and support feet 4. The umbrella canopy 2 is foldably mounted on the umbrella pole 1. The handle 3 and support feet 4 are respectively connected to both ends of the umbrella pole 1. An intelligent controller 5, a power module 6, a communication module, a positioning and attitude perception module 7, an emergency medicine compartment module, a vital signs detection module 9, and a voice interaction module 10 are integrated on the cane umbrella body. An installation cavity is provided inside the handle 3 of the cane umbrella body. The power module 6 supplies power to the intelligent controller 5, the communication module, the positioning and attitude perception module 7, the vital signs detection module 9, and the voice interaction module 10; and is uniformly coordinated and controlled by the intelligent controller 5. The emergency medicine storage module includes a medicine storage shell 8, which contains up to four medicine storage units 81. Each medicine storage unit 81 has a weighing sensor 82 at its bottom to detect the amount of medicine stored in it, used to sense the opening status of the medicine storage unit and the instantaneous mass change ΔW of the medicine in each unit. The medicine storage shell 8 is equipped with an access control 83. The four medicine storage units 81 are numbered sequentially from the outside to the inside. One side of the handle 3 has an opening communicating with the mounting cavity. The medicine storage shell 8 is movably inserted into the mounting cavity through the opening of the handle 3. The medicine storage shell 8 is equipped with an adjustable limit post 84. The handle 3 has an adjustment mechanism along the moving direction of the medicine storage shell 8 for the adjustable limit post 84 to move. The medicine storage unit 8 moves on the adjustment groove 31 of the handle 3 via the adjustable limiting post 84, causing the medicine storage unit 8 to move out of the mounting cavity of the handle 3 to open the corresponding medicine storage unit 81 to take out medicine or close the medicine storage unit 81 to insert into the mounting cavity of the handle 3. The adjustable limiting post 84 is telescopically inserted on the adjustment groove 31 via the pressing spring 841. When the medicine storage unit 8 is to be moved out, the adjustable limiting post 84 is pressed inward to compress the spring 841, so that the locking block 842 on the adjustable limiting post 84 disengages from the locking groove 311 of the adjustment groove 31. After the medicine storage unit 81 of the corresponding compartment of the medicine storage unit 8 is moved out, the locking block 842 of the adjustable limiting post 84 is locked in the locking groove 311. After taking out the medicine, the adjustable limiting post 84 is pressed to compress the spring 841 to move the position of the medicine storage unit 8. The vital signs detection module 9 includes a single-wavelength PPG sensor 91, a dual-wavelength PPG sensor 92, and a microcurrent bioimpedance sensor 93. The single-wavelength PPG sensor 91, the dual-wavelength PPG sensor 92, and the microcurrent bioimpedance sensor 93 are arranged in an array in the grip area of ​​the handle 3 to synchronously collect the user's heart rate (HR), blood oxygen saturation (SpO2), and skin conductance response (SCR) in a high-frequency sampling mode under trigger conditions. The grip area of ​​the handle 3 is provided with a non-slip, skin-friendly silicone block, and the microcurrent bioimpedance sensor 93 uses electrode sheets embedded in the silicone block of the grip area of ​​the handle 3. The positioning and attitude perception module 7 includes a positioning unit and a nine-axis inertial measurement unit. Both the positioning unit and the nine-axis inertial measurement unit are located inside the umbrella pole 1 to detect the user's position information in real time and identify whether the user's posture is abnormal or has fallen. The positioning unit is a GPS positioning module, and the nine-axis inertial measurement unit is an MPU9250 nine-axis sensor. The nine-axis inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. The voice interaction module 10 is mounted on the handle 3, and the voice interaction module 10 includes a microphone and a speaker; The communication module is a 4G communication module; The intelligent controller 5 is mounted on the mounting cavity of the handle 3 and includes a storage unit, a control panel 51, and a display screen 52. The control panel 51 allows the user to actively operate the intelligent controller 5 to initiate an emergency call or communicate with a pre-set alarm contact via the voice interaction module 10 and communication module to request help or notify the alarm to be deactivated. The display screen 52 displays vital signs data, the current status, and other information. The access control 83 of the medicine storage unit 8 collects information on whether the medicine storage unit 8 is open in real time and sends it to the intelligent controller 5. The weighing sensor 82 sends real-time information on the amount of medicine stored in the medicine storage unit 81 to the intelligent controller 5. The vital signs detection module... The system 9 collects the user's vital signs data, including heart rate, blood oxygen saturation, and skin conductance signals, and sends them to the intelligent controller 5 in real time. The positioning and posture perception module 7 detects the user's location and posture information in real time and sends them to the intelligent controller 5. The storage unit stores: medication guidance information for different drug categories; three-level alarm thresholds for heart rate, blood oxygen saturation, and skin conductance signals; and a three-level response execution plan corresponding to the three-level alarm thresholds. Upon initial application, the intelligent controller 5 activates the vital signs detection module 9 to collect the user's historical resting vital signs data as negative sample data and stores it in the storage unit. The storage unit also stores vital signs data simulating the user's illness state as positive sample data.

[0020] S2. Event-triggered data fusion and correlation analysis: The intelligent controller 5 monitors the signal of access control 83 in real time. When it detects that access control 83 has opened, it immediately records the opening time. And according to the weighing sensor 82 in The data collected before and after the data collection were used to calculate the drug quality change matrix for each drug storage unit 81. ;like Any element in satisfies ,in If the standard quality of a single drug is N, the preset number of drugs to be taken is δ, then it is determined to be a valid drug taking event, and the corresponding drug type Dn is recorded. At the same time, in response to the valid drug taking event, the intelligent controller 5 immediately triggers the vital signs detection module 9 to switch from low power mode to high frequency sampling mode, and activates the high precision output mode of the positioning and attitude perception module 7.

[0021] S3. Motion artifact elimination and physiological feature extraction: The intelligent controller 5 synchronously receives high-frequency vital sign data detected by the vital sign detection module 9 and attitude data detected by the positioning and attitude perception module 7 after step S2 is triggered; based on the motion intensity output by the nine-axis inertial measurement unit... , Acceleration data were collected along the x, y, and z axes, with MI representing motion intensity. An adaptive filtering algorithm (LMS algorithm) was used with motion intensity MI(t) as a reference input to denoise the heart rate data HR(t) in the PPG signal from the single-wavelength PPG sensor 91, resulting in artifact-corrected heart rate data HCrorr(t). Then, the variability indices of the artifact-corrected heart rate (such as SDNN and RMSSD) and the rate of decrease in blood oxygen saturation (SpO2) were calculated and compared with the three-level alarm thresholds of heart rate, blood oxygen saturation, and skin conductance signals pre-stored in the storage unit to determine whether the user has reached the set alarm threshold. Based on the calibration... After the signal is positive, the heart rate variability (HRV) and skin conductance response (SCR) peak characteristics are extracted. The intelligent controller 5 comprehensively determines the fall event through the following steps: Step 1, based on the three-axis accelerometer data, an instantaneous impact exceeding a preset threshold is detected; Step 2, within a time window after the impact is detected, the data from the three-axis gyroscope and three-axis magnetometer are fused and analyzed to determine whether the human posture changes from an upright state to a non-upright state and lasts for more than a preset time; Step 3, the vital signs data after the impact are analyzed simultaneously. If the heart rate signal disappears, the heart rate drops sharply, or the blood oxygen saturation drops by more than a preset percentage in a short period of time, then the fall is finally determined by combining the above steps.

[0022] S4. Perform hierarchical response decision-making and dynamic escalation / de-escalation for multimodal perception and recognition: The intelligent controller 5 compares the features extracted in step S3 with the preset personalized three-level alarm thresholds associated with the drug type Dn in the storage unit. Level 1 Response: If the feature value exceeds the Level 1 threshold, the voice interaction module 10 plays an emergency guidance voice matching the drug type Dn, and the communication module sends a first warning message containing the location, drug type Dn, and heart rate HRcorr(t) summary to the preset Level 1 contact; during the preset first monitoring period The system continuously monitors the data; if the characteristic value falls below the threshold, an alert cancellation message is sent. Level 2 Response: If the characteristic value continues to deteriorate and exceeds the Level 2 threshold within the first monitoring period T1, the Level 2 response is initiated. The second stage of emergency care guidance is provided through the voice interaction module 10, and the location and upgraded distress information containing real-time vital signs data are sent to the Level 2 contact person and the preset remote medical platform. During the second monitoring period T2, continuous monitoring is carried out. If the characteristic value improves, the response level is downgraded back to Level 1. If the value deteriorates, the Level 3 response is triggered. Level 3 Response: If the feature value exceeds the Level 3 threshold, or if the positioning and posture perception module 7 detects and identifies a user falling and there is no effective posture recovery within the preset T3 time, then the Level 3 response is initiated: a local high-decibel voice distress call is made through the voice interaction module 10, and a direct voice connection channel with the nearest emergency center is automatically established through the communication module to synthesize voice and automatically broadcast the location, user identity, and real-time key vital signs data. At the same time, the highest-level alarm is sent to all preset contacts and the real-time location is continuously shared, and vital signs data is continuously acquired. If the vital signs data are detected to improve and fall back to the Level 2 or Level 1 threshold range, the intelligent controller 5 will downgrade the current response level to Level 2 or Level 1 accordingly and execute the corresponding level of response decision. If the positioning and posture perception module 7 detects that the user has fallen and then stood up on their own, the current vital signs data will be reassessed, and the corresponding level of response decision will be executed based on the assessment results.

[0023] The following explanation uses the example of a walking cane umbrella used by a user with heart disease. The umbrella shaft and support legs are made of aluminum alloy. The umbrella shaft is 90cm long when unfolded, and the support legs are 10cm long. The canopy is made of 190T waterproof polyester fabric with a diameter of 100cm. The handle is made of non-slip rubber and has an internal cavity that houses the intelligent controller, power module, communication module, emergency medicine compartment module, vital signs detection module, and voice interaction module. The positioning and posture perception module is located inside the umbrella shaft, and its output is electrically connected to the input of the intelligent controller. The medicine compartment housing for emergency medications is made of medical-grade ABS material. The housing utilizes the aforementioned adjustable limiting post mechanism, which moves the housing across the handle's adjustment slot to open the corresponding medication storage unit for retrieval or close the unit for insertion into the handle's mounting cavity. The medicine compartment housing uses a Hall effect sensor for access control and a weighing sensor for remaining medication levels. Emergency medications are stored in the storage units: Unit 1 holds 10 nitroglycerin tablets, and Unit 2 holds 10 non-enteric-coated aspirin tablets. A single-wavelength PPG sensor, a dual-wavelength PPG sensor, and a microcurrent bioimpedance sensor are arranged in an array in the handle's grip area to synchronously and in real-time collect the user's heart rate, blood oxygen saturation, and skin conductance signals. The single-wavelength PPG sensor uses 530nm green light as the emission source to collect heart rate data. It illuminates the skin's capillary network. The absorption rate of green light by hemoglobin in the blood changes periodically with the heartbeat (higher blood volume during systole → stronger absorption; lower blood volume during diastole → weaker absorption). The sensor captures minute fluctuations in transmitted or reflected light intensity using photodiodes, and extracts the heart rate per minute (BPM) through filtering and peak detection algorithms. The sampling frequency is 100Hz. The dual-wavelength PPG sensor 92 simultaneously emits 660nm red light and 940nm infrared light to collect blood oxygen saturation. Oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb) have different absorption rates for these two wavelengths: HbO2 absorbs more infrared light, while Hb absorbs more red light. By calculating the attenuation ratio (R value) of the two wavelengths and combining it with an empirical calibration curve, the peripheral blood oxygen saturation (SpO2) sampling frequency is calculated to be 60Hz. The microcurrent bioimpedance sensor uses 100Hz AC excitation to acquire skin electrical responses and measures skin conductivity (GSR) using a constant current excitation method. A safe, weak AC current (100Hz) is applied to the skin surface, forming a circuit with silver / silver chloride (Ag / AgCl) electrodes embedded in silicone blocks. When the user is emotionally stressed or anxious, the sympathetic nervous system activates sweat glands to secrete sweat, and electrolytes in the sweat (… , Increased concentration of [a substance] leads to an increase in skin conductivity (a decrease in resistance). The microcurrent bioimpedance sensor detects minute voltage changes between electrodes using a high-impedance operational amplifier, converting these changes into conductivity values ​​measured in micro-Siemens (μS). This reflects the level of autonomic nervous system arousal. Skin conductivity level (SCL) (long-term activation level): a slowly changing baseline reflecting long-term emotional tone; Skin conductance response (SCR) (transient spike component): a transient spike corresponding to specific stimuli (such as fright or decision-making stress), serving as a sensitive indicator of emotional arousal.

[0024] The intelligent controller uses an STM32F103 microcontroller as its main control chip and a 16GB Flash memory for storage. It pre-stores five emergency voice guidance messages, two sets of preset contact numbers, three alarm thresholds, and three-level response decisions. During initial use, the vital signs detection module collects heart rate, blood oxygen, and other data from the user in different states (resting, light activity) to establish a personal health baseline. When the user stores a specific medication (such as nitroglycerin) in the medication compartment module, they can associate the medication with a specific condition (such as angina) via the APP or the interactive interface on the intelligent controller, and retrieve a preset alarm threshold based on the personal baseline and matching the condition from the storage unit. The communication module uses a 4G full-network compatible module, supporting 4G networks of China Mobile, China Unicom, and China Telecom; the positioning module uses a GPS / BeiDou dual-mode positioning module with a positioning accuracy of ≤5m. The voice interaction module has a 3W speaker and a noise-canceling microphone; the handle features a one-button confirmation button and an emergency call button, made of comfortable silicone. The power module uses a 2000mAh rechargeable lithium battery with a Type-C charging interface, and can work continuously for 72 hours when fully charged.

[0025] The three-level alarm threshold settings are as follows: Level 1 alarm threshold: corresponds to early signs of angina pectoris / compensated myocardial ischemia.

[0026] Objective: The vital signs monitoring module collects and transmits data to the intelligent controller. The intelligent controller identifies early signs of increased cardiac load or mild ischemia, reminds the user to rest and take medication, and notifies family members to pay attention. Status: The user may experience chest tightness and shortness of breath, but is conscious and able to obtain medication independently. Triggering conditions (meeting two or more of the following conditions, or any one of the first conditions): 1) Heart rate: An increase of >30% compared to one's resting heart rate for >2 minutes; 2) Blood oxygen saturation: SpO2 decreases by 3-5% from the individual's baseline (e.g., from 98% to 93-95%); 3) Heart rate variability: HRV (such as SDNN) decreases significantly in the short term; 4) Activity status: The above changes occur when the person is at rest or in a low activity state.

[0027] Level 2 alarm threshold: worsening myocardial ischemia / early signs of heart failure.

[0028] Objective: The vital signs monitoring module collects and transmits data to the intelligent controller, which then identifies the progression of the condition and indicates that the user may no longer be able to effectively alleviate the symptoms on their own, requiring intervention from medical professionals. Status: The user may experience worsening chest pain, difficulty breathing, profuse sweating, and severely limited mobility. Triggering conditions (escalation from Level 1 alert, or direct fulfillment of any of the following key indicators): 1) Heart rate: consistently >130 beats / min, or <50 beats / min (severe bradycardia); 2) Blood oxygen saturation: SpO2 consistently <90%; 3) Cardiac arrhythmias: Frequent premature ventricular contractions (>5 times / min) and short runs of ventricular tachycardia were detected; 4) Vital signs continue to deteriorate: After the Level 1 alarm response, all abnormal indicators do not improve or continue to deteriorate within 5-10 minutes.

[0029] Level 3 alarm threshold: Precursor to cardiac arrest / malign arrhythmia.

[0030] Objective: The vital signs detection module collects and transmits data to the intelligent controller, which then identifies an immediate life-threatening situation and requires the activation of the highest level of emergency response. Status: The vital signs detection module is detecting possible loss of consciousness, convulsions, or respiratory arrest in the user. Triggering conditions (it should be triggered immediately if any of the following conditions are met): 1) Heart rate: Ventricular fibrillation, pulseless ventricular tachycardia, or cardiac arrest (ECG signal disappearance > 5 seconds) were detected; 2) Severe tachycardia: heart rate >160 beats / min and accompanied by a sudden drop in blood oxygen (<85%); 3) Severe bradycardia: Heart rate <40 beats / min accompanied by signs of loss of consciousness (sudden collapse without struggling detected by a nine-axis inertial measurement unit IMU) 4) Collapse of vital signs: blood oxygen saturation <85% and pulse is weak and undetectable (judged by PPG signal quality); 5) Posture and no response: The nine-axis inertial measurement unit confirmed "sudden fall" and then no body movement, loss of heart rate signal or extreme abnormality for more than 30 seconds.

[0031] The three-level alarm threshold corresponds to the three-level response decision settings: Level 1 Response: 1) The intelligent controller controls the voice interaction module to broadcast: "We have detected an increase in your heart load. We suggest you sit down and rest immediately and consider taking medication as prescribed by your doctor." 2) The intelligent controller sends an observational warning SMS to a pre-set first-level contact (such as a child) via the communication module: "[User Name]'s heart rate and blood oxygen levels have shown early abnormalities. Emergency medication has been administered, and the user is under close observation. Location: [Real-time Positioning]" 3) Initiate a 5-minute enhanced monitoring cycle.

[0032] Level 2 response: 1) The intelligent controller controls the voice interaction module to broadcast voice messages: "Your condition requires urgent medical attention. Medical assistance is being contacted. Please remain still and do not move." 2) The intelligent controller automatically sends emergency assistance information to the second-level contact (such as a community doctor or a contracted telemedicine platform) through the communication module. The information includes detailed vital signs data, medication records, and location: [real-time location]. 3) Initiate a 5-minute enhanced monitoring cycle, ready to upgrade to Level 3 alert at any time.

[0033] Level 3 response: 1) The intelligent controller controls the voice interaction module to broadcast voice messages: The maximum volume of the voice broadcast is: "Cardiac arrest risk detected! Calling emergency services! Please help call or provide emergency assistance!" to guide nearby rescuers; 2) The intelligent controller automatically dials the local emergency center (such as 120) via the communication module. After the call is connected, it uses the synthesized voice stored in the pre-stored unit to clearly announce: "This is the intelligent emergency equipment, suspected cardiac arrest alarm. Patient name [XXX], located at [real-time location address], location beacon activated; 3) At the same time, the intelligent controller sends the highest level alarm to all preset contacts through the communication module and continuously shares real-time location.

[0034] During use, the Hall sensor continuously monitors whether the medicine compartment door is open, and the data is transmitted in real time to the STM32F103 microcontroller in the intelligent controller. When the user opens the medicine compartment door due to heart discomfort, the Hall sensor sends a "door open" signal to the intelligent controller (STM32F103 microcontroller). The STM32F103 microcontroller controls the speaker of the voice interaction module to play the corresponding voice instructions for the medicine according to the number of the medicine storage unit taken by the user, such as: "Please sit down immediately, take one nitroglycerin tablet under your tongue, do not swallow." At the same time, the STM32F103 microcontroller sends a pre-edited SOS text message through the communication module: "[User Name] has early abnormalities in heart rate and blood oxygen, has taken emergency medication, is under close observation, current heart rate 110 beats / minute, location: No. XX, XX Road, XX District, XX City, please come to rescue!" to the designated contact person (user's child or community doctor). The STM32F103 microcontroller continuously monitors heart rate data for 10 minutes after the medicine compartment door is opened. If the user's heart rate remains >130 beats / min or <50 beats / min (severe bradycardia) for 3 consecutive minutes, it is considered a persistent abnormal heart rate, triggering the second-level alarm threshold. The STM32F103 microcontroller controls the speaker of the voice interaction module to broadcast a voice announcement: "Your condition requires urgent medical attention. Medical assistance is being contacted. Please remain still and do not move." The STM32F103 microcontroller automatically sends an emergency request for help to the second-level contact (such as a community doctor or a contracted telemedicine platform). The STM32F103 microcontroller transmits a pre-edited SOS text message via the communication module: "[User Name] has experienced persistent abnormal heart rate and blood oxygenation. Emergency medication has been taken. The user is under close observation. Current heart rate: 135 beats / min. Location: XX City, XX District, XX Road, XX..." "I need immediate rescue!" is sent to designated contacts (user's children, community doctor, or contracted telemedicine platform); a 5-minute enhanced monitoring cycle is initiated. Users can also use the microphone on the smart controller to make a voice call to the contact, and the alert is ready to escalate to Level 3 at any time. When the Level 3 alarm threshold is triggered, the STM32F103 microcontroller controls the speaker of the voice interaction module to broadcast a message at maximum volume: "Cardiac arrest risk detected! Calling emergency services! Please call or provide emergency assistance!" to guide nearby rescuers. Simultaneously, the STM32F103 microcontroller automatically dials the local emergency center (e.g., 120). After the call connects, a clear, pre-stored synthesized voice is broadcast: "This is a smart emergency device, suspected cardiac arrest alarm. Patient name [XXX], located at [real-time location address], location beacon activated." The highest-level alarm is also sent to all pre-set contacts, and the real-time location is continuously shared.If the user's vital signs recover, such as a heart rate returning to 80 beats per minute within 10 minutes, the warning process ends. When the weighing sensor detects that the remaining medication is ≤2 tablets, the STM32F103 microcontroller controls the speaker of the voice interaction module to play a reminder: "Insufficient emergency medication, please replenish promptly."

[0035] The intelligent controller, access control, power module, communication module, positioning unit, nine-axis inertial measurement unit, single-wavelength PPG sensor, dual-wavelength PPG sensor, and microcurrent bioimpedance sensor are all existing modules, such as the intelligent controller disclosed in Chinese patent document CN201921281712.X, a multifunctional electronic cane. The nine-axis inertial measurement unit can adopt existing modules such as the IMU inertial navigation module of Beijing Bohang Hengye Technology Co., Ltd. or the coverage tilt sensor of Shenzhen Ruiguan Technology Co., Ltd. The single-wavelength PPG sensor can adopt the MAX30105 module of Analog Devices, Inc. The dual-wavelength PPG sensor can adopt the ADPD144RI module or MAX30102 module of Analog Devices, Inc. The microcurrent bioimpedance sensor can adopt the Shimmer3 Ebio Unit module of Shimmer Sensing. The communication module can also adopt a 4G / 5G dual-mode communication module, a 5G communication module, or an NB-IoT / 4G dual-mode communication module. The positioning unit can also adopt a Beidou module.

[0036] In addition, an extra step can be added for handling situations without a network signal: S5, Network Disconnection Redundancy Safety Protection: When initiating the third-level response, if the communication module detects no network signal, the intelligent controller automatically activates the emergency information display window on the handle, illuminating and displaying preset personal medical information and emergency contact details. Furthermore, the intelligent controller controls the emergency flashlight on the handle to emit a light signal in SOS mode. The intelligent controller also controls the voice interaction module to broadcast a local emergency distress message. Alternatively, an emergency light can be added to the end of the handle furthest from the medicine compartment housing. This emergency light is connected to and controlled by the intelligent controller; when a response alarm is triggered at night, the intelligent controller activates the emergency light for auxiliary guidance. Furthermore, an emergency information display window and an emergency flashlight, electrically connected to the intelligent controller, can be added to the surface of the handle. When the communication module has no network signal and the vital signs data detection is in the third-level response alarm state, the intelligent controller can control the emergency information display window to light up and display preset personal key medical information, allergy history, and emergency contact phone numbers. At the same time, the intelligent controller controls the voice interaction module to broadcast a local emergency SOS message. The intelligent controller also controls the emergency flashlight to flash in SOS mode, making it easier for people in the vicinity to help call 120 or other emergency numbers for first aid, further preventing the user from missing the golden time for rescue. When executing the response decision in step S4, the intelligent controller can also establish a two-way voice channel for remote medical platforms or emergency center paramedics to provide rescue guidance to the user.

[0037] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Although the present invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail can be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A multimodal data fusion and recognition response method based on a walking cane umbrella, characterized in that: Includes the following steps: S1. Multimodal perception and recognition layer configuration: The cane umbrella body integrates an intelligent controller, power module, communication module, positioning and attitude perception module, first aid medicine compartment module, vital sign detection module and voice interaction module. The cane umbrella body has an installation cavity inside the handle. The vital sign detection module is set in the grip area of ​​the cane umbrella body handle. The pull-out first aid medicine compartment module is set in the installation cavity of the handle. The positioning and attitude perception module is set in the umbrella shaft of the cane umbrella body. All of these are uniformly coordinated and controlled by the intelligent controller. The emergency medicine storage module includes a medicine storage shell with access control and weighing sensors located at the bottom of each medicine storage unit inside the medicine storage shell, which are used to sense the opening status of the medicine storage and the instantaneous mass change ΔW of each unit's medicine in real time. The vital signs detection module includes a PPG sensor array and a bioimpedance sensor, which are used to synchronously collect the user's heart rate (HR), blood oxygen saturation (SpO2), and skin conductance response (SCR) in a high-frequency sampling mode under trigger conditions. The positioning and attitude perception module includes a positioning unit and a nine-axis inertial measurement unit. The positioning unit and the nine-axis inertial measurement unit detect the user's position information in real time and identify whether the user's posture is abnormal or whether the user has fallen. The intelligent controller includes a storage unit that stores: medication instructions for different drug categories; three-level alarm thresholds for heart rate, blood oxygen saturation, and skin conductance signals; and three-level response execution schemes corresponding to the three-level alarm thresholds. S2. Event-triggered data fusion and correlation analysis: The intelligent controller monitors the access control signal in real time. When the access control is detected to be open, the intelligent controller judges whether the drug retrieval action is valid based on the change in drug inventory. If the drug retrieval is valid, the intelligent controller immediately triggers the vital signs detection module to switch from low power mode to high frequency sampling mode and activates the high-precision output mode of the positioning and attitude perception module. S3. Motion artifact elimination and physiological feature extraction: The intelligent controller synchronously receives high-frequency vital sign data and posture data triggered in step S2, and extracts the correct user vital sign data after real-time motion artifact elimination correction of the received data. S4. Perform hierarchical response decision-making and dynamic upgrade / downgrade for multimodal perception and recognition: The intelligent controller compares the real-time feature data extracted in step S3 with the preset personalized three-level alarm thresholds in the storage unit and performs corresponding three-level response processing. Level 1 response: If the feature value exceeds the Level 1 threshold, the voice interaction module will play the emergency guidance voice matching the drug type Dn, and the communication module will send the first warning information containing the location, drug type Dn and heart rate HRcorr(t) summary to the preset Level 1 contact. Level 2 response: If the characteristic value exceeds the Level 2 threshold, the second stage of emergency guidance will be provided through the voice interaction module, and the location and upgraded distress information containing real-time vital signs data will be sent to the Level 2 contact person and the preset remote medical platform. Level 3 Response: If the feature value exceeds the Level 3 threshold, or if the positioning and posture perception module detects and identifies a user falling and there is no effective posture recovery within a preset time, a local high-decibel voice distress call will be made through the voice interaction module. A direct voice connection with the nearest emergency center will be automatically established through the communication module to synthesize and automatically broadcast the location, user identity, and real-time key vital signs data. At the same time, the highest-level alarm will be sent to all preset contacts and the real-time location will be continuously shared to continuously acquire vital signs data. If the positioning and posture perception module detects that the user has fallen and stood up on their own, the current vital signs data will be reassessed, and the corresponding level of response decision will be executed based on the assessment results.

2. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: Also includes: S5. Network Disconnection Redundancy Safety Protection: When the third-level response is initiated, if the communication module detects no network signal, the intelligent controller automatically activates the emergency information display window set on the handle, illuminates and displays the preset personal illness information and emergency contact information, and the intelligent controller controls the emergency flashlight set on the handle to emit a light signal in SOS mode. The intelligent controller also controls the voice interaction module to perform local emergency distress voice broadcast.

3. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: Step S2, valid medication retrieval judgment process: The intelligent controller monitors the access control signal in real time. When it detects that the access control has opened, it immediately records the opening time. And according to the weighing sensor Calculate the drug quality change matrix for each drug storage unit based on the data collected before and after. ;like Any element in satisfies ,in If the standard quality of a single drug is given, N is the preset quantity of drugs to be taken, and δ is the error threshold, then it is determined to be a valid drug taking event, and the corresponding drug type Dn is recorded.

4. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: The emergency medication compartment module includes a medication compartment shell, which contains at least two medication storage units. Each medication storage unit has a weighing sensor at its bottom to detect the amount of medication stored in it. The medication compartment shell is equipped with an access control system. The medication compartment shell is movably mounted on a handle mounting cavity, and can be moved out of the handle mounting cavity to open for medication retrieval or closed and inserted into the handle mounting cavity. The medication compartment shell includes multiple medication storage units, each with a different number. Users can take corresponding emergency medications based on different threshold levels reached by their vital signs data.

5. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 4, characterized in that: The handle has an opening on one side that communicates with the mounting cavity. The medicine container housing is movably inserted through the opening in the handle into the mounting cavity. The medicine container housing is provided with an adjustable limiting post. The handle has an adjustment groove along the moving direction of the medicine container housing for the adjustable limiting post to move. The medicine container housing moves along the adjustment groove in the handle via the adjustable limiting post, causing the medicine container housing to move out of the handle mounting cavity to open the corresponding medicine storage unit to take out medicine or close the medicine storage unit to insert into the handle mounting cavity.

6. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: The vital signs detection module includes a single-wavelength PPG sensor, a dual-wavelength PPG sensor, and a microcurrent bioimpedance sensor. The single-wavelength PPG sensor, the dual-wavelength PPG sensor, and the microcurrent bioimpedance sensor are arranged in an array in the grip area of ​​the handle to synchronously and in real time collect the user's heart rate, blood oxygen saturation, and skin conductance response signals.

7. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: In step S3, the motion artifact elimination and correction process involves the intelligent controller synchronously receiving high-frequency vital sign data and attitude data triggered in step S2; and based on the motion intensity output by the nine-axis inertial measurement unit. , Acceleration data were collected along the x, y, and z axes, with MI representing motion intensity. An adaptive filtering algorithm was used with MI(t) as the reference noise to perform real-time motion artifact removal on the raw heart rate HR(t) and blood oxygen saturation SpO2(t) signals collected by the PPG sensor of the vital signs detection module, resulting in the corrected clean signals of user heart rate HRcorr(t) and blood oxygen saturation SpO2corr(t). Based on the corrected signals, the peak features of heart rate variability (HRV) and skin conductance response (SCR) were extracted.

8. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 7, characterized in that: The nine-axis inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer; the intelligent controller comprehensively determines the fall event through the following steps: Step 1, based on the three-axis accelerometer data, an instantaneous impact exceeding a preset threshold is detected; Step 2, within a time window after the impact is detected, the data from the three-axis gyroscope and the three-axis magnetometer are fused and analyzed to determine whether the human posture changes from an upright state to a non-upright state and continues for more than a preset time; Step 3: Simultaneously analyze vital signs data after the impact. If the heart rate signal disappears, the heart rate drops suddenly, or the blood oxygen saturation drops by more than a preset percentage in a short period of time, then the final determination is a fall based on the above steps.

9. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: It also includes an emergency light, which is connected to and controlled by an intelligent controller. When a response alarm is triggered at night, the intelligent controller controls the emergency light to turn on for auxiliary guidance.

10. The multimodal data fusion recognition and response method based on walking cane umbrella according to claim 1, characterized in that: When the intelligent controller executes the second-level and third-level responses, it also establishes a two-way voice channel for paramedics from remote medical platforms or emergency centers to provide assistance and guidance to users.

Images