A multi-modal fusion shedding detection method and a wearable device

Abstract

The application provides a multi-modal fusion shedding detection method and a wearable device, wherein the multi-modal fusion shedding detection method comprises the following steps: step 1, collecting multiple data of the wearable device, and checking and repairing each data to form multi-modal data; step 2, classifying and counting the multi-modal data, inputting the multi-modal data into a shedding monitoring algorithm analysis model, and outputting a device shedding detection state. The application has the beneficial effect that the wearable device is monitored in real time, whether it is correctly worn is monitored, and normal collection of physical sign data of the wearable device is ensured.

Description

Technical Field

[0001] This invention relates to the field of smart wearable information technology, and in particular to a multimodal fusion detachment detection method and wearable device. Background Technology

[0002] Currently, the number of users of wearable devices, such as smart bracelets, watches, or other wearable equipment, is increasing daily. More and more people want to understand their real-time physical condition, obtain vital sign data through smart wearables, and receive interventions based on this data to guide wearers to take correct or positive actions.

[0003] Wearable devices are personal items worn for extended periods of time, besides clothing, sometimes even 24 / 7, to monitor vital signs such as heart rate, blood pressure, and posture. This is particularly important for the elderly and those with underlying health conditions who require continuous 24-hour monitoring. However, within 24 hours, wearable devices cannot maintain the correct wearing posture at all times due to human movement. Incorrect placement, detachment, or sweating can lead to incomplete, erroneous, or inaccurate data collection, affecting the accurate assessment of health conditions. Summary of the Invention Summary of the Invention This invention proposes a multimodal fusion detachment detection method and wearable device for real-time detection of whether the wearable device is worn correctly, ensuring that the wearable device can collect vital sign data normally. It is achieved through the following technical solution.

[0005] A multimodal fusion shedding detection method includes the following steps: Step 1: Collect multiple data points from wearable devices, and troubleshoot and repair each data point to form multimodal data; Step 2: Classify and statistically analyze the multimodal data, input the multimodal data into the detachment monitoring algorithm analysis model, and output the equipment detachment detection status.

[0006] A wearable device for performing a multimodal fusion shedding detection method, the wearable device comprising at least one of an optical vital signs sensor, an inertial sensor, a temperature sensor, and a GPS / BeiDou sensor.

[0007] The beneficial effects of this invention are: real-time monitoring of wearable devices to ensure they are worn correctly and to guarantee that wearable devices can collect vital sign data normally. Attached Figure Description

[0008] Figure 1 This is a flowchart of the multimodal fusion shedding detection method according to an embodiment of the present invention.

[0009] Figure 2This is a flowchart of the optical vital sign sensing data processing of a wearable device according to an embodiment of the present invention.

[0010] Figure 3 This is a flowchart of the temperature data processing of a wearable device according to an embodiment of the present invention.

[0011] Figure 4 This is a flowchart of the wearable device detachment detection method according to an embodiment of the present invention.

[0012] Figure 5 This is a block diagram of the analysis model structure of the shedding monitoring algorithm according to an embodiment of the present invention.

[0013] Figure 6 This is a functional diagram of the multimodal data shedding monitoring algorithm analysis model according to an embodiment of the present invention. Detailed Implementation

[0014] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0015] The wearable devices in the embodiments of the present invention are not limited to wearable watches, but can also be wearable devices such as bracelets, rings, earrings, armbands / arm bags, and headbands.

[0016] This invention proposes a multimodal fusion-based fall-off detection algorithm for wearable devices, wherein the multimodal form includes different analog signals collected by various sensors such as optical vital signs sensors, inertial sensors, temperature sensors, and GPS / BeiDou sensors.

[0017] In this embodiment of the invention, the optical vital signs sensor includes optical vital signs sensors such as PPG, blood oxygen, and blood pressure, and records signals such as PPG, blood oxygen, and blood pressure through a photoelectric heart rate sensor.

[0018] In this embodiment of the invention, the inertial sensor includes an accelerometer (or accelerometer sensor) and an angular velocity sensor (gyroscope) as well as their single, dual, and triaxial combination IMU (inertial measurement unit) and AHRS (inertial attitude sensor including a magnetic sensor).

[0019] In this embodiment of the invention, the wearable device includes at least one optical vital sign sensor, an inertial sensor, a temperature sensor, and a GPS / BeiDou sensor.

[0020] like Figure 1 , 6 As shown, a multimodal fusion shedding detection method includes the following steps: S1. Collect multiple data points from wearable devices, and troubleshoot and repair each data point to form multimodal data; S2. Classify and statistically analyze multimodal data, input the multimodal data into the detachment monitoring algorithm analysis model, and output the equipment detachment detection status.

[0021] Before starting the test, GPS / BeiDou data is acquired to determine if the location data is changing and to confirm whether the wearable device is being worn on the human body.

[0022] In this embodiment of the invention, it is necessary to collect optical vital sign sensing data, attitude and motion data, body temperature data, and GPS / BeiDou data, and the collected data form multimodal data.

[0023] In this embodiment of the invention, the main functions of the wearable device are to drive different sensor modules, configure relevant register parameters, and collect and classify raw data.

[0024] Optical vital sign sensing data: Drive the optical heart rate sensor and configure and adjust relevant register values ​​(such as light intensity, sampling frequency, etc.) to acquire raw data including PPG signals in real time.

[0025] Inertial sensor data: Drive sensors, including triaxial accelerometers, configure their relevant parameters (such as accuracy, output speed, etc.), and sample them once at a frequency of 4Hz.

[0026] Temperature data: Two temperature sensors (for measuring body surface and environment) are driven and their output accuracy is configured to acquire data at a frequency of once per minute.

[0027] Location signals, using GPS / BeiDou and other location information, determine whether a wearable device is being worn on the human body based on changes in location signals.

[0028] The data collected by the wearable device is multimodal data.

[0029] In this embodiment of the invention, the processing of multimodal data is included, and the data preprocessing module includes the exclusion and repair of abnormal data and the statistics of valid data.

[0030] Abnormal data removal and repair: Due to factors such as equipment malfunction, diverse wearing methods, and uncertainties in wrist movement, the data acquisition module may collect data with anomalies. Different processing methods are used for different signals within the multimodal signal.

[0031] like Figure 2 , 6 As shown, the steps for processing optical sensor data are as follows: S11 will collect a large amount of correct data under normal conditions, then model this data, and import it into the storage space after establishing multiple models.

[0032] S12. Upon receiving new optical vital sign sensor data, the data will be compared and analyzed with the existing data model. If the similarity or trend of the data is found to be too different, the data will be discarded. If the similarity between the data and the model reaches a certain level, the data will be repaired with reference to the model.

[0033] The steps for processing inertial sensor data are as follows: Because inertial sensor data is collected continuously, after receiving the data, it will compare and analyze the data within a certain range before and after. If it is found that the currently received data does not conform to the value that would appear under normal circumstances, the value will be lost or it will be repaired by comparing the two values ​​before and after.

[0034] In the analysis and processing of inertial sensor data, the changes in attitude and motion data generated by the inertial sensor can be compared with the changes in position data generated by GPS / BeiDou to determine whether the inertial sensor data is consistent. If the data is consistent, the consistent data is saved; if the data is inconsistent, it is discarded.

[0035] During the data restoration process, the data from the inertial sensor is combined with the position change data from GPS / BeiDou to restore the accurate attitude and motion data.

[0036] like Figure 3 , 6 As shown, the steps for processing temperature sensor data are as follows: S21. Set a threshold for temperature. The threshold parameters are different for body surface temperature and ambient temperature.

[0037] S22. Upon receiving a new temperature value, it will determine whether the current value is within the set threshold range. If it is not within the range, it will analyze the temperature values ​​within a certain range before, and then repair and control the current temperature value within the set threshold range.

[0038] After processing the optical sensor data, inertial sensor data, and temperature sensor data, processed multimodal data is generated. Then, the effective data is statistically analyzed, and the invalid data is discarded.

[0039] Statistical analysis of valid data: Considering that in some special cases it may be impossible to monitor the device's detachment status in real time using only the current multimodal signals, or that real-time detection may be erroneous, we need to collect and analyze multimodal signals over a period of time for fault tolerance.

[0040] In this embodiment of the invention, the algorithm for multimodal data analysis is: The algorithm processes and analyzes the pre-processed and statistically analyzed multimodal signal data, and the output result is the current detachment status of the equipment.

[0041] like Figure 5 As shown, the analysis model of the detachment monitoring algorithm includes a real-time detachment monitoring algorithm model and a non-real-time detachment monitoring algorithm model. The real-time detachment monitoring algorithm model is based on optical vital sign sensor data and temperature values, while the non-real-time detachment monitoring algorithm model is based on attitude and motion data.

[0042] like Figure 4 , 6 As shown, the preprocessed raw data of the multimodal signal is imported into the algorithm model for analysis.

[0043] S31. First, the optical sensor data is processed and analyzed. The input optical sensor data is compared and analyzed with the previously established model. Since the model is different when the device is worn and when the device is detached, it can be determined whether the user is wearing the wearable device.

[0044] However, under certain special conditions, such as when the device is placed in a specific location, there is a possibility that it may not be detected as detached, or it may be judged as detached even if it is not detached, due to the device not being properly attached. Therefore, the detachment status determined by the inertial sensor, along with the raw values ​​of the inertial sensor statistics from a previous period, will be input into the algorithm model for analysis.

[0045] In this embodiment of the invention, PPG signals can be used as an example for analysis. PPG involves first collecting PPG signals from the human body, and then inputting these signals into a model for comparison. The following analysis uses PPG signals and the PPG algorithm as examples.

[0046] S32. Analyze the statistical values ​​of the inertial sensor.

[0047] The system checks whether the device reached a set threshold within a certain period of time to determine if there was any activity. If the PPG algorithm initially identifies the device as detached, and the inertial sensor's statistical value exceeds the set threshold within a later period, the PPG algorithm's detachment detection result is incorrect; otherwise, the detection result is correct. Conversely, if the PPG algorithm initially identifies the device as worn, and the inertial sensor's value is below the set threshold within a certain period of time, it indicates the device was stationary during that period, suggesting the detachment detection result is incorrect; otherwise, the detection result is correct.

[0048] S33. Input the previously determined shedding status and the original temperature value into the final verification algorithm model for analysis.

[0049] The system verifies whether the wearable device is being worn by comparing the difference between the body temperature sensor readings and the ambient temperature over a previous period with the current body temperature and the ambient temperature. If the previous status was "dislodged," and the current body temperature is within the set temperature threshold while the previous temperature statistics are below the set threshold, PPG monitoring is performed again; otherwise, the current status is confirmed as "dislodged." If the previous status was "worn," the system monitors whether the previous temperature statistics are within the set threshold. If so, it is confirmed as "worn"; otherwise, PPG algorithm monitoring is performed again.

[0050] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal fusion shedding detection method, characterized in that, Includes the following steps: Step 1: Collect multiple data points from wearable devices, and troubleshoot and repair each data point to form multimodal data; The steps for preprocessing multimodal data and then importing the preprocessed raw multimodal signal data into the algorithm model for analysis are as follows: Step 31: Process and analyze the optical vital signs sensor data. Compare and analyze the input optical vital signs sensor data with the previously established model to determine whether the user is wearing a wearable device. Step 32: Analyze the statistical values ​​from the inertial sensor; Step 33 involves inputting the previously determined detachment status and the original temperature value into the final verification algorithm model for analysis; Step 2: Classify and statistically analyze the multimodal data, input the multimodal data into the wearable device detachment monitoring algorithm analysis model, and output the wearable device detachment detection status.

2. The multimodal fusion shedding detection method according to claim 1, characterized in that, The analysis model of the detachment monitoring algorithm includes a real-time detachment monitoring algorithm model and a non-real-time detachment monitoring algorithm model. The real-time detachment monitoring algorithm model is based on optical vital sign sensor data and temperature values, while the non-real-time detachment monitoring algorithm model is based on attitude and motion data.

3. The multimodal fusion shedding detection method according to claim 1, characterized in that, The data investigation and repair method for processing optical vital sign sensor data includes the following steps: Step 11: Collect a large amount of correct data under normal conditions, then model this data, and import the data into the storage space after establishing multiple models; Step 12: Upon receiving new optical vital sign sensor data, compare and analyze it with the existing data model, discard data with large similarity or significant differences in data trends, and repair data whose similarity to the model reaches a certain level.

4. The multimodal fusion shedding detection method according to claim 1, characterized in that, The data investigation and repair process, specifically the steps for processing inertial sensor data, includes: After receiving the data, the system will compare and analyze the data within a certain range before and after. If it finds that the currently received data does not conform to the value that would appear under normal circumstances, the value will be lost or it will be repaired by comparing the two values ​​before and after.

5. The multimodal fusion shedding detection method according to claim 1, characterized in that, The steps for data investigation and repair, specifically the processing of temperature sensor data, are as follows: Step 21: Set a threshold for body surface temperature and an ambient temperature respectively; Step 22: After a new temperature value is measured, determine whether the current value is within the set threshold range. If it is not within the range, analyze the temperature values ​​within a certain range before, and then repair and control the current temperature value within the set threshold range.

6. The multimodal fusion shedding detection method according to claim 1, characterized in that, In step 31, the detachment state determined by PPG is added to the raw values ​​of the inertial sensor statistics over a previous period, and these raw values ​​are then used in the algorithm model for analysis.

7. The multimodal fusion shedding detection method according to claim 1, characterized in that, In step 32, it is determined whether the device has reached the set threshold in the previous or subsequent period of time, so as to determine whether there has been an activity trajectory. If the PPG algorithm determines that the device has detached, and the inertial sensor statistics are higher than a set threshold within a certain period after the device is detected, then the detachment detection result of the PPG algorithm is incorrect; otherwise, the detection result is correct. When the preceding PPG algorithm determines that the device is being worn, if it detects data from the device's internal inertial sensor within a certain period of time... If the value is less than the set threshold, it means that the device has been stationary for a period of time, which indicates that the detachment monitoring result is incorrect; otherwise, it means that the monitoring result is correct.

8. The multimodal fusion shedding detection method according to claim 1, characterized in that, In step 33, the difference between the body temperature sensor readings and the ambient temperature readings from a previous period and the current body temperature and ambient temperature is checked against a threshold to further verify whether the wearable device is being worn. If the previously monitored status is detachment, and the current body temperature is detected to be below the set temperature threshold and the temperature statistics value in the previous period is below the set threshold, then PPG monitoring will be performed again; otherwise, the current status will be confirmed as detachment. When the previously monitored status is "wearing", the system monitors whether the temperature statistics over a period of time are within the set threshold. If so, it confirms that the device is wearing the device; otherwise, it performs PPG algorithm monitoring again.

9. A wearable device, characterized in that, For performing the multimodal fusion decoupling as described in any one of claims 1-8 The wearable device includes at least one of the following sensors: an optical vital signs sensor, an inertial sensor, a temperature sensor, and a GPS / BeiDou sensor.

Images