A time sequence synchronization method, system, terminal and storage medium for multi-modal wearable sensor data fusion

Abstract

The application discloses a kind of multi-modal wearable sensor data fusion-oriented timing synchronization method, system, terminal and storage medium, the method includes: obtaining absolute timestamp as global clock source;Global clock source is as the master clock of precision time protocol, a plurality of wearable sensors are configured as slave clock, through precision time protocol, multiple rounds of message interaction are performed between master clock and slave clock, and the timestamp corresponding to each round of interaction is obtained;According to the corresponding timestamp, the current path delay and the current clock offset are calculated, and according to the preset path delay threshold and clock offset threshold, the current path delay and the current clock offset are iteratively judged, whether the local clock is corrected according to the current clock offset is determined according to the iterative judgment result, until slave clock and global clock source achieve timing synchronization;After achieving time synchronization, the periodic data sampling tasks of a plurality of wearable sensors are phase-aligned.The application significantly improves timing synchronization accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a time synchronization method, system, terminal, and computer-readable storage medium for multimodal wearable sensor data fusion. Background Technology

[0002] In the research and application of smart wearable technologies related to sports and health, data fusion from multiple sensors has been a focus. In particular, the combination of physiological electrical signal sensors (such as electrocardiograms and electromyograms) and motion sensors plays a crucial role in monitoring human exercise and health. Physiological electrical signal sensors can provide important information such as heart rate, heart rate variability, and muscle activity, while motion sensors provide information on exercise status, posture, and intensity. The fusion of these two technologies enhances the comprehensive assessment of an individual's exercise and health status. By analyzing the fused data, real-time monitoring of physiological indicators during exercise is possible.

[0003] Data fusion and monitoring of multimodal wearable sensors typically require wearing multiple sensors on different parts of the body. The signals from these sensors need to be transmitted to a specific receiver (software platform) via a Wireless Body Sensor Network (WBSN). However, under wireless network communication conditions, the receiver's timing synchronization is insufficient when processing concurrent data transmissions from multiple wireless sensor nodes. This prevents zero-latency, microsecond-level data frame alignment between sensors, leading to inconsistencies in multimodal wearable sensor data acquisition and affecting the accuracy of data analysis.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a timing synchronization method, system, terminal, and computer-readable storage medium for multimodal wearable sensor data fusion. It aims to solve the problem that in the prior art, the timing synchronization accuracy of the receiving end is insufficient when processing concurrent data transmission from multiple wireless sensor nodes, and it is impossible to achieve zero-delay, microsecond-level data frame alignment between sensors, resulting in inconsistent data acquisition of multimodal wearable sensors and difficulty in guaranteeing the accuracy of data analysis.

[0006] To achieve the above objectives, the present invention provides a timing synchronization method for multimodal wearable sensor data fusion, the timing synchronization method for multimodal wearable sensor data fusion comprising the following steps:

[0007] A wearable real-time clock module is deployed in a wireless body domain sensor network. The wearable real-time clock module is connected to the Internet and an absolute timestamp is obtained from the Internet time server via the Network Time Protocol as the global clock source of the wireless body domain sensor network.

[0008] The global clock source is used as the master clock of the precision time protocol, and multiple wearable sensors are configured as slave clocks. Through the precision time protocol, multiple rounds of message interaction are performed between the master clock and the slave clocks to obtain the timestamp corresponding to each round of interaction.

[0009] For each round of interaction, the current path delay and current clock offset are calculated based on the corresponding timestamp. Based on the preset path delay threshold and clock offset threshold, the current path delay and current clock offset are iteratively judged. Based on the iterative judgment result, it is decided whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve timing synchronization.

[0010] After achieving time synchronization, the periodic data sampling tasks of the multiple wearable sensors are phase-aligned so that the multiple wearable sensors perform data acquisition at the same physical moment.

[0011] Optionally, in the aforementioned timing synchronization method for multimodal wearable sensor data fusion, both the wearable real-time clock module and the wearable sensor use a microcontroller with integrated WiFi communication function as the main control chip.

[0012] The master clock periodically sends time synchronization information to the slave clock via multicast through a wireless network.

[0013] Optionally, in the timing synchronization method for multimodal wearable sensor data fusion, the timestamp includes: a local transmission timestamp, a local reception timestamp, a slave clock timestamp, and a master clock reception timestamp.

[0014] The timestamp is obtained by performing message exchange between the master clock and the slave clock using the precise time protocol, specifically including:

[0015] The master clock sends a synchronization message to the slave clock, and records a local transmission timestamp at the moment of transmission.

[0016] The synchronization message is received from the clock, and a local timestamp is recorded at the moment of reception.

[0017] After the hardware or software timestamp measurement is completed, a follow message is sent to the slave clock again through the master clock. The follow message carries the local transmission timestamp of the synchronization message.

[0018] The following message is received from the clock, and the local transmission timestamp is obtained from the following message;

[0019] The slave clock sends a delay request message to the master clock and records the slave clock timestamp at the moment of transmission.

[0020] The master clock receives the delay request message and records the master clock's reception timestamp at the moment of reception;

[0021] The master clock sends a delayed response message to the slave clock, and the delayed response message carries the master clock's received timestamp.

[0022] Optionally, the timing synchronization method for multimodal wearable sensor data fusion, wherein calculating the current path delay and current clock offset based on the corresponding timestamp specifically includes:

[0023] Based on the assumption of symmetric network path round-trip delay between master and slave clocks, the current path delay and current clock offset are obtained according to the local transmit timestamp, the local receive timestamp, the slave clock timestamp, and the master clock receive timestamp:

[0024] ;

[0025] ;

[0026] in, Indicates the current clock offset, This indicates the current path delay. This represents the local transmission timestamp. This indicates the local received timestamp. This indicates the clock timestamp. This indicates the timestamp received by the master clock. Indicates the master clock. This indicates the clock, and the subscript number indicates the order of events.

[0027] Optionally, the timing synchronization method for multimodal wearable sensor data fusion, wherein the step of iteratively judging the current path delay and the current clock offset based on preset path delay thresholds and clock offset thresholds, and deciding whether to correct the local clock according to the current clock offset based on the iterative judgment result, until the slave clock achieves timing synchronization with the global clock source, specifically includes:

[0028] If the current path delay is greater than the path delay threshold, the result of this round is determined to be an abnormal sample and discarded. No clock correction is performed, and the next round of iteration begins.

[0029] If the current path delay is less than or equal to the path delay threshold, and the current clock offset is greater than the clock offset threshold, then the local clock of the slave clock is corrected using the current clock offset;

[0030] If the current path delay is less than or equal to the path delay threshold, and the current clock offset is less than or equal to the clock offset threshold, then this round is recorded as a valid synchronization round.

[0031] When the number of consecutively recorded valid synchronization cycles reaches a preset number, it is determined that the slave clock has successfully synchronized with the global clock source.

[0032] Optionally, the timing synchronization method for multimodal wearable sensor data fusion, wherein, after achieving time synchronization, phase alignment is performed on the periodic data sampling tasks of the multiple wearable sensors, specifically includes:

[0033] Once time synchronization is complete, calculate the phase offset between the slave clock at the current moment and the next preset sampling period time node;

[0034] Set the timer's duration to the phase offset and start the timer;

[0035] When the timer reaches the specified duration, the slave clock is controlled to perform a data acquisition task, and the slave clock is kept in sync with the other slave clocks in the wireless body domain sensor network.

[0036] Optionally, the timing synchronization method for multimodal wearable sensor data fusion further includes: performing the following task flow in the wearable sensor:

[0037] Start and run the data acquisition task, which is triggered by a timed interrupt and executed cyclically according to a fixed period to acquire raw sensor data at equal intervals;

[0038] A high-precision timing synchronization task is started and run concurrently. The high-precision timing synchronization task is used to perform message interaction, iterative judgment, clock correction and sampling phase alignment.

[0039] After the high-precision timing synchronization task is successfully completed, the original sensor data acquired by the data acquisition task is timestamped based on the synchronized local clock, and the timestamped sensor data is sent to the cloud server.

[0040] Furthermore, to achieve the above objectives, the present invention also provides a timing synchronization system for multimodal wearable sensor data fusion, wherein the timing synchronization system for multimodal wearable sensor data fusion includes:

[0041] A global clock establishment module is used to deploy a wearable real-time clock module in a wireless body domain sensor network, connect the wearable real-time clock module to the Internet, and obtain an absolute timestamp from the time server of the Internet through the Network Time Protocol as the global clock source of the wireless body domain sensor network.

[0042] The master-slave interaction module is used to use the global clock source as the master clock of the precision time protocol, configure multiple wearable sensors as slave clocks, and perform multiple rounds of message interaction between the master clock and the slave clock through the precision time protocol to obtain the timestamp corresponding to each round of interaction.

[0043] The judgment and correction module is used to calculate the current path delay and current clock offset based on the corresponding timestamp for each round of interaction, and to iteratively judge the current path delay and current clock offset according to the preset path delay threshold and clock offset threshold. Based on the iterative judgment result, it decides whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve timing synchronization.

[0044] The phase alignment module is used to perform phase alignment on the periodic data sampling tasks of multiple wearable sensors after time synchronization is achieved, so that the multiple wearable sensors can perform data acquisition at the same physical moment.

[0045] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a timing synchronization program for multimodal wearable sensor data fusion stored in the memory and executable on the processor, wherein when the timing synchronization program for multimodal wearable sensor data fusion is executed by the processor, it implements the steps of the timing synchronization method for multimodal wearable sensor data fusion as described above.

[0046] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a timing synchronization program for multimodal wearable sensor data fusion, and when the timing synchronization program for multimodal wearable sensor data fusion is executed by a processor, it implements the steps of the timing synchronization method for multimodal wearable sensor data fusion as described above.

[0047] In this invention, an absolute timestamp is obtained as the global clock source. A wearable real-time clock module is deployed in a wireless body-domain sensor network, connected to the Internet, and obtains an absolute timestamp from an Internet time server via a network time protocol as the global clock source for the wireless body-domain sensor network. Multiple wearable sensors are configured as slave clocks. Through a precision time protocol, multiple rounds of message exchanges are performed between the master clock and slave clocks to obtain the timestamp corresponding to each round of exchanges. The current path delay and current clock offset are calculated based on the corresponding timestamps. Based on preset path delay thresholds and clock offset thresholds, the current path delay and current clock offset are iteratively judged. The iterative judgment result determines whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve time synchronization. After time synchronization is achieved, the periodic data sampling tasks of multiple wearable sensors are phase-aligned. This invention significantly improves the time synchronization accuracy of multimodal wearable sensors in the data fusion process, ensuring the consistency and accuracy of sensor data. Attached Figure Description

[0048] Figure 1 This is a flowchart of a preferred embodiment of the timing synchronization method for multimodal wearable sensor data fusion of the present invention;

[0049] Figure 2 This is a technical schematic diagram of the timing synchronization method for multimodal wearable sensor data fusion according to the present invention;

[0050] Figure 3 This is a task flowchart for wearable sensors in the timing synchronization method for multimodal wearable sensor data fusion of the present invention.

[0051] Figure 4 This is a schematic diagram illustrating the principle of sampling phase alignment in the timing synchronization method for multimodal wearable sensor data fusion of the present invention;

[0052] Figure 5 This is a structural diagram of a preferred embodiment of the timing synchronization system for multimodal wearable sensor data fusion of the present invention;

[0053] Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0054] This application provides a timing synchronization method, system, and terminal for multimodal wearable sensor data fusion. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0056] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0057] The preferred embodiment of the present invention describes a timing synchronization method for multimodal wearable sensor data fusion, such as... Figure 1 and Figure 2 As shown, the timing synchronization method for multimodal wearable sensor data fusion includes the following steps:

[0058] Step S10: Deploy a wearable real-time clock module in the wireless body domain sensor network, connect the wearable real-time clock module to the Internet, and obtain an absolute timestamp from the time server of the Internet through the Network Time Protocol as the global clock source of the wireless body domain sensor network.

[0059] This invention uses the ESP32 as the main control chip for both the RTC module (Real-Time Clock, wearable real-time clock module) and the sensor module, enabling both to communicate wirelessly via WiFi. The RTC module acts as a global network clock source, periodically obtaining a high-precision absolute timestamp from an internet time server via NTP (Network Time Protocol) to calibrate its internal clock, ensuring long-term consistency with standard time.

[0060] Specifically, in the wireless body-domain sensor network, a wearable real-time clock module based on a microcontroller (preferably an ESP32 series chip with integrated Wi-Fi functionality) is deployed. This RTC module connects to a local wireless router through its wireless communication unit, thereby accessing the Internet and completing the basic network layer connection.

[0061] The RTC module runs a Network Time Protocol (NTP) client program, which periodically sends time query requests to one or more time service centers (i.e., NTP time servers) located on the Internet. Through NTP protocol exchange, the RTC module obtains high-precision absolute timestamp information, as well as related clock precision, jitter, and other parameters from the server's response.

[0062] Furthermore, the RTC module uses the acquired absolute timestamp to calibrate its internal hardware clock / timer to eliminate accumulated errors in the local clock. This calibration process is performed continuously and periodically, ensuring that the RTC module's local clock maintains microsecond to millisecond-level synchronization accuracy with Coordinated Universal Time (UTC) or other standard times over a long period. After calibration, the RTC module's clock is established as the unique and reliable global clock source within the entire wireless volume domain sensor network.

[0063] As can be seen, this invention directly establishes a self-contained, high-precision global clock source within the sensor network. This eliminates the need for external edge computers or host computers as time mediators, simplifying the network topology and reducing system complexity and cost. Furthermore, because the clock source is located within the network and continuously synchronized with standard time, it establishes an absolute time reference for subsequent microsecond-level high-precision relative synchronization between sensor nodes, ensuring the long-term accuracy and consistency of the entire system's time system.

[0064] Step S20: Use the global clock source as the master clock of the precision time protocol, configure multiple wearable sensors as slave clocks, and perform multiple rounds of message interaction between the master clock and the slave clocks through the precision time protocol to obtain the timestamp corresponding to each round of interaction.

[0065] Based on the global clock source established in the wireless volume domain sensor network, a master-slave hierarchical architecture using the PTP (Precision Time Protocol) is adopted within the wireless volume domain sensor network. The RTC module acts as the master clock node of the PTP, broadcasting time synchronization information to the wearable sensors, which act as slave clock nodes, via UDP multicast.

[0066] Both the wearable real-time clock module and the wearable sensor use a microcontroller with integrated WiFi communication function as the main control chip.

[0067] Furthermore, the timestamp includes: a local transmission timestamp, a local reception timestamp, a slave clock timestamp, and a master clock reception timestamp.

[0068] The master clock and the slave clock will perform multiple rounds of message exchange. This embodiment takes one round of message exchange as an example. Through the precise time protocol, message exchange is performed between the master clock and the slave clock to obtain a timestamp, specifically including:

[0069] The master clock sends a synchronization message to the slave clock, and records a local transmission timestamp at the moment of transmission.

[0070] The synchronization message is received from the clock, and a local timestamp is recorded at the moment of reception.

[0071] After the hardware or software timestamp measurement is completed, a follow message is sent to the slave clock again through the master clock. The follow message carries the local transmission timestamp of the synchronization message.

[0072] The following message is received from the clock, and the local transmission timestamp is obtained from the following message;

[0073] The slave clock sends a delay request message to the master clock and records the slave clock timestamp at the moment of transmission.

[0074] The master clock receives the delay request message and records the master clock's reception timestamp at the moment of reception;

[0075] The master clock sends a delayed response message to the slave clock, and the delayed response message carries the master clock's received timestamp.

[0076] Understandably, the master and slave nodes periodically exchange PTP messages on a pre-defined UDP multicast address or multicast group to acquire and calculate bidirectional timestamp information. The PTP message types include: Synchronization message (SYNC): used to synchronize the master and slave clocks, sending the current timestamp. Follow message (FOLLOW_UP): used to provide additional information about the timestamp in the synchronization message. Delay request message (DELAY_REQ): the slave clock sends a delay request to the master clock to calculate the round-trip delay. Delay response message (DELAY_RESP): after responding to the delay request, the master clock sends a message containing the timestamp of when it received the delay request.

[0077] During the above message exchange process, the PTP protocol obtains four key timestamps from the clock, including: the local transmission timestamp. (The time when the master clock sends the SYNC message), and the local received timestamp. (Time of receiving the SYNC message from the clock), from the clock timestamp (Time when the DELAY_REQ message is sent from the clock) and the master clock receiving timestamp (The time when the master clock receives the DELAY_REQ message).

[0078] Specifically, taking a single slave node as an example, the timestamp exchange process between the PTP master and slave clocks in one round includes the following parts:

[0079] Synchronization message transmission: The master clock records the local transmission timestamp at the instant it broadcasts a synchronization message (SYNC) to the slave clock. ;

[0080] Synchronization message reception: The slave clock records the local reception timestamp the instant it receives the synchronization message (SYNC). .

[0081] Follow-up message sending: In order to accurately send the timestamp ( The message is passed to the slave clock; the master clock does not include this message when sending the SYNC message. After completing the hardware or software timestamp measurement, send a follow-up message (FOLLOW_UP) to... The precise value is passed to the slave clock. This method reduces the real-time requirements of message transmission path timing and is suitable for implementation on resource-constrained platforms.

[0082] Delay Request Sending: The slave clock sends a Delay Request message DELAY_REQ to the master clock, and records the slave clock timestamp at the time of sending. .

[0083] Delayed Request Reception: The master clock records the reception timestamp the instant it receives the DELAY_REQ message. .

[0084] Delayed response transmission: The master clock sends a delayed response message DELAY_RESP, which includes the timestamp. Return it to the corresponding slave clock node.

[0085] This invention employs a precision time protocol message interaction step to decouple the generation of high-precision timestamps from the message sending action, ensuring that critical timestamps with microsecond-level precision can still be reliably captured on resource-constrained wearable embedded platforms, laying a solid foundation for all subsequent calculations.

[0086] Step S30: For each round of interaction, calculate the current path delay and current clock offset based on the corresponding timestamp, and iteratively judge the current path delay and current clock offset according to the preset path delay threshold and clock offset threshold. Based on the iterative judgment result, decide whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve timing synchronization.

[0087] The calculation of the current path delay and current clock offset based on the corresponding timestamp specifically includes:

[0088] Based on the assumption of symmetric network path round-trip delay between master and slave clocks, the current path delay and current clock offset are obtained according to the local transmit timestamp, the local receive timestamp, the slave clock timestamp, and the master clock receive timestamp:

[0089] ;

[0090] ;

[0091] in, Indicates the current clock offset, This indicates the current path delay. This represents the local transmission timestamp. This indicates the local received timestamp. This indicates the clock timestamp. This indicates the timestamp received by the master clock. Indicates the master clock. This indicates the clock, and the subscript number indicates the order of events.

[0092] It is understandable that the PTP protocol assumes that the round-trip time of the network path between the master and slave clocks is statistically symmetrical. Based on this, the clock offset between the master and slave clocks can be calculated. Transmission delay between master and slave links The specific calculation method is as shown in the formula above. Finally, the sensor (from the clock) obtains the current... Then, by calibrating the local timer, time alignment with the RTC module (master clock) can be achieved. Since the RTC module itself is synchronized with the Internet standard time, the sensor (slave clock) is indirectly aligned with a unified global time base. The above method is applicable to all sensors within the local area network, thus achieving time synchronization for all sensors within the local area network.

[0093] Further, the step of iteratively judging the current path delay and the current clock offset based on preset path delay thresholds and clock offset thresholds, and deciding whether to correct the local clock according to the current clock offset based on the iterative judgment result, until the slave clock achieves timing synchronization with the global clock source, specifically includes:

[0094] If the current path delay is greater than the path delay threshold, the result of this round is determined to be an abnormal sample and discarded. No clock correction is performed, and the next round of iteration begins.

[0095] If the current path delay is less than or equal to the path delay threshold, and the current clock offset is greater than the clock offset threshold, then the local clock of the slave clock is corrected using the current clock offset;

[0096] If the current path delay is less than or equal to the path delay threshold, and the current clock offset is less than or equal to the clock offset threshold, then this round is recorded as a valid synchronization round.

[0097] When the number of consecutively recorded valid synchronization cycles reaches a preset number, it is determined that the slave clock has successfully synchronized with the global clock source.

[0098] Understandably, the PTP protocol assumes that the round-trip time of the network path between the master and slave clocks is statistically symmetrical. However, in a typical WiFi network environment, the physical channel uses a CSMA / CA (Carrier Sense Multiple Access with Collision Avoidance) contention mechanism, resulting in significant instantaneous network latency jitter. The uplink and downlink during message exchange are difficult to make perfectly symmetrical, making it challenging to guarantee symmetrical round-trip time of the network path between the master and slave clocks at all times. Directly using a single PTP synchronization would introduce significant clock errors, leading to uncertainty in time synchronization, or even synchronization failure.

[0099] To address this, this invention introduces an iterative strategy based on a dynamic delay-offset threshold, building upon the traditional PTP mechanism. This strategy performs PTP synchronization iteratively and automatically implements multi-round PTP synchronization and result-based judgment and filtering at the embedded software level. The strategy mainly includes the following steps:

[0100] Dynamic Delay-Offset Threshold Setting: From an application perspective, potential users' WiFi network environments vary. Therefore, this invention allows users to preset path delay thresholds based on the synchronization accuracy required by the target application (e.g., 1ms) and anticipated wireless network conditions. and clock offset threshold The former limits the acceptable network latency for the user in the current network environment. The latter limits the maximum acceptable synchronization error for the user and is also used in iterative algorithms to determine whether the current master-slave clock offset needs to continue iterating (i.e., clock correction).

[0101] Multiple rounds of iteration and judgment: Under the premise of the above threshold setting, the master and slave clocks repeatedly execute the above PTP message interaction process and record the current path delay. With clock offset , where superscript Indicates the first Round iteration.

[0102] Dual threshold joint judgment: For the clock synchronization result after each iteration, record the current path delay. and current clock offset ,in, , .

[0103] When the current path delay is greater than the path delay threshold, i.e. > If the result is not found, it indicates that the network is congested or the signal quality is poor. This round's result is considered an "abnormal sample" and judged as a "synchronization failure." Without clock correction, the next iteration will proceed automatically.

[0104] When the current path latency is detected to be less than or equal to the path latency threshold, and the current clock offset is greater than the clock offset threshold, it indicates that the current WiFi network link is in a low-latency, relatively stable, and good state. However, if there is still a significant observable clock offset between the master and slave, the slave clock needs to utilize the current... Correct the slave clock.

[0105] Synchronization is considered successful when the current path delay is less than or equal to the path delay threshold and the current clock offset is greater than the clock offset threshold three times consecutively (reaching a preset number of times).

[0106] Through the above design, this invention adds an iterative filtering mechanism only to the communication application layer between the master and slave clocks without modifying the PTP message format and the underlying protocol stack. This mechanism utilizes the random fluctuations in wireless channel latency to proactively discard "abnormal samples" when the network condition deteriorates, and actively identifies and filters the instantaneous state of the WiFi link. It ensures that clock calibration is only performed when the WiFi environment is good and the current master-slave clock offset is significant, thereby solving the problem of accurate time synchronization for wearable sensors in wireless network environments and significantly improving the time success rate and synchronization accuracy between multiple sensor nodes.

[0107] Step S40: After achieving time synchronization, perform phase alignment on the periodic data sampling tasks of the multiple wearable sensors so that the multiple wearable sensors can perform data acquisition at the same physical moment.

[0108] like Figure 3 As shown, it is understandable that after the high-precision clock synchronization task is successfully executed, the timestamps of different sensors can be unified. However, after the sensor is powered on, it usually automatically executes the task of "acquiring sensor data from the sensor chip".

[0109] Specifically, the timing synchronization method for multimodal wearable sensor data fusion further includes: performing the following task flow in the wearable sensor:

[0110] Start and run the data acquisition task, which is triggered by a timed interrupt and executed cyclically according to a fixed period to acquire raw sensor data at equal intervals;

[0111] A high-precision timing synchronization task is started and run concurrently. The high-precision timing synchronization task is used to perform message interaction, iterative judgment, clock correction and sampling phase alignment.

[0112] After the high-precision timing synchronization task is successfully completed, the original sensor data acquired by the data acquisition task is timestamped based on the synchronized local clock, and the timestamped sensor data is sent to the cloud server.

[0113] In other words, the task of "acquiring sensor data from the sensor chip" automatically enters a loop with 10ms intervals after power-on, based on a timer interrupt, thereby achieving 100Hz cyclic isochronous sampling. Simultaneously, the main control chip performs a high-precision timing synchronization task and timestamps the sensor data upon completion. Ultimately, the data sent to the cloud server (data receiving software) includes both sensor data and timestamps.

[0114] However, when the frequency is set to 100Hz, this task will perform cyclic isochronous sampling at 10ms intervals based on timer interrupts. That is, the sensor data exists within a 10ms isochronous grid. However, as... Figure 4 As shown in (a), in practical applications, the power-on time of wearable sensors may vary. This can lead to a difference in the actual time for the sensor to acquire data after clock synchronization and timestamp alignment due to the presence of isochronous grids, resulting in a phase offset time of less than 10ms.

[0115] To address this issue, this invention proposes a sampling phase alignment strategy. The step of performing phase alignment on the periodic data sampling tasks of multiple wearable sensors after achieving time synchronization specifically includes:

[0116] Once time synchronization is complete, calculate the phase offset between the slave clock at the current moment and the next preset sampling period time node;

[0117] Set the timer's duration to the phase offset and start the timer;

[0118] When the timer reaches the specified duration, the slave clock is controlled to perform a data acquisition task, and the slave clock is kept in sync with the other slave clocks in the wireless body domain sensor network.

[0119] like Figure 4 As shown in (b), to ensure that multiple sensors acquire data at the same physical moment, the embedded system of the sensors calculates the phase offset time of the current clock alignment relative to the next 10ms isochronous grid after clock synchronization is completed. Subsequently, a single-trigger timer is started to perform phase compensation. This timer triggers a clock callback when the phase offset time is exhausted, thereby unifying the time for all sensors to perform sensor data acquisition and transmission tasks to the same physical moment. Based on this moment, isochronous sampling of data at the same time is continued through high-precision timer interrupts.

[0120] Furthermore, to verify the data synchronization effect obtained by the above method, the experiment used 5 wearable sensors and 1 RTC module to form a wearable body sensor network, and sent the sensor data and RTC timestamps to the data receiving end in the cloud server through the WiFi signal of a wireless router (TP LinkWDR5600AC).

[0121] The experiment recorded the timestamps of the first frame received by the cloud server from each sensor (sensor-1 to sensor-5) and the RTC module under two conditions: "no sampling phase alignment after high-precision clock synchronization" (Experiment 1) and "sampling phase alignment after high-precision clock synchronization" (Experiment 2). To verify the stability of the method, the experiment was repeated 5 times. The clock offset threshold was set to 1000 μs, and the path delay threshold was set to 5000 μs.

[0122] Specifically, as shown in Table 1 below, Table 1 presents the timestamp results (13-bit UNIX timestamp accurate to milliseconds) obtained in 5 experiments under the condition of "no sampling phase alignment after high-precision clock synchronization" (Experiment 1).

[0123] Table 1: Timestamp Results of the First Frame of Experiment 1 Data

[0124]

[0125] As shown in Table 2, after converting the timestamp to UTD+8 Beijing time, the time differences between each sensor (sensor-1, sensor-2, sensor-3, sensor-4, sensor-5) and the RTC module are as follows. It can be seen that all time differences are less than 10ms. This indicates that the high-precision clock synchronization was successful, but there is a phase offset in the sensor data acquisition time.

[0126] Table 2: Time difference between the first frame for each sensor and RTC module in Experiment 1

[0127]

[0128] Table 3 below shows the timestamp results obtained in five experiments under the condition of "high-precision clock synchronization followed by sampling phase alignment" (Experiment 2). It can be seen that after converting the timestamps to Beijing time (UTD+8), the time difference between each sensor and the RTC module is significant. It can also be seen that in all five experiments, the timestamps of the first frame received by the cloud server from each sensor and the RTC module are completely consistent. This result demonstrates that the high-precision timing synchronization method proposed in this invention is effective and stable.

[0129] Table 3: Timestamp Results of the First Frame of Experiment 2 Data

[0130]

[0131] As can be seen, this invention employs a miniature wearable real-time clock (RTC) module deployed within a wearable sensor body network as the network's global clock source. This allows the module to directly obtain clock information from an internet time synchronization center via an internet connection, enabling wearable sensor data (without time synchronization via an edge computer) to be directly sent and stored on a cloud server. Simultaneously, the innovative PTP precision time protocol proposed in this invention, based on a wireless network, accurately synchronizes the wearable sensor's clock with the RTC module's clock, thereby achieving precise time synchronization for the wearable sensor. The most crucial element is the delay-offset dynamic threshold iterative synchronization strategy in a wireless network environment. Furthermore, a sampling phase alignment strategy ensures that multiple wearable sensors collect and transmit data at the same physical moment, enabling millisecond-level data timing alignment.

[0132] Furthermore, such as Figure 5 As shown, based on the above-mentioned timing synchronization method for multimodal wearable sensor data fusion, the present invention also provides a timing synchronization system for multimodal wearable sensor data fusion, wherein the timing synchronization system for multimodal wearable sensor data fusion includes:

[0133] The global clock establishment module 51 is used to deploy a wearable real-time clock module in a wireless body domain sensor network, connect the wearable real-time clock module to the Internet, and obtain an absolute timestamp from the time server of the Internet through the Network Time Protocol as the global clock source of the wireless body domain sensor network.

[0134] The master-slave interaction module 52 is used to use the global clock source as the master clock of the precision time protocol, configure multiple wearable sensors as slave clocks, and perform multiple rounds of message interaction between the master clock and the slave clock through the precision time protocol to obtain the timestamp corresponding to each round of interaction.

[0135] The judgment and correction module 53 is used to calculate the current path delay and current clock offset based on the corresponding timestamp for each round of interaction, and to perform iterative judgment on the current path delay and current clock offset based on the preset path delay threshold and clock offset threshold, and to decide whether to correct the local clock according to the current clock offset based on the iterative judgment result, until the slave clock and the global clock source achieve timing synchronization.

[0136] The phase alignment module 54 is used to perform phase alignment on the periodic data sampling tasks of multiple wearable sensors after time synchronization is achieved, so that multiple wearable sensors can perform data acquisition at the same physical moment.

[0137] Furthermore, such as Figure 6 As shown, based on the above-mentioned timing synchronization method and system for multimodal wearable sensor data fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0138] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a timing synchronization program 40 for multimodal wearable sensor data fusion, which can be executed by the processor 10 to implement the timing synchronization method for multimodal wearable sensor data fusion in this application.

[0139] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the timing synchronization method for multimodal wearable sensor data fusion.

[0140] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0141] In one embodiment, when the processor 10 executes the timing synchronization program 40 for multimodal wearable sensor data fusion in the memory 20, the following steps are performed:

[0142] A wearable real-time clock module is deployed in a wireless body domain sensor network. The wearable real-time clock module is connected to the Internet and an absolute timestamp is obtained from the Internet time server via the Network Time Protocol as the global clock source of the wireless body domain sensor network.

[0143] The global clock source is used as the master clock of the precision time protocol, and multiple wearable sensors are configured as slave clocks. Through the precision time protocol, multiple rounds of message interaction are performed between the master clock and the slave clocks to obtain the timestamp corresponding to each round of interaction.

[0144] For each round of interaction, the current path delay and current clock offset are calculated based on the corresponding timestamp. Based on the preset path delay threshold and clock offset threshold, the current path delay and current clock offset are iteratively judged. Based on the iterative judgment result, it is decided whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve timing synchronization.

[0145] After achieving time synchronization, the periodic data sampling tasks of the multiple wearable sensors are phase-aligned so that the multiple wearable sensors perform data acquisition at the same physical moment.

[0146] Both the wearable real-time clock module and the wearable sensor use a microcontroller with integrated WiFi communication function as the main control chip.

[0147] The master clock periodically sends time synchronization information to the slave clock via multicast through a wireless network.

[0148] The timestamps include: local transmission timestamp, local reception timestamp, slave clock timestamp, and master clock reception timestamp;

[0149] The timestamp is obtained by performing message exchange between the master clock and the slave clock using the precise time protocol, specifically including:

[0150] The master clock sends a synchronization message to the slave clock, and records a local transmission timestamp at the moment of transmission.

[0151] The synchronization message is received from the clock, and a local timestamp is recorded at the moment of reception.

[0152] After the hardware or software timestamp measurement is completed, a follow message is sent to the slave clock again through the master clock. The follow message carries the local transmission timestamp of the synchronization message.

[0153] The following message is received from the clock, and the local transmission timestamp is obtained from the following message;

[0154] The slave clock sends a delay request message to the master clock and records the slave clock timestamp at the moment of transmission.

[0155] The master clock receives the delay request message and records the master clock's reception timestamp at the moment of reception;

[0156] The master clock sends a delayed response message to the slave clock, and the delayed response message carries the master clock's received timestamp.

[0157] Specifically, the calculation of the current path delay and current clock offset based on the corresponding timestamp includes:

[0158] Based on the assumption of symmetric network path round-trip delay between master and slave clocks, the current path delay and current clock offset are obtained according to the local transmit timestamp, the local receive timestamp, the slave clock timestamp, and the master clock receive timestamp:

[0159] ;

[0160] ;

[0161] in, Indicates the current clock offset, This indicates the current path delay. This represents the local transmission timestamp. This indicates the local received timestamp. This indicates the clock timestamp. This indicates the timestamp received by the master clock. Indicates the master clock. This indicates the clock, and the subscript number indicates the order of events.

[0162] Specifically, the step of iteratively judging the current path delay and the current clock offset based on preset path delay thresholds and clock offset thresholds, and determining whether to correct the local clock according to the current clock offset based on the iterative judgment result, until the slave clock achieves timing synchronization with the global clock source, includes:

[0163] If the current path delay is greater than the path delay threshold, the result of this round is determined to be an abnormal sample and discarded. No clock correction is performed, and the next round of iteration begins.

[0164] If the current path delay is less than or equal to the path delay threshold, and the current clock offset is greater than the clock offset threshold, then the local clock of the slave clock is corrected using the current clock offset;

[0165] If the current path delay is less than or equal to the path delay threshold, and the current clock offset is less than or equal to the clock offset threshold, then this round is recorded as a valid synchronization round.

[0166] When the number of consecutively recorded valid synchronization cycles reaches a preset number, it is determined that the slave clock has successfully synchronized with the global clock source.

[0167] Specifically, the step of performing phase alignment on the periodic data sampling tasks of multiple wearable sensors after achieving time synchronization includes:

[0168] Once time synchronization is complete, calculate the phase offset between the slave clock at the current moment and the next preset sampling period time node;

[0169] Set the timer's duration to the phase offset and start the timer;

[0170] When the timer reaches the specified duration, the slave clock is controlled to perform a data acquisition task, and the slave clock is kept in sync with the other slave clocks in the wireless body domain sensor network.

[0171] The timing synchronization method for multimodal wearable sensor data fusion further includes: performing the following task flow in the wearable sensor:

[0172] Start and run the data acquisition task, which is triggered by a timed interrupt and executed cyclically according to a fixed period to acquire raw sensor data at equal intervals;

[0173] A high-precision timing synchronization task is started and run concurrently. The high-precision timing synchronization task is used to perform message interaction, iterative judgment, clock correction and sampling phase alignment.

[0174] After the high-precision timing synchronization task is successfully completed, the original sensor data acquired by the data acquisition task is timestamped based on the synchronized local clock, and the timestamped sensor data is sent to the cloud server.

[0175] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a timing synchronization program for multimodal wearable sensor data fusion, and the timing synchronization program for multimodal wearable sensor data fusion, when executed by a processor, implements the steps of the timing synchronization method for multimodal wearable sensor data fusion as described above.

[0176] In summary, this invention provides a timing synchronization method, system, terminal, and storage medium for multimodal wearable sensor data fusion. The method includes: acquiring an absolute timestamp as a global clock source; deploying a wearable real-time clock module in a wireless body domain sensor network, connecting the wearable real-time clock module to the Internet, and obtaining an absolute timestamp from an Internet time server via a network time protocol as the global clock source for the wireless body domain sensor network; configuring multiple wearable sensors as slave clocks; performing multiple rounds of message interaction between the master clock and slave clocks via a precision time protocol to obtain the timestamp corresponding to each round of interaction; calculating the current path delay and current clock offset based on the corresponding timestamps, and iteratively judging the current path delay and current clock offset according to preset path delay thresholds and clock offset thresholds; determining whether to correct the local clock according to the current clock offset based on the iterative judgment result, until the slave clock and the global clock source achieve timing synchronization; after achieving time synchronization, performing phase alignment on the periodic data sampling tasks of multiple wearable sensors. This invention significantly improves the timing synchronization accuracy of multimodal wearable sensors during data fusion, ensuring the consistency and accuracy of sensor data.

[0177] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0178] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0179] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A timing synchronization method for data fusion in multimodal wearable sensors, characterized in that, The aforementioned timing synchronization method for multimodal wearable sensor data fusion includes: A wearable real-time clock module is deployed in a wireless body domain sensor network. The wearable real-time clock module is connected to the Internet and an absolute timestamp is obtained from the Internet time server via the Network Time Protocol as the global clock source of the wireless body domain sensor network. The global clock source is used as the master clock of the precision time protocol, and multiple wearable sensors are configured as slave clocks. Through the precision time protocol, multiple rounds of message interaction are performed between the master clock and the slave clocks to obtain the timestamp corresponding to each round of interaction. For each round of interaction, the current path delay and current clock offset are calculated based on the corresponding timestamp. Based on the preset path delay threshold and clock offset threshold, the current path delay and current clock offset are iteratively judged. Based on the iterative judgment result, it is decided whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve timing synchronization. After achieving time synchronization, the periodic data sampling tasks of the multiple wearable sensors are phase aligned so that the multiple wearable sensors can perform data acquisition at the same physical moment. The process of iteratively judging the current path delay and the current clock offset based on preset path delay and clock offset thresholds, and determining whether to correct the local clock according to the current clock offset based on the iterative judgment result, until the slave clock achieves timing synchronization with the global clock source, specifically includes: If the current path delay is greater than the path delay threshold, the result of this round is determined to be an abnormal sample and discarded. No clock correction is performed, and the next round of iteration begins. If the current path delay is less than or equal to the path delay threshold, and the current clock offset is greater than the clock offset threshold, then the local clock of the slave clock is corrected using the current clock offset; If the current path delay is less than or equal to the path delay threshold, and the current clock offset is less than or equal to the clock offset threshold, then this round is recorded as a valid synchronization round. When the number of consecutively recorded valid synchronization rounds reaches a preset number, it is determined that the slave clock has successfully synchronized with the global clock source; The path delay threshold and clock offset threshold are dynamic thresholds pre-configured by the user based on the target synchronization accuracy and wireless network conditions. After achieving time synchronization, the phase alignment of the periodic data sampling tasks of the multiple wearable sensors specifically includes: Once time synchronization is complete, calculate the phase offset between the slave clock at the current moment and the next preset sampling period time node; Set the timer's duration to the phase offset and start the timer; When the timer reaches the specified duration, the slave clock is controlled to perform a data acquisition task, and the slave clock is kept in sync with the sampling period of other slave clocks in the wireless body domain sensor network. The timer is a single-trigger timer that triggers a clock callback when the phase offset time is exhausted, so as to unify the start time of multiple wearable sensors performing data acquisition tasks to the same physical time, and use this time as a reference to achieve subsequent isochronous sampling through high-precision timer interrupt.

2. The timing synchronization method for multimodal wearable sensor data fusion according to claim 1, characterized in that, Both the wearable real-time clock module and the wearable sensor use a microcontroller with integrated WiFi communication function as the main control chip. The master clock periodically sends time synchronization information to the slave clock via multicast through a wireless network.

3. The timing synchronization method for multimodal wearable sensor data fusion according to claim 1, characterized in that, The timestamps include: local transmission timestamp, local reception timestamp, slave clock timestamp, and master clock reception timestamp; The timestamp is obtained by performing message exchange between the master clock and the slave clock using the precise time protocol, specifically including: The master clock sends a synchronization message to the slave clock, and records a local transmission timestamp at the moment of transmission. The synchronization message is received from the clock, and a local timestamp is recorded at the moment of reception. After the hardware or software timestamp measurement is completed, a follow message is sent to the slave clock again through the master clock. The follow message carries the local transmission timestamp of the synchronization message. The following message is received from the clock, and the local transmission timestamp is obtained from the following message; The slave clock sends a delay request message to the master clock and records the slave clock timestamp at the moment of transmission. The master clock receives the delay request message and records the master clock's reception timestamp at the moment of reception; The master clock sends a delayed response message to the slave clock, and the delayed response message carries the master clock's received timestamp.

4. The timing synchronization method for multimodal wearable sensor data fusion according to claim 3, characterized in that, The calculation of the current path delay and current clock offset based on the corresponding timestamp specifically includes: Based on the assumption of symmetric network path round-trip delay between master and slave clocks, the current path delay and current clock offset are obtained according to the local transmit timestamp, the local receive timestamp, the slave clock timestamp, and the master clock receive timestamp: ； ； in, Indicates the current clock offset, This indicates the current path delay. This represents the local transmission timestamp. This indicates the local received timestamp. This indicates the clock timestamp. This indicates the timestamp received by the master clock. Indicates the master clock. This indicates the clock, and the subscript number indicates the order of events.

5. The timing synchronization method for multimodal wearable sensor data fusion according to claim 1, characterized in that, The timing synchronization method for multimodal wearable sensor data fusion further includes: performing the following task flow in the wearable sensor: Start and run the data acquisition task, which is triggered by a timed interrupt and executed cyclically according to a fixed period to acquire raw sensor data at equal intervals; A high-precision timing synchronization task is started and run concurrently. The high-precision timing synchronization task is used to perform message interaction, iterative judgment, clock correction and sampling phase alignment. After the high-precision timing synchronization task is successfully completed, the original sensor data acquired by the data acquisition task is timestamped based on the synchronized local clock, and the timestamped sensor data is sent to the cloud server.

6. A timing synchronization system for multimodal wearable sensor data fusion, characterized in that, The timing synchronization system for multimodal wearable sensor data fusion is used to implement the timing synchronization method for multimodal wearable sensor data fusion as described in any one of claims 1-5, wherein the timing synchronization system for multimodal wearable sensor data fusion comprises: A global clock establishment module is used to deploy a wearable real-time clock module in a wireless body domain sensor network, connect the wearable real-time clock module to the Internet, and obtain an absolute timestamp from the time server of the Internet through the Network Time Protocol as the global clock source of the wireless body domain sensor network. The master-slave interaction module is used to use the global clock source as the master clock of the precision time protocol, configure multiple wearable sensors as slave clocks, and perform multiple rounds of message interaction between the master clock and the slave clock through the precision time protocol to obtain the timestamp corresponding to each round of interaction. The judgment and correction module is used to calculate the current path delay and current clock offset based on the corresponding timestamp for each round of interaction, and to iteratively judge the current path delay and current clock offset according to the preset path delay threshold and clock offset threshold. Based on the iterative judgment result, it decides whether to correct the local clock according to the current clock offset until the slave clock and the global clock source achieve timing synchronization. The phase alignment module is used to perform phase alignment on the periodic data sampling tasks of multiple wearable sensors after time synchronization is achieved, so that the multiple wearable sensors can perform data acquisition at the same physical moment.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a timing synchronization program for multimodal wearable sensor data fusion stored in the memory and executable on the processor. When the timing synchronization program for multimodal wearable sensor data fusion is executed by the processor, it implements the steps of the timing synchronization method for multimodal wearable sensor data fusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a timing synchronization program for multimodal wearable sensor data fusion, which, when executed by a processor, implements the steps of the timing synchronization method for multimodal wearable sensor data fusion as described in any one of claims 1-5.

Images