Timestamp estimation method of double-tense non-uniform sampling data stream and marking system of double-tense non-uniform sampling data stream

Abstract

The invention discloses a timestamp estimation method for a dual-tense non-uniform sampling data stream. The timestamp estimation method comprises the following steps: step 1, starting a UI thread and a timer thread; 2, motion links are segmented according to a state machine, soft trigger signals are sent to all the devices on the basis of the state machine, and it is ensured that all the devices conduct synchronous collection in the specific motion stage; 3, setting an experiment process in a graphical mode; 4, a Vertx bus is used for building a data channel, and experimental data are transmitted in an asynchronous mode; and 5, reconstructing the timestamps of the experimental data collected in the step 4, and uniformly marking the experimental data after the timestamps are reconstructed. According to the invention, the problems of data loss and out-of-order timestamps caused by data transmission based on a network protocol are solved.

Description

Technical Field

[0001] The present invention belongs to the field of data communication, and relates to a technology for processing timestamps of data. Specifically, it is a method for estimating timestamps of a dual-temporal non-uniform sampling data stream and its tagging system. Background Art

[0002] During the network transmission of data, there are two important temporal states: the time point when the data is generated at the source (such as sensors, devices) and the time when the data is received by the receiving end (such as servers, controllers), that is, dual-temporal.

[0003] In the field of rehabilitation medicine, the uniqueness of rehabilitation research lies in the need to capture various physiological signals generated by the human body during rehabilitation interventions. The frequency domain characteristics of physiological signals from different sources have a large span. Among them, neural electrical signals belong to the level of thousands to tens of thousands of hertz, kinematic and kinetic signals belong to the level of dozens to hundreds of hertz, and cerebral blood oxygen signals belong to the level of a few to dozens of hertz. Therefore, the sampling rates of rehabilitation research equipment vary widely. This makes it very difficult to synchronize sampling and perform collaborative data analysis among multiple devices and systems. The bottleneck lies in that most devices do not have a clock signal and need to synchronize according to timestamps after transmitting signals through network protocols. However, the current data transmission methods based on network protocols have the characteristics of burstiness and non-uniformity, resulting in the loss of true timestamps of the data. The current conventional method is to default that the data is not lost and is uniformly sampled, and after all the data is collected, it is rearranged according to the sampling frequency based on the timestamp of the first signal arrival. This technical solution does not consider data loss caused by factors such as bandwidth, and the post-processing method cannot realize real-time data calling and display.

[0004] Therefore, it is necessary to design a method capable of estimating the timestamps of a dual-temporal non-uniform sampling data stream to achieve the recovery of true timestamps and real-time data calling.

[0005] For multi-device collaborative analysis, it is also necessary to share trigger signals among devices. Conventional inter-device tagging software, such as E-PRIME for psychological research, often sends soft trigger signals to each device through the input of external devices such as keyboards and mice. This method does not require reading device data and thus cannot monitor the status of each device. In rehabilitation research, synchronization is often required according to the human body movement stage. To achieve precise inter-device tagging, it is necessary to read limb kinematic data and design an automatic inter-device synchronization tagging system based on the motion state to achieve synchronous acquisition of multi-modal and multi-device data at a specific human body movement stage. Summary of the Invention

[0006] In view of the fact that for devices synchronized through network protocols in the prior art, the data transmission has the characteristics of bursts and non-uniformity, resulting in problems such as out-of-order timestamps when the consumer end reads the production end data in real time, the present invention proposes a timestamp estimation method for dual-temporal non-uniformly sampled data streams, which can: 1. Solve the problems of data loss and out-of-order timestamps caused by data transmission based on network protocols.

[0007] At the same time, in view of the fact that existing software for sharing trigger signals between devices, such as E-Prime, marks the production end through a soft trigger method and cannot read the production end signals, so it cannot meet the requirements of real-time signal reading and device status monitoring, nor does it have the ability to monitor whether the status of each device is normal. The present invention provides a marking system, which can solve the problem of inability to synchronize and segment data according to the human motion stage among multiple devices, and also solve the problem of inability to monitor the status of each device during multi-device synchronous observation.

[0008] The present invention provides a timestamp estimation method for dual-temporal non-uniformly sampled data streams, including the following steps:

[0009] Step 1: Start the UI thread and the timer thread;

[0010] Step 2: Segment the motion links according to the state machine, and send soft trigger signals to each device based on the state machine to ensure that all devices perform synchronous acquisition at a specific motion stage;

[0011] Step 3: Set the experimental process in a graphical way;

[0012] Step 4: Use the Vertx bus to build a data channel and transmit the experimental data asynchronously;

[0013] Step 5: Reconstruct the timestamps of the experimental data collected in Step 4, and uniformly mark the experimental data after reconstructing the timestamps.

[0014] Preferably, Step 5 specifically includes:

[0015] Step 51: After collecting the experimental data, first sort the collected data of each device according to the reception time;

[0016] Step 52: For the experimental data with missing timestamps, use the equal-spacing estimation method or the non-equal-spacing estimation method to complete the filling;

[0017] Step 53: Use the Bayesian estimation method to further adjust the timestamps to reduce the time error caused by network delay;

[0018] Step 54: Finally, uniformly mark the reconstructed data to ensure that the data of different devices can be analyzed on the same time axis.

[0019] Further preferably, the equal-spacing estimation method in step 52 is as follows: For every two adjacent data points, assuming that the sampling is uniform, use the time difference between adjacent timestamps for equal-spacing interpolation to reconstruct the missing timestamps in the middle.

[0020] Further preferably, the non-equal-spacing estimation method in step 52 is as follows: According to the trusted tense, perform uniform estimation at intervals of the theoretical sampling frequency, and respectively select the trusted generation time or the trusted reception time to reconstruct the timestamps.

[0021] Further preferably, the Bayesian estimation method in step 53 is as follows: Extract all the data of adjacent timestamps, calculate the actual sampling rate during the data transmission process through the Bayesian formula, and reconstruct the timestamps based on this to minimize the impact of loss or delay on the time series.

[0022] Further preferably, when the actual number of sampled points is less than or equal to the standard number of sampled points, at this time, the trusted data generation time or the data arrival time can be trusted, and the equal-spacing estimation method, the non-equal-spacing estimation method or the Bayesian estimation method is used to reconstruct the timestamps according to the data generation time or the data arrival time.

[0023] Further preferably, when the actual number of sampled points is more than the standard number of sampled points, at this time, only the data arrival time can be trusted, and the equal-spacing estimation method, the non-equal-spacing estimation method or the Bayesian estimation method is used to reconstruct the timestamps according to the data arrival time.

[0024] The present invention also provides a marking system applying the above method, including

[0025] Data acquisition device: Each data acquisition device in the system is responsible for acquiring specific types of experimental data;

[0026] Device interface module: The data acquisition device is connected to the device interface module through a network port, USB or local area network;

[0027] Terminal: The terminal is connected to the device interface module through a network port, USB or local area network; The terminal sends instructions to the data acquisition device through the device interface module for acquiring experimental data.

[0028] Preferably, the device interface module includes: a data acquisition interface, a data packing interface and a data transmission interface; The data acquisition interface issues a data acquisition instruction to the data acquisition device through the hardware acquisition instruction provided by the terminal; The data packing interface is provided with a timer, which acquires data once every fixed time and packs the acquired data, the device name of the data source and the physical time of the acquired data into a data packet; The data transmission interface is provided with a fixed-length circular queue, and the packed data packets are stored in the queue in sequence. A timer is used to periodically take out the unsent data packets from the queue and transmit them to the terminal in order.

[0029] Preferably, the terminal includes a data rectification module and a status feedback module; the data rectification module is used to perform the reconstruction of timestamps; the status feedback module real-time feedbacks the operation status of each device through the color of the LED light strip.

[0030] Compared with the prior art, the present application has achieved the following technical effects:

[0031] 1. Solve the problems of data loss and out-of-order timestamps caused by data transmission based on network protocols.

[0032] 2. Solve the problem that data cannot be synchronized and segmented according to the human motion stage among multiple devices.

[0033] 3. Solve the problem that the status of each device cannot be monitored during multi-device synchronous observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a bi-temporal schematic diagram of the data generation time and the data arrival time;

[0035] Figure 2 It is a schematic diagram of timestamp repetition caused by the bursty characteristics of the network protocol;

[0036] Figure 3 It is a schematic diagram of the equal-spacing estimation method;

[0037] Figure 4 It is a schematic diagram of the non-equal-spacing estimation method (trust arrival time);

[0038] Figure 5 It is a schematic diagram of the non-equal-spacing estimation method (trust generation time);

[0039] Figure 6 It is a control flow chart of the state machine;

[0040] Figure 7 It is a schematic diagram of graphically editing the state machine based on kinematic data;

[0041] Figure 8 It is an expected state determination flow chart. DETAILED DESCRIPTION OF THE INVENTION

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0043] Embodiment 1

[0044] During network transmission, there are two important temporal states: the time point when data is generated at the source (such as sensors, devices) and the time when the data is received by the receiving end (such as servers, controllers), which are the data generation time and the data arrival time respectively, that is, bi-temporal, as Figure 1as shown

[0045] However, in the actual process, due to the bursty nature of data transmission in the network protocol, the data actually received at the receiving end is non-real-time, and the problem of multiple data points being assigned the same timestamp will occur. For example Figure 1 as shown, at the time point of t 0 and the time point of t 1 there are multiple data.

[0046] The conventional approach is to default that the data generation time is consistent with the receiving end and no data is lost. After all the data is completely collected, according to the timestamp of the first data point, it is rearranged according to the sampling frequency. This technical solution does not consider data loss caused by factors such as bandwidth, and the post-processing method cannot achieve real-time data calling and display.

[0047] This embodiment provides a timestamp estimation method for a dual-temporal non-uniform sampling data stream, which transforms a non-real-time system based on network protocol data transmission into a real-time system. In this embodiment, three timestamp estimation methods are provided in total: equal-spacing estimation method, non-equal-spacing estimation method, and Bayesian estimation method.

[0048] When the scientific research equipment leaves the factory, the standard theoretical frequency will be marked. In actual scientific research activities, due to various reasons, the actual number of sampling points may be less than, equal to, or more than the theoretical number of sampling points.

[0049] Case 1: The actual number of sampling points is less than or equal to the standard number of sampling points

[0050] Method 1: Equal-spacing estimation method. The basic assumption of the equal-spacing estimation method is that regardless of whether data points are lost, the data collected within every two adjacent timestamps is evenly distributed. The timestamp is reconstructed through the following formula:

[0051]

[0052] For example Figure 3 as shown, the equal-spacing uniform estimation assumes that the generation time of these data points is actually evenly generated between t 0 and t 1 . m is the total number of backlogged data points at time t 1 . When m = n, the arrival time of the last data point is defaulted to t 1 .

[0053] Method 2: Non-equal-spacing estimation method

[0054] The basic assumption of the non-equal-spacing estimation method is to perform uniform estimation according to the trusted temporal state at the interval of the theoretical sampling frequency.

[0055] 1) Trusted arrival time

[0056] Reconstruct the timestamp through the following formula:

[0057]

[0058] where m is the total number of data points backlogged at time t 1 and F is the theoretical sampling frequency of the device. As Figure 4 shown, due to choosing to trust the arrival time, the non-uniform estimation method assumes that the data collected at time t 1 is uniformly distributed at intervals of 1 / F and arranged in reverse order. When data is missing, the data can be filled in as needed according to requirements.

[0059] 2) Trust the arrival time

[0060] Reconstruct the timestamp through the following formula:

[0061]

[0062] where m is the total number of data points backlogged at time t 1 and F is the theoretical sampling frequency of the device. As Figure 5 shown, due to choosing to trust the generation time, the non-uniform estimation method assumes that the data collected at time t 0 is uniformly distributed at intervals of 1 / F and arranged in forward order. When data is missing, the data can be filled in as needed according to requirements.

[0063] Method 3: Bayesian estimation

[0064] Find the actual frequency: Extract all the data of two adjacent timestamps and calculate the actual frequency within this segment according to the following formula. The specific formula is:

[0065]

[0066] where t 0 is the first repeated timestamp, t 1 is the next repeated timestamp adjacent to it, and n 1 is the total number of data points with timestamp t 1 . In this way, a time series of the actual frequency f is obtained, and the sampling rate within the upcoming estimation interval is calculated according to the Bayesian formula, and then non-uniform uniform estimation is performed.

[0067] Case 2: The actual number of sampled points is more than the standard number of sampled points

[0068] In this case, there is a network congestion phenomenon. The number of time sampled points in the previous packet is often less than the theoretical number of sampled points, and only the arrival time can be trusted.

[0069] Method 1: Uniform interval estimation method, which is the same as the processing method in Case 1.

[0070] Method 2: Trust the time of arrival and adopt a non-uniform spacing estimation method;

[0071] 1) The previous estimate did not interpolate and complement the data

[0072] According to the time of arrival, arrange from back to front and calculate the time stamp normally.

[0073] 2) The previous estimate complemented the data according to the difference

[0074] According to the time of arrival, arrange from back to front, and all data with a time stamp earlier than t 0 will be deleted.

[0075] Based on the above, the time stamp estimation method for the dual-temporal non-uniformly sampled data stream provided in this embodiment includes the following steps:

[0076] Step 1: Program thread design; The program is divided into a main thread, a UI thread, and a timer thread. The main thread is a standard Thread program used to start and maintain the states of the UI thread and the timer thread. The UI thread is based on JavaFX and is used to provide a convenient and smooth interface. The timer thread is implemented using Timer and is used for the uniform output of experimental data.

[0077] Step 2: State machine control and data acquisition synchronization; Cut the motion links according to the state machine, and send soft trigger signals to each device based on the state machine to ensure synchronous acquisition of all devices in a specific motion stage. The specific settings are as Figure 6 shown in the state machine to control the process.

[0078] Step 3: Edit the state machine based on kinematic data using a graphical interface; As Figure 7 shown, the experimental process control is completed through the Xross tools set, and the experimental process is graphically designed by dragging and dropping icons such as paths, judgments, loops, and branches.

[0079] Step 4: Read and share data; Use the Vertx bus to build a data channel and perform data transmission asynchronously. Realize data sharing and data output between threads.

[0080] Step 5: Detection during abnormal operation. The expected state of the device is set in each running state and compared in real time: For all devices including the robot, 9 expected state detections are set: NONE,

[0081] UNCHECKED, CHECKING, CHECKED, CONNECTING, UNCONNECTED, CONNECTED, INITING, SERVING. The determination process is as Figure 8 shown.

[0082] Step 6: Timestamp Reconstruction and Processing; specifically:

[0083] Bitemporal Timestamp Recording: Each collected data point contains two timestamps, namely the data generation time and the data arrival time.

[0084] 1) After the collection is completed, first sort the collected data of each device according to the reception time.

[0085] 2) For missing timestamps, use the method of equal-spacing or non-equal-spacing estimation to complete them. Among them, the equal-spacing estimation method is for every two adjacent data points. Assuming that the sampling is uniform, use the time difference between adjacent timestamps for equal-spacing interpolation to reconstruct the missing timestamps in the middle; the non-equal-spacing estimation method is to make a uniform estimation according to the interval of the theoretical sampling frequency according to the trusted tense, and select the trusted generation time or the trusted reception time respectively for timestamp reconstruction.

[0086] 3) Use Bayesian estimation to further adjust the timestamps to reduce the time error caused by network latency. Extract all the data of adjacent timestamps, calculate the actual sampling rate during data transmission through the Bayesian formula, and reconstruct the timestamps based on this to minimize the impact of loss or delay on the time series.

[0087] 4) Finally, uniformly label the reconstructed data to ensure that the data of different devices can be analyzed on the same time axis.

[0088] Embodiment 2

[0089] This embodiment provides a multi-device automatic marking and synchronization system based on a state machine

[0090] 1. Design of the state machine: Use kinematic data to define different motion stages (such as starting, maintaining, stopping, etc.), and define each stage as a state of the state machine. Use the state machine to control synchronous marking to ensure that all devices collect data in the same motion stage. Use a graphical tool to edit the experimental process, and the state machine controls the synchronous collection and state change notification of the devices.

[0091] 2. Soft trigger signal: Read device data through methods such as network cable and local area network, synchronously record the state of the state machine and the data of each device, or based on the state machine, send a soft trigger signal when the state changes to mark the start and end of the motion stage, and synchronously segment the signals of different motion stages according to the state machine information.

[0092] 3. Real-time data processing: Introduce a real-time data processing module to realize real-time segmentation and display of data collection through multi-threading.

[0093] Embodiment 3

[0094] This embodiment provides a multi-device status monitoring system

[0095] 1. The system adopts a heartbeat detection mechanism: a timer thread (using the Java Timer class) is used to periodically send Udp / tcp network status detection requests to the device. Receiving a return packet at a fixed frequency is regarded as a success flag to confirm whether the device working status is normal and whether the response time times out. For Dll status detection, the status values of specified objects are compared to achieve status detection.

[0096] 2. Through two methods, namely, automatic detection within the software and manual inspection by capturing real-time data, the online status of the device at startup is ensured; during abnormal operation detection, the expected status of the device is set in each running state and compared in real time; for Com serial port status detection, a new object is created to synchronize status information and the status value is updated at a fixed frequency.

[0097] Details not described in this invention are all well-known technologies to those skilled in the art.

[0098] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art shall fall within the protection scope determined by the claims.

[0099] Embodiment 4

[0100] The marking system in this embodiment includes device modules, specifically as follows:

[0101] 1) Scientific research equipment interface module:

[0102] The hardware part of this module is a circuit board including a network port, a USB port, and Wi-Fi. External scientific research equipment can access this device through the network port, USB, or local area network. The software part of this module performs functions such as connecting to, calibrating, initializing, and monitoring the status of scientific research equipment through standardized network interfaces.

[0103] 2) Data packaging and transmission module:

[0104] Data acquisition interfaces, data packaging interfaces, and data transmission interfaces are set in the software interface of the data production module

[0105] Port. A data acquisition interface acquires data from scientific research equipment through hardware acquisition instructions provided by a computer; a data packaging interface sets a timer to acquire data once every fixed time, and packs the acquired data, the device name of the data source, and the physical time of the acquired data into a data packet; a data transmission interface sets a fixed-length circular queue, stores the packed data packets into the queue in sequence. A timer is used to periodically retrieve the unsent data packets from the queue and transmit them to the remote end in sequence.

[0106] 3) Data rectification module:

[0107] According to the evaluation method of this application, timestamp recovery is performed. The non-uniform interval reconstruction is used in this patent. Other alternative implementation methods also include uniform interval reconstruction (suitable for short delay or small amount of loss) and Bayesian estimation (speculating the most likely sampling rate based on historical data, suitable for frequently lost or delayed data).

[0108] 4) Status feedback module:

[0109] The operating status of each device is real-time feedback through the color of the LED light strip.

Claims

1. A timestamp estimation method for a bi-temporal non-uniformly sampled data stream, characterized in that: The steps include: Step 1: Start the UI thread and timer thread; Step 2: Divide the motion links according to the state machine, and send soft trigger signals to each device based on the state machine to ensure that all devices perform synchronous acquisition at a specific motion stage; Step 3: Set the experimental process graphically; Step 4: Use Vertx bus to build a data channel and transmit experimental data asynchronously; Step 5: Reconstruct the timestamps of the experimental data collected in step 4, and uniformly mark the experimental data after the reconstructed timestamps.

2. The method for estimating timestamps of a bi-temporal non-uniformly sampled data stream according to claim 1, characterized in that: The step 5 specifically includes: Step 51: After collecting the experimental data, first sort the collected data of each device according to the receiving time; Step 52: For experimental data with missing timestamps, use equal-interval estimation method or non-equal-interval estimation method to complete them; Step 53: Use the Bayesian estimation method to further adjust the timestamp to reduce the time error caused by network delay; Step 54: Finally, the reconstructed data is uniformly labeled to ensure that data from different devices can be analyzed on the same timeline.

3. The method for estimating timestamps of a bi-temporal non-uniformly sampled data stream according to claim 2, characterized in that: The equidistant estimation method in step 52 is as follows: for every two adjacent data points, assuming that sampling is performed uniformly, the time difference between adjacent timestamps is used to perform equidistant interpolation to reconstruct the timestamps lost in the middle.

4. The method for estimating timestamps of a bi-temporal non-uniformly sampled data stream according to claim 2, characterized in that: The non-uniform interval estimation method in step 52 is: according to the trusted temporal state, uniform estimation is performed at intervals of the theoretical sampling frequency, and the trusted generation time or the trusted reception time is selected to reconstruct the timestamp.

5. The method for estimating timestamps of a bi-temporal non-uniformly sampled data stream according to claim 2, characterized in that: The Bayesian estimation method in step 53 is: extract all data of adjacent time stamps, calculate the actual sampling rate in the data transmission process by the Bayesian formula, and reconstruct the time stamp based on this to minimize the impact of loss or delay on the time series.

6. The method for estimating timestamps of a bi-temporal non-uniformly sampled data stream according to claim 2, characterized in that: When the actual number of sampling points is less than or equal to the standard number of sampling points, the data generation time or the data arrival time can be trusted, and the timestamp can be reconstructed using the equal-interval estimation method, the non-equal-interval estimation method, or the Bayesian estimation method based on the data generation time or the data arrival time.

7. The method for estimating timestamps of a bi-temporal non-uniformly sampled data stream according to claim 2, characterized in that: When the actual number of sampling points is greater than the standard number of sampling points, only the data arrival time can be trusted, and the timestamp is reconstructed using the equal-interval estimation method, non-equal-interval estimation method, or Bayesian estimation method based on the data arrival time.

8. A marking system using the method according to any one of claims 1 to 7, characterized in that: include, Data acquisition device: Each data acquisition device in the system is responsible for collecting a specific type of experimental data; Device interface module: The data acquisition device is connected to the device interface module via the network port, USB or LAN; Terminal: The terminal is connected to the device interface module through the network port, USB or local area network; the terminal sends instructions to the data acquisition device through the device interface module to collect experimental data.

9. The marking system according to claim 8, characterized in that: The device interface module includes: a data acquisition interface, a data packaging interface and a data transmission interface; The data acquisition interface sends data acquisition instructions to the data acquisition device through the hardware acquisition instructions provided by the terminal; The data packaging interface is provided with a timer, which collects data once at a fixed time, and packages the collected data, the device name of the data source and the physical time of the data collection into a data packet; The data transmission interface is provided with a fixed-length circular queue, and the packaged data packets are stored in the queue in order. A timer is used to regularly take out the unsent data packets from the queue and transmit them to the terminal in order.

10. The marking system according to claim 8, characterized in that: The terminal includes a data rectification module and a state feedback module; The data rectification module is used to perform timestamp reconstruction; The status feedback module provides real-time feedback of the operating status of each device through the color of the LED light strip.

Images