Method and device for high-precision timestamp stamping and synchronization in a low-power sensor system

Abstract

The present invention describes a method for high-precision timestamping and synchronization in a low-power sensor system. Among them, the method is executed by a device and includes: receiving, by a sensor hub of the device, an interrupt signal from a sensor and performing interrupt processing to obtain an interrupt timestamp obtained through a latch, wherein the interrupt timestamp is obtained from a continuously running unified time reference; obtaining, by the sensor hub, sensor data from the sensor; predicting, by the sensor hub, a predicted timestamp based on the amount of sensor data and the interrupt timestamp by using a filtering algorithm; and correcting, by the sensor hub, the timestamp of each sensor data based on the predicted timestamp.

Description

Technical Field

[0001] The present invention generally relates to stamping and synchronization technology, and in particular, to a high-precision time stamping and synchronization method and device in a low-power sensor system. Background Art

[0002] The background description provided herein is for the purpose of presenting the content of the present invention in general. Within the scope of the work described in this background section, the work of the inventors currently named and the aspects of the specification that could not be characterized as prior art at the time of filing are not explicitly or implicitly admitted as prior art with respect to the present invention.

[0003] Currently, mobile devices have increasingly higher requirements for sensor data, especially hot applications such as augmented reality (AR), virtual reality (VR), and cameras, which have put forward higher requirements for accelerometer and gyroscope data. The above high requirements may include but are not limited to:

[0004] 1. Higher data frequency;

[0005] 2. Smaller fluctuations in data timestamp differences;

[0006] 3. The matching degree between the stamping time of data and the time point of real data generation;

[0007] 4. The data timestamp is monotonically increasing, etc.

[0008] In the present invention, the sensor hub (also referred to as a sensor coprocessor) connects various sensor devices and processes data from various sensor devices with low power consumption. Typically, the sensor hub processes data from sensors such as accelerometers and gyroscopes, and performs fusion processing of the data from each sensor. After the sensor hub receives data from the sensor device, the sensor hub calibrates the timestamp of the received data according to the sensor hub time system. This process of calibrating timestamps is called poking. After the application processor receives data from the sensor hub, it calibrates the timestamp of the received data with the application processor time system. This calibration process is called synchronization. During the above-mentioned poking and synchronization process, when the data timestamp is not processed properly, the actual generation time of the data and the timestamp will not match, which will result in a very poor user interaction experience of the application.

[0009] The extremely poor interactive experience is mainly manifested in:

[0010] 1. Applications such as AR and VR clearly perceive poor tracking and high hysteresis;

[0011] 2. Camera and other applications can clearly sense increased jitter and difficulty focusing.

[0012] Current timestamping method used by the current low-power sensor hub:

[0013] 1. Ordinary sensors: The sensor generates an interrupt signal (DRI) to notify the low-power sensor hub. When the sensor hub receives the interrupt signal, the sensor hub records the current time, obtains data through the communication interface, and stamps the recorded time on each piece of data.

[0014] 2. High-end sensors: When each piece of data is generated, a timestamp is bound through its time reference. The sensor hub and the sensor regularly synchronize to obtain the time deviation. Data and timestamps are obtained through the communication interface, and each piece of data timestamp is corrected again using the time deviation.

[0015] Currently, the time synchronization method between the low-power sensor hub and the application processor is as follows: The application processor accesses its time reference, the low-power sensor hub accesses its time reference, and the application processor and the sensor hub regularly synchronize to obtain the time deviation in the non-standby mode. The application processor corrects the sensor timestamp using the obtained time deviation.

[0016] The process of recording timestamps in the timestamping scheme of ordinary sensors is affected by interrupt jitter, latency, and the system response of the sensor hub, resulting in loss of accuracy. Although high-end sensors can retain a certain degree of timestamp accuracy to some extent in the said timestamping scheme, the selection of sensors will increase costs. Regular synchronization between the sensor hub and the sensor is affected by transmission jitter and latency of the communication interface, resulting in loss of accuracy, and at the same time, it will also increase the power consumption of the entire system. The regular synchronization of the two time references between the processors is affected by transmission jitter and latency of the communication interface, resulting in loss of accuracy, and at the same time, it will also increase the power consumption of the entire system.

[0017] Therefore, there is an urgent need for a high-precision timestamping and synchronization method and device in a low-power sensor system to solve the above problems. Summary of the Invention

[0018] The following summary of the invention is illustrative only and is not intended to be limiting in any way. That is, the following summary of the invention is provided to introduce the concepts, key points, benefits, and advantageous effects of the novel and non-obvious technologies described herein. The selected embodiments are further described in the following detailed description. Therefore, the following summary of the invention is not intended to identify the basic features of the claimed subject matter nor to be used to determine the scope of the claimed subject matter.

[0019] Therefore, the main object of the present invention is to provide a high-precision timestamping and synchronization method and its device in a low-power sensor system to overcome the above disadvantages.

[0020] In an exemplary embodiment, the method for high-precision timestamp stamping and synchronization in a low-power sensor system is performed by a device and includes: receiving, by a sensor hub of the device, an interrupt signal from a sensor and performing interrupt processing to obtain an interrupt timestamp obtained through a latch, wherein the interrupt timestamp is obtained from a continuously running unified time reference; obtaining, by the sensor hub, sensor data from the sensor; predicting, by the sensor hub, a predicted timestamp based on the amount of sensor data and the interrupt timestamp by using a filtering algorithm; and correcting, by the sensor hub, the timestamp of each sensor data based on the predicted timestamp.

[0021] In an embodiment, when the signal level output from a sensor interrupt pin changes, the interrupt timestamp is obtained from the continuously running unified time reference, wherein the sensor interrupt pin is located between the sensor and the sensor hub.

[0022] In an embodiment, when an interrupt controller of the sensor hub initiates an interrupt, the interrupt timestamp is obtained from the continuously running unified time reference.

[0023] In an embodiment, the filtering algorithm is a Kalman filter.

[0024] In an embodiment, the step of predicting the predicted timestamp based on the amount of sensor data and the interrupt timestamp by using the filtering algorithm further includes: selecting filtering parameters according to the amount of sensor data and the interrupt timestamp; calculating a confidence interval based on the filtering parameters; obtaining a Kalman filtering matrix according to the confidence interval, the interrupt timestamp, and an event trigger type; and predicting the predicted timestamp according to the Kalman filtering matrix.

[0025] In an embodiment, the step of correcting the timestamp of each sensor data based on the predicted timestamp includes: determining whether the predicted timestamp is before the interrupt timestamp and whether a first predicted timestamp is after the timestamp; performing Kalman filter update to update an average interval between the predicted timestamp and the sensor data; and correcting the timestamp of each sensor data based on the predicted timestamp and the average interval.

[0026] In an embodiment, when an in-band interrupt signal output from an improved inter-integrated circuit controller conforms to a setting in an improved inter-integrated circuit protocol, the interrupt timestamp is obtained from the continuously running unified time reference.

[0027] In an embodiment, the system time of an application processor, the system time of a sensor hub, and the system time of a latch are derived from the time of the continuously running unified time reference.

[0028] In an embodiment, when the device does not have the sensor hub, the application processor executes the steps executed by the sensor hub.

[0029] In an embodiment, the method further includes: obtaining, by the sensor hub, the system time of the sensor hub and obtaining a first time from the unified time reference that runs throughout; obtaining, by the application processor of the device, the system time of the application processor according to the interrupt signal and obtaining a second time from the unified time reference that runs throughout; calculating, by the application processor, a time deviation between the system time of the sensor hub and the system time of the application processor according to the system time of the sensor hub and the system time of the application processor by using a filtering algorithm; and re-calibrating, by the application processor, the timestamp of each data sent from the sensor hub by using the time deviation.

[0030] In an embodiment, the method further includes: obtaining, by the sensor hub, the system time of the sensor through a controller; when the controller discovers a time synchronization protocol, obtaining, by the latch, a first time from the time of the unified time reference that runs throughout; obtaining, by the sensor hub, a time offset by using a filtering algorithm according to the system time and the first time transmitted by the latch; and writing, by the sensor hub through the controller, the time offset into the sensor for time synchronization between the sensor hub and the sensor; wherein the system time is based on a time reference different from the unified time reference that runs throughout.

[0031] In an embodiment, the sensor is a high-end sensor, and the controller is a communication interface of an internal integrated circuit / serial peripheral interface.

[0032] In an embodiment, a device for high-precision timestamp stamping and synchronization in a low-power sensor system is provided, including: a sensor hub, coupled to a latch and a sensor, and the sensor hub executes: receiving an interrupt signal from the sensor and performing interrupt processing to obtain an interrupt timestamp obtained through the latch, wherein the interrupt timestamp is obtained from the unified time reference that runs throughout; obtaining sensor data from the sensor; predicting a predicted timestamp based on the amount of sensor data and the interrupt timestamp by using a filtering algorithm; and calibrating the timestamp of each sensor data based on the predicted timestamp. Description of the Drawings

[0033] The accompanying drawings included are used to provide a further understanding of the invention and are incorporated into and form a part of this invention. The drawings illustrate embodiments of the invention and, together with the description, are used to explain the principles of the invention. It can be understood that, for the purpose of clearly illustrating the concepts of the invention, the drawings are not necessarily drawn to scale, and some of the components shown may be shown in a scale larger than their actual size in the actual embodiment.

[0034] Figure 1 Shows a schematic diagram of a mobile device in the prior art.

[0035] Figure 2 Shows a schematic diagram of a mobile device according to an exemplary embodiment of the present invention.

[0036] Figure 3 Shows schematic diagrams of different time bases used in a mobile device in the prior art.

[0037] Figure 4 Shows a schematic diagram of a continuously running unified time base used in a mobile device according to an exemplary embodiment of the present invention.

[0038] Figure 5 Shows a schematic diagram of the time deviation of an interrupt signal DRI in the prior art.

[0039] Figure 6A Shows a schematic diagram of a latch according to an exemplary embodiment of the present invention.

[0040] Figure 6B Shows a schematic diagram of the triggering of a latch according to an exemplary embodiment of the present invention.

[0041] Figure 7 Shows a schematic diagram of a method for estimating timestamps in a sensor hub in the prior art.

[0042] Figure 8 Shows a flowchart of a method for high-precision timestamp stamping and synchronization in a low-power sensor system according to an exemplary embodiment of the present invention.

[0043] Figure 9 Shows a flowchart of a predicted timestamp process according to an exemplary embodiment of the present invention.

[0044] Figure 10 Shows a flowchart of an updated timestamp process according to an exemplary embodiment of the present invention.

[0045] Figure 11 Shows a schematic diagram of the inter-processor synchronization deviation in the prior art.

[0046] Figure 12A Shows a schematic diagram of a time synchronization method between a sensor hub and an application processor according to an exemplary embodiment of the present invention.

[0047] Figure 12B Shows a flowchart of a time synchronization method according to an exemplary embodiment of the present invention.

[0048] Figure 13 Shows a schematic diagram of the triggering of a latch according to another exemplary embodiment of the present invention.

[0049] Figure 14A A schematic diagram of a mobile device showing an exemplary embodiment of the present invention is presented.

[0050] Figure 14B A flowchart showing the synchronization between a high - end sensor and a sensor hub in an exemplary embodiment of the present invention is presented.

[0051] Figure 15 A schematic diagram of a mobile device showing an exemplary embodiment of the present invention is presented.

[0052] Figure 16 A schematic diagram of a mobile device showing another exemplary embodiment of the present invention is presented. Detailed Description of the Invention

[0053] In the specification and claims, certain terms are used to refer to specific elements. Those skilled in the art should understand that hardware manufacturers may use different terms to refer to the same element. The specification and claims do not distinguish elements by the difference in name, but by the difference in function. The term "comprising" mentioned throughout the specification and claims is an open - ended term and should be interpreted as "including but not limited to". In addition, the term "coupled" herein includes any direct and indirect means of electrical connection. Therefore, if it is described in the text that a first device is coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means.

[0054] The following description is of the best - mode embodiments for implementing the present invention, which is for the purpose of describing the principles of the present invention and not a limitation thereof. It can be understood that the embodiments of the present invention can be implemented by software, hardware, firmware, or any combination thereof.

[0055] Figure 1 A schematic diagram of a mobile device 100 in the prior art is presented. As Figure 1 shown, there are two processing methods for stamping between the sensor hub 102 and the sensor 103, namely: processing method 1.a and processing method 1.b. Processing method 1.a is the stamping method of the sensor hub 102 for ordinary sensors, and the sensor 103 generates an interrupt signal (DRI) to notify the sensor hub 102. After receiving the interrupt signal, the sensor hub 102 records the current time, through the communication interface ( Figure 1obtain data (not shown in the figure), and timestamp each piece of data. In processing mode 1.a, the process of recording timestamps is affected by interrupt jitter, latency, and the response of the sensor hub 102, resulting in loss of accuracy. In processing mode 1.b, each piece of data generated by the sensor 103 is timestamped through its time reference 3. The sensor hub 102 and the sensor 103 are periodically synchronized to obtain the time deviation 2. The sensor hub 102 obtains data and timestamps from the sensor 103 through the communication interface, and uses the time deviation 2 to recalibrate the timestamps of each piece of data. Although processing mode 1.b can retain a certain timestamp accuracy to a certain extent, the selection of sensors will increase costs. Similar to processing mode 1.a, the periodic synchronization between the sensor hub and the sensor is affected by transmission jitter and latency of the communication interface, resulting in loss of accuracy and also increasing the power consumption of the entire system.

[0056] In the prior art solution, the synchronization between the sensor hub 102 and the application processor 101 is processing mode 2. The application processor 101 accesses the time reference 1, and the sensor hub 102 accesses the time reference 2. In the non - standby mode, the application processor 101 and the sensor hub 102 are periodically synchronized to obtain the time deviation 1, and the timestamps of the sensor 103 are recalibrated through the obtained time deviation 1. Since the periodic synchronization of the two time references of the application processor 101 and the sensor hub 102 is affected by transmission jitter and latency of the communication interface, it results in loss of accuracy and also increases the power consumption of the entire system.

[0057] Figure 2 A schematic diagram of a mobile device 200 according to an exemplary embodiment of the present invention is shown. The mobile device 200 includes an application processor 201, a sensor hub 202, at least one sensor 203, at least one latch 204, and a unified time reference 220 that always runs.

[0058] The application processor 201 can be embodied as any type of processor capable of performing the functions described herein. For example, the application processor 201 can be embodied as a single - core or multi - core processor, a single - or multi - socket processor, a digital signal processor, a graphics processor, a microcontroller, or other processor or processing / control circuit.

[0059] The sensor hub 202 can be embodied as any device or group of devices capable of performing the described functions. The sensor hub 202 can be embodied as an integrated circuit, an embedded system, components on a printed circuit board, etc. In some embodiments, a part of the sensor hub 202 can be integrated with the application processor 201, the memory ( Figure 2 not shown in the figure) and the I / O subsystem ( Figure 2Some or all of them (not shown in the figure) are embodied as part of a System on Chip (SoC) together, where another part of the sensor hub 202 (such as one or more sensors 203) is separate from the above-mentioned SoC.

[0060] The sensor 203 can include any type of sensor that can be used with the mobile device 200. For example, the sensor 203 can include a microphone, an image sensor, an accelerometer, a gyroscope, a light sensor, and / or similar sensors.

[0061] The always-running unified time base 220 can be embodied as any type of clock to generate a timing value indicating the passage of time. For example, a crystal oscillator or a ceramic oscillator, an inductor-capacitor (LC) clock, a resistor-capacitor (RC) clock, etc.

[0062] In Figure 2 it, the always-running unified time base 220 is applied to the mobile device 200, and both the system time 206 and the system time 207 are generated based on the always-running unified time base 220. The sensor 203 sends an interrupt signal (DRI) to the sensor hub 202, and the latch 204 can monitor this interrupt signal. The sensor hub 202 executes a filtering algorithm 209, and the application processor 201 executes a filtering algorithm 208. In Figure 2 the design, by designing the acquisition of the DRI time of the interrupt signal through the latch 204 and the design of the confidence filtering algorithm for the DRI time to correct the data timestamp, the problem of the loss of timestamp stamping accuracy after the sensor hub 202 acquires data can be solved. Through the design of the always-running unified time base and the design of the debounce filtering algorithm, the synchronization transmission deviation between the application processor 201 and the sensor hub 202 can be eliminated, and the problem of the loss of timestamp synchronization accuracy between the sensor hub 202 and the application processor 201 can be solved. Specifically, when the application processor 201 enters the standby mode, the always-running unified time base 220 keeps running and maintains high precision (13M) and low power consumption. When the sensor 203 completes sampling, it triggers an interrupt signal DRI to notify the latch 204 and the sensor hub 202. The latch 204 monitors the change in the interrupt signal level. When the above-mentioned level changes, the latch 204 obtains the time from the always-running unified time base 220 and stores it. The sensor hub 202 receives the interrupt signal, obtains the time recorded by the latch 204 and the data provided by the sensor 203, and executes a filtering algorithm 209 to add a timestamp to each piece of data. The application processor 201 accepts the interrupt, calculates the time deviation between the sensor hub 202 and the application processor 201 through the filtering algorithm 208, and re-corrects the timestamp of the data transmitted by the sensor hub 203.

[0063] It should be noted that filtering algorithms 208 and 209 are presented as general solutions and are not intended to limit the concepts of the present invention to a specific form (e.g., Kalman filtering, moving window filtering, etc.), but are intended to cover the functions of all modifications, alternatives, and equivalents of algorithms similar to the filtering algorithms in the present invention.

[0064] The illustrative mobile device 200 can be any type of computing device capable of performing the functions described herein. For example, the mobile device 200 can be or include, but is not limited to, a wearable computer, an embedded computing system, a system-on-chip (SoC), a smart phone, a cellular phone, a tablet computer, a laptop computer, a notebook computer, a server computer, a desktop computer, a cell phone, a messaging device, a camera device, a multiprocessor system, a processor-based system, a consumer electronic device, and / or any other computing device.

[0065] In some embodiments, one or more of the illustrative components of the mobile device 200 can be incorporated into another component or otherwise form a part of another component. For example, in some embodiments, the sensor hub 202 or a portion thereof can be incorporated into the application processor 201.

[0066] Those skilled in the art will understand that the element circuits in the mobile device 200 generally include transistors that are configured to control the operation of the circuit according to the functions and operations described herein. It can be further understood that the specific structure or interconnection of the transistors is generally determined by a compiler, such as a register transfer language (RTL) compiler. The RTL compiler can be operated by a processor according to a script that is very similar to an assembly language code to compile the script into a form for the layout or manufacture of the final circuit. In fact, RTL is well-known for its role and use in facilitating the electronic and digital system design process.

[0067] Figure 3 A schematic diagram showing different time bases used in a prior art mobile device 300 is shown. As Figure 3 shown in the prior art, since the sensor hub 302 cannot access time base 1 when the application processor 301 is in the standby mode, the application processor 301 and the sensor hub 302 need to have their own corresponding time bases 1 and 2, respectively. The application processor 301 generates the system time 320 based on time base 1, and the sensor hub 302 generates the system time 321 based on time base 2. Therefore, there can be a large deviation in the synchronization of the system times based on different time bases. Especially when the application processor 301 has been in the standby mode for some time, the deviation between the application processor 301 and the sensor hub 302 may be even greater when the first synchronization is performed.

[0068] Figure 4A schematic diagram of a continuously running unified time reference used in the mobile device 400 of an exemplary embodiment of the present invention is shown. In Figure 4 , the continuously running unified time reference 410 is used in the mobile device 400 and maintains low power consumption and high precision (maintaining 13M precision and low power consumption when the application processor 401 enters the standby mode). The time references of the application processor 401, the sensor hub 402, and the latch 403 all originate from the continuously running unified time reference 410. The first timestamp of the sensor data is based on the system time 421 of the sensor hub 402. After the sensor data is transmitted to the application processor 401, a second synchronization is performed to remap the timestamp based on the system time 421 of the sensor hub 402 into a timestamp based on the system time 420 of the application processor 401. By sharing the time reference 410 among the application processor 401, the sensor hub 402, and the latch 403, the deviation between the system times is only the deviation caused by the inconsistent processing of the time reference by different processors, and the above deviation is very small and can be easily removed by a filtering algorithm.

[0069] Figure 5 A schematic diagram showing the time error of the interrupt signal DRI in the prior art is shown. As Figure 5 shown, when the sensor data sampling is completed, the sensor sends the interrupt signal DRI to the interrupt controller of the sensor hub at time T k '. Then, at time T k , the interrupt controller sends the interrupt signal to the application processor to execute software operations. In this process, the overall error ΔT = T k - T k '. This error is mainly divided into a hardware part and a software part. In the hardware part, the interrupt controller performs a priority judgment after receiving the interrupt signal. If the current is a high priority, the interrupt controller waits. Therefore, the jitter and delay of the interrupt controller cause the first error. In the software part, after the interrupt controller sends the interrupt signal to the application processor, the interrupt controller is affected by the system closing the interrupt and cannot be immediately executed temporarily. Therefore, the system response causes the second error.

[0070] Figure 6A A schematic diagram of the latch 602 of an exemplary embodiment of the present invention is shown. In Figure 6A shown, the latch 602 includes a latch module 603, an interrupt status detection module 604, and an edge detection module 605. The interrupt status detection module 604 receives a signal from the interrupt controller 601 of the sensor hub 600. The edge detection module 605 receives the interrupt signal DRI from the sensor 606. The latch module 603 receives the time of the continuously running unified time reference 607.

[0071] Figure 6B FIG. shows a schematic diagram of a trigger latch according to an exemplary embodiment of the present invention. In Figure 6B , the latch 602 monitors the signal level output by the sensor interrupt pin (DRI), which is located between the sensor and the sensor hub, to notify the sensor hub when the sensor data is ready. When the signal level output from the sensor interrupt pin changes, the latch 602 obtains an interrupt timestamp from the always-running unified timebase 607. Specifically, a falling edge of the DRI level or a rising edge of the DRI level triggers the latch 602 to obtain an interrupt timestamp from the always-running unified timebase 607. As Figure 6B shown, the interrupt controller 601 can terminate or initiate an interrupt. The latch 602 determines whether to obtain and store the time from the always-running unified timebase 607 by referring to the interrupt signal DRI and the signal output by the interrupt controller 601. In Figure 6B the example shown, the triggering mode of the latch 602 does not refer to the interrupt status of the interrupt controller 601 in the sensor hub 600. As long as the edge of the level changes, the interrupt time is obtained from the always-running unified timebase at time 608 and stored by the latch 602.

[0072] Referring to Figure 6A and Figure 6B the exemplary scheme shown, the latch 602 obtains the interrupt time from the always-running unified timebase and stores the interrupt time for subsequent filtering algorithms, thereby eliminating the error ΔT and greatly improving the accuracy of the timestamp.

[0073] Figure 7 FIG. shows a schematic diagram of a timestamp estimation method for a sensor hub in the prior art. As Figure 7 shown, it is assumed that the sensor generates an interrupt every 4 data. The sensor can send an interrupt to the sensor hub at time T k '. When there is no latch, the time recorded by the sensor hub can be T k , resulting in an error ΔT. Since the calculated average interval D between the data is greater than the actual interval, the timestamp of each data deviates from the true time point when the data is generated. Using the timestamp to calculate the calculated average interval D may not be accurate enough to truly track the frequency at which the sensor generates data. The average interval D is represented by the following formula:

[0074] T k-1 = T k -(N - 1)D

[0075] where, T k ' is the time when the sensor currently generates an interrupt, and T k-1is the time when the sensor last generated an interruption, T k is the time recorded by the sensor hub when it received the interruption, and N is the amount of data.

[0076] Figure 8 is a flowchart of a method 800 for timestamping and synchronizing with a high-precision timestamp in a low-power sensor system according to an exemplary embodiment of the present invention, where the method is executed by a device. In Figure 8 exemplary scenarios, the sensor hub uses a trust filter algorithm for the time of the interruption signal DRI stored in the latch and the time of the interruption signal DRI to predict the data frequency and correct the timestamp of the sensor data. The timestamping process executed by the sensor hub is as follows.

[0077] In step S801, the sensor hub of the device receives an interruption signal from the sensor and executes an Interrupt Service Routine (ISR) to obtain an interruption timestamp obtained through the latch, where the interruption timestamp is obtained from a continuously running unified time reference. In another embodiment, after obtaining the interruption timestamp, the sensor hub further converts the interruption timestamp into a timestamp based on the system time of the sensor hub.

[0078] In step S802, the sensor hub obtains sensor data from the sensor through a communication interface.

[0079] In step S803, the sensor hub predicts a predicted timestamp based on the sensor data volume and the interruption timestamp by using a filter algorithm. (It should be noted that the present invention provides a general solution, and the filter algorithm includes various algorithms. Therefore, the present invention takes the Kalman filter algorithm as an example).

[0080] In step S804, the sensor hub corrects the timestamp of each sensor data based on the predicted timestamp.

[0081] Figure 9 is a flowchart of a process 900 for predicting a timestamp according to an exemplary embodiment of the present invention. This flowchart is Figure 8 a more detailed description of step S803 in Figure 9 In, the predicted timestamp is predicted through a filter algorithm. Taking the Kalman filter algorithm as an example, the P, Q, and R matrices are dynamically set and used to predict the predicted timestamp according to the trust of the interruption timestamp.

[0082] In step S901, the sensor hub obtains the data volume N from the sensor through a communication interface. In step S902, the sensor hub obtains the interruption timestamp Z from the latch T , where the interruption timestamp Z Tis the interruption time when the sensor issues an interruption signal. In step S903, the sensor hub detects event trigger type A, where event trigger type A is how the sensor hub reads the sensor. There are two ways for the sensor hub to read the sensor. One is that the sensor actively triggers an interruption to notify the sensor hub, and the other is that the sensor hub actively reads data periodically. These two ways may affect the dynamic settings of the Q matrix and the R matrix of the Kalman filter.

[0083] In step S904, the sensor hub determines whether to select the filtering parameters at the (k - 1)th moment (such as in step S905) or the filtering parameters at the (k - 2)th moment (such as in step S906) based on the first data after the sensor is completed and the interruption timestamp Z T , where T' is the predicted value of the interruption timestamp, and Z T is the observed value of the interruption timestamp.

[0084] In step S907, the sensor hub calculates D' (D' is the average value) and the confidence interval Trust at the kth moment through the filtering parameters, and the confidence percentage is 20%. D' and Trust are expressed by the following formulas:

[0085] D′ = (Z T - T′) / N

[0086] Trust = D × 20%

[0087] In step S908, when the event trigger type is that the sensor hub actively reads data (that is, the timestamp Z T obtained by the latch is not trustworthy), the trust prediction value of the Q matrix of the Kalman filter is reduced (as shown in step S909). In the case where the event trigger type is that the sensor actively triggers an interruption, when the first data meets the 20% confidence interval or Z T is much larger than T' (that is, the timestamp Z T obtained by the latch is reliable), the trust measurement value of the R matrix of the Kalman filter is reduced (as shown in step S910).

[0088] In step S911, when the measured value (predicted value) is trustworthy at the (k - 1)th moment and the sensor hub determines through calculation that the predicted value (measured value) should be more trustworthy at the kth moment, the P matrix is reset so that the Kalman filter algorithm can quickly follow the state change.

[0089] In step S912, the sensor hub predicts the predicted timestamp through the P, Q, and R matrices of the Kalman filter using the Kalman filter state equation, where the Kalman filter state equation is expressed by the following formula:

[0090]

[0091]

[0092] Figure 10 It is a flowchart of process 1000 for updating a timestamp shown according to an exemplary embodiment of the present invention. This flowchart is Figure 8 a more detailed description of step S804. In Figure 10 it, the sensor hub updates the timestamp using a filtering algorithm. In this embodiment, taking the Kalman filtering algorithm as an example, the prediction result is iteratively corrected to correct the timestamp.

[0093] In step S1001, the sensor hub obtains a predicted timestamp through the Kalman filtering algorithm. In step S1002, the sensor hub determines whether the predicted timestamp T is before Z T When the predicted timestamp T is ahead of Z T the sensor hub calculates the lead amount δ of each timestamp, adds an adjustment scale trim to the lead amount δ, and performs Kalman filtering to re-predict the predicted timestamp (as shown in step S1010), where δ is the adjustment amount.

[0094] In step S1003, the sensor hub determines whether the predicted timestamp T1 is after T'. When the predicted timestamp T1 is after T', the sensor hub calculates the lag amount δ of each timestamp, adds an adjustment scale trim to the lag amount δ, and performs Kalman filtering to re-predict the predicted timestamp (as shown in step S1020).

[0095] In step S1011, when the number of re-predictions exceeds 3 times and there are still lead or lag problems (that is, the inertia of the parameters in the P matrix is too large, and the lead amount and lag amount are large, resulting in the algorithm being unable to converge quickly), the P matrix is reset (as in step S1012) to make the algorithm converge quickly.

[0096] In step S1004, the sensor hub performs an update of the Kalman filter to update the P matrix, checks the positive definiteness of the P matrix, and updates T and D.

[0097] In step S1005, the sensor hub uses T and D to re-correct the timestamp of each data.

[0098] Figure 11 shows a schematic diagram of the synchronization deviation between processors in the prior art. As Figure 11 shown, due to the jitter, delay of the interrupt transmission, and the influence of the system response of the application processor, it cannot be guaranteed to obtain T1 and T3 simultaneously, so the calculated time deviation amount may be greater than the true time deviation ΔT'.

[0099] Figure 12AThe figure shows a schematic diagram of a time synchronization method between a sensor hub and an application processor according to an exemplary embodiment of the present invention. In Figure 12A the shown time synchronization method between the sensor hub and the application processor uses a continuously running unified time reference and a de-jitter filtering algorithm 1203 to eliminate synchronization transmission deviation and correct time stamps (steps 1202 and 1210). As Figure 12A shown, the sensor hub 1206 obtains T2 from the continuously running unified time reference 1211 and obtains T1 from the system time 1209. The application processor 1200 obtains T4 from the continuously running unified time reference 1211 and obtains T3 from the system time 1201. Both the system times 1201 and 1209 are based on the continuously running unified time reference 1211.

[0100] Figure 12B is a flowchart of the time synchronization method according to an exemplary embodiment of the present invention. Figure 12B The figure shows the synchronization process between the application processor and the sensor hub.

[0101] In step S1201, the sensor hub obtains the system time T1 and obtains the time T2 from the continuously running unified time reference at the interruption termination. Then, the sensor hub sends the data T1 and T2 to the application processor through an interrupt signal.

[0102] In step S1202, the application processor receives the interrupt signal, obtains the system time T3 through an interrupt handling function, and obtains the time T4 from the continuously running unified time reference. Then, the application processor calculates T3' = T1 + ΔT' (ΔT' is the interrupt time, that is, the time between the system time T1 and the system time T3. Due to the influence of the load on the application processor, the interruption may be greatly delayed. The continuously running unified time reference used in the present invention can eliminate the influence of the interrupt transmission, so the application processor can remap T1 to T3').

[0103] In step S1203, the application processor calculates the time deviation ΔT between the system time of the sensor hub and the system time of the application processor according to T3 and T3' using a filtering algorithm. It should be noted that the filtering algorithm can include various algorithms. In the present invention, the moving window filtering algorithm is taken as an example for illustration. When the interval between the current synchronization time and the previous synchronization time is too large (i.e., the synchronization frequency is too low), the window filter is reset to prevent excessive hysteresis from causing synchronization jitter, so as to quickly follow the latest ΔT. When the interval between the current synchronization time and the previous synchronization time is too small (i.e., the synchronization frequency is too high), the synchronization frequency should be suppressed to prevent overfitting. The application processor averages the time differences (T3' - T3) stored in the window to obtain ΔT.

[0104] In step S1204, the application processor uses ΔT to re - calibrate the timestamp of each data sent from the sensor hub.

[0105] In Figure 12B is shown an example of a triggered latch. Figure 13 Shown is a schematic diagram of a latch being triggered according to another exemplary embodiment of the present invention. In Figure 13 , the latch can determine whether to obtain and store time from a continuously running unified time reference based on the interrupt signal DRI and the signal output by the interrupt controller. When the interrupt controller of the sensor hub terminates the interrupt, the latch does not obtain the interrupt timestamp at the rising (falling) edge of the interrupt signal DRI. When the interrupt controller of the sensor hub initiates an interrupt, at time 1300, the latch is triggered to obtain and store the interrupt timestamp from the continuously running unified time reference.

[0106] Figure 14A Shown is a schematic diagram of a mobile device 1400 according to an exemplary embodiment of the present invention. In Figure 14A , the application processor 1401 includes a system time 1404 and a filtering algorithm 1405. The sensor 1403 is a high - end sensor that integrates a system time 1411, a time deviation 1412, and a data generator 1413. Among them, a time reference 1421 provides time to the system time 1411. The data generator 1413 generates data with time based on the system time 1411 and the time deviation 1412. The sensor hub 1402 synchronizes time with the sensor 1403 through a controller 1410 and updates the time deviation 1412. The controller 1410 can be an internal integrated circuit / serial peripheral interface (I2C / SPI) communication interface. The synchronization between the sensor hub 1402 and the sensor 1403 is also affected by the jitter of the I2C / SPI controller of the communication protocol, the time delay and response of the sensor hub 1402, resulting in an inaccurate deviation between the system time of the sensor 1403 and the system time of the sensor hub 1402 and affecting the accuracy of the timestamp. Therefore, a latch 1409 is added in the mobile device 1400 to monitor the controller 1410. The time of the latch 1409 is provided by a continuously running unified time reference 1420. The latch 1409 can be accessed by the sensor hub 1402 to obtain time. The system time 1406 of the sensor hub 1402 is provided by a continuously running unified time reference 1420. The sensor hub 1402 calculates the time deviation according to the system time 1406, the latch time 1409, and the system time 1411 through a filtering algorithm 1407 and writes the time deviation into the time deviation 1412 through the controller 1410. The data acquisition 1408 of the sensor hub 1402 reads data from the sensor 1403 through the controller 1410. Additionally, the mobile device 1400 also includes a time reference 1421.

[0107] Figure 14B shows a reference Figure 14A flowchart of the synchronization between a high - end sensor and a sensor hub according to an exemplary embodiment of the present invention.

[0108] In step S1401, the sensor hub obtains the system time T2 of the sensor through the controller, where the sensor is a high - end sensor and the controller has an I2C / SPI communication interface.

[0109] In step S1402, when the controller discovers the time synchronization protocol, it triggers the latch to obtain the first time T1 from the time of the always - running unified time reference and records the first time T1, where the system time T2 of the sensor is obtained based on a time reference different from the always - running unified time reference.

[0110] In step S1403, the sensor hub obtains the time offset by using a filtering algorithm according to the system time T2 and the first time T1 transmitted by the latch. Specifically, after obtaining the first time T1 and the system time T2, the sensor hub calculates ΔT (ΔT = T1 - T2) and performs a filtering algorithm with ΔT to obtain the time offset.

[0111] In step S1404, the sensor hub writes the time offset into the sensor through the controller for time synchronization between the sensor hub and the sensor.

[0112] In Figure 14A and 14B In the example scenarios, a latch is added to monitor the controller. When the controller finds the time synchronization protocol and obtains the system time of the sensor, the controller triggers the latch to immediately obtain the time of the always - running unified time reference and record the time. Then, the controller can notify the sensor hub to execute the filtering algorithm. As Figure 14A and 14B shown in the operations can effectively eliminate the jitter delay of the controller, the jitter delay of the interrupt sent from the controller to the sensor hub, and the influence of the response sent by the sensor hub, improve the time synchronization accuracy between the sensor hub and the sensor, and improve the accuracy of the data timestamp in the sensor.

[0113] Figure 15 shows a schematic diagram of a mobile device 1500 according to an exemplary embodiment of the present invention. In Figure 15 , the improved Inter - Integrated Circuit (I3C) controller 1508 is a controller that supports the I3C protocol. With the rapid development of the I3C protocol, many sensors have started to support the I3C protocol, and the I3C protocol provides an In - Band Interrupt (IBI) interrupt, which can replace Figure 2DRI interrupt of the shown sensor. However, the problem is that the latch also needs to be able to monitor the I3C IBI signal. Therefore, in Figure 15 , a latch 1507 is added to monitor the change of the IBI signal from the I3C controller 1508. The time of the latch is provided by the always-running unified time reference 1510, and the latch can be accessed by the sensor hub 1502. When the IBI signal conforms to the setting in the I3C protocol, the latch obtains a timestamp from the always-running unified time reference 1510 and stores the timestamp. The way to trigger the latch does not involve the interrupt status of the interrupt controller of the sensor hub ( Figure 15 , not shown). As long as the signal conforms to the setting in the I3C protocol, the latch is triggered to obtain and store the time from the always-running unified time reference. In the above way, the latch records the time before the IBI is sent to the interrupt controller of the sensor hub, thereby eliminating the deviation ΔT and greatly improving the accuracy of the timestamp.

[0114] Figure 16 FIG. shows a schematic diagram of a mobile device 1600 according to another exemplary embodiment of the present invention. In Figure 16 , the latch 1607 monitors the interrupt signal DRI from the sensor 1603 and monitors the I2C / SPI controller 1609, and the latch 1606 monitors the I3C / IBI controller 1608. In addition, Figure 16 the mobile device 1600 in does not have a sensor hub; instead, the processor 1601 of the mobile device 1600 executes a filtering algorithm instead of the sensor hub. The processor 1601 can be an application processor, and the processor 1601 in Figure 16 can be used to ensure the timestamp accuracy of some time-sensitive devices (such as touchscreens, ISPs). The processor 1601 can be a flight control device, an AR device, a VR device, a wearable device, an IOT device, etc. The mobile device 1600 ensures the accuracy of the sensor timestamp to improve the user experience. The sensor can be shown as the sensor 1602, which communicates with the processor 1601 through the I3C controller 1608, or the sensor can be shown as the sensor 1603, which communicates with the processor 1601 through the I2C / SPI controller 1609. When there are multiple sensors in the mobile device 1600, the sensors can also be a combination of various forms. The key is that the system time 1604 of the processor 1601 and the system time of the latch 1606 are based on the same time reference, that is, the always-running unified time reference 1610, and accurate timestamps are achieved through an effective filtering algorithm 1605, thereby improving the user experience.

[0115] As described above, the method and device for high-precision timestamp stamping and synchronization in a low-power sensor system according to the present invention can effectively utilize a latch to eliminate timestamp jitter caused by interrupt jitter, and effectively obtain a time close to the true generation time of sensor data by using a filtering algorithm. Especially in the case of high load or full load of the sensor hub, the device can also ensure high precision of the timestamp and a high degree of matching with the true generation time through the latch and filtering algorithms. The always-running unified time reference introduced by the present invention can ensure that the sensor hub and the application processor are on the same time reference. Since different processors handle the hardware time reference inconsistently, the de-jitter filtering algorithm is adopted to effectively eliminate the offset between processors, thereby optimizing the user experience.

[0116] The foregoing description is presented to enable a person skilled in the art to implement the present invention according to a specific application and the content required thereof. Various modifications to the embodiments are obvious to a person skilled in the art, and the basic principles defined above can be applied to other embodiments. Therefore, the present invention is not limited to the specific embodiments described, but conforms to the broadest scope consistent with the principles and novel features disclosed. In the above detailed description, various specific details are described in order to provide a thorough understanding of the present invention. However, a person skilled in the art can understand that the present invention is implementable.

[0117] The embodiments of the present invention described above can be implemented in various hardware, software codes, or a combination of both. For example, an embodiment of the present invention can be a circuit integrated into a video compression chip or a process code integrated into video compression software to execute the above process. An embodiment of the present invention can also be a process code that executes the above process in a digital signal processor (DSP). The present invention can also relate to various functions executed by a computer processor, a digital signal processor, a microprocessor, or a field programmable gate array (FPGA). The above processors can be configured according to the present invention to execute specific tasks, which are completed by executing machine-readable software code or firmware code that defines the specific methods disclosed by the present invention. The software code or firmware code can be developed into different process languages and different formats or forms. The software code can also be compiled for different target platforms. However, different code styles, types, and languages of the software code and other types of configuration codes that execute tasks according to the present invention do not depart from the spirit and scope of the present invention.

[0118] The present invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The described examples are considered illustrative of all aspects and are not limiting. Therefore, the scope of the present invention is indicated by the claims, rather than the foregoing description. All changes within the scope and meaning equivalent to the claims are covered by the present invention.

[0119] It should be understood that any specific order or step hierarchy in any disclosed process is an example. It should be understood that, based on design preferences, the specific order or hierarchy of steps in a process can be rearranged while remaining within the scope of the present invention. The appended method claims present the elements of the various steps in an example order and do not imply limitation to the specific order or hierarchy presented.

[0120] Ordinal terms such as "first", "second", "third", etc., that modify elements in the claims do not themselves imply any priority, precedence, or that one element is superior to another or the temporal order in which a method is performed, but are merely labels used to distinguish one element having a specific name from another element having the same name (except for the modifying ordinal term).

[0121] Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the scope of the present invention. Any person skilled in the art, without departing from the spirit and scope of the present invention, makes equivalent changes and modifications, all of which fall within the scope of the present invention.

Claims

1. A method for high-precision timestamp stamping and synchronization in a low-power sensor system, where the method is executed by a device, and the method includes: Receiving, by a sensor hub of the device, an interrupt signal from a sensor and performing interrupt processing to obtain an interrupt timestamp obtained through a latch, where the interrupt timestamp is obtained from a continuously running unified time reference; Obtaining, by the sensor hub, sensor data from the sensor; Predicting, by the sensor hub, a predicted timestamp based on the sensor data volume and the interrupt timestamp by using a filtering algorithm, where the step of predicting the predicted timestamp includes: selecting filtering parameters according to the sensor data volume and the interrupt timestamp; calculating a confidence interval based on the filtering parameters; obtaining a Kalman filter matrix according to the confidence interval, the interrupt timestamp, and an event trigger type; and predicting the predicted timestamp according to the Kalman filter matrix; and Correcting, by the sensor hub, the timestamp of each sensor data based on the predicted timestamp.

2. The high-precision timestamp stamping and synchronization method in the low-power sensor system according to claim 1, characterized in that, When the signal level output from a sensor interrupt pin changes, obtaining the interrupt timestamp from the continuously running unified time reference, where the sensor interrupt pin is located between the sensor and the sensor hub.

3. The high-precision timestamp stamping and synchronization method in the low-power sensor system according to claim 1, wherein When an interrupt is initiated by an interrupt controller of the sensor hub, obtaining the interrupt timestamp from the continuously running unified time reference.

4. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 1, characterized in that, The filtering algorithm is a Kalman filter.

5. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 4, characterized in that, The step of correcting the timestamp of each sensor data based on the predicted timestamp includes: Determining whether the predicted timestamp is before the interrupt timestamp; Performing Kalman filter update to update the average interval between the predicted timestamp and the sensor data; and Correcting the timestamp of each sensor data based on the predicted timestamp and the average interval.

6. The high-precision timestamp stamping and synchronization method in the low-power sensor system according to claim 1, characterized in that, When an in-band interrupt signal output from an improved inter-integrated circuit controller conforms to the settings in an improved inter-integrated circuit protocol, obtaining the interrupt timestamp from the continuously running unified time reference.

7. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 1, wherein The system time of the application processor, the system time of the sensor hub, and the system time of the latch are derived from the time of the continuously running unified time reference.

8. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 1, characterized in that, When the device does not have the sensor hub, the application processor executes the steps executed by the sensor hub.

9. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 1, characterized in that, The method further includes: Obtaining, by the sensor hub, the system time of the sensor hub and obtaining a first time from the continuously running unified time reference; Obtaining, by the application processor of the device, the system time of the application processor according to the interrupt signal and obtaining a second time from the continuously running unified time reference; Calculating, by the application processor, a time deviation between the system time of the sensor hub and the system time of the application processor according to the system time of the sensor hub and the system time of the application processor by using a filtering algorithm; and Using, by the application processor, the time deviation to re-correct the timestamp of each data sent from the sensor hub.

10. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 1, characterized in that The method further includes: Obtaining, by the sensor hub, the system time of the sensor through a controller; When the controller discovers the time synchronization protocol, the latch obtains a first time from the time of the always-running unified time reference. The sensor hub obtains a time offset by using a filtering algorithm based on the system time and the first time transmitted by the latch. And The sensor hub writes the time offset to the sensor through the controller for time synchronization between the sensor hub and the sensor. Wherein, the system time is based on a time reference different from the always-running unified time reference.

11. The high-precision timestamp stamping and synchronization method in a low-power sensor system according to claim 10, characterized in that, The sensor is a high-end sensor, and the controller is a communication interface of an internal integrated circuit / serial peripheral interface.

12. An apparatus for high-precision timestamping and synchronization in a low-power sensor system, comprising: A sensor hub coupled to a latch and a sensor, and the sensor hub performs: Receiving an interrupt signal from the sensor and performing interrupt processing to obtain an interrupt timestamp obtained through the latch, wherein the interrupt timestamp is obtained from an always-running unified time reference; Obtaining sensor data from the sensor; Predicting a predicted timestamp based on the sensor data volume and the interrupt timestamp by using a filtering algorithm, wherein the step of predicting the predicted timestamp includes: selecting a filtering parameter according to the sensor data volume and the interrupt timestamp; calculating a confidence interval based on the filtering parameter; obtaining a Kalman filter matrix according to the confidence interval, the interrupt timestamp, and the event trigger type; and predicting the predicted timestamp according to the Kalman filter matrix; and Correcting the timestamp of each sensor data based on the predicted timestamp.

13. The device for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 12, characterized in that, When the signal level output from the sensor interrupt pin changes, the interrupt timestamp is obtained from the always-running unified time reference, wherein the sensor interrupt pin is located between the sensor and the sensor hub.

14. The apparatus for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 12, characterized in that, The apparatus further includes: An interrupt controller of the sensor hub; Wherein, when the interrupt controller of the sensor hub initiates an interrupt, the interrupt timestamp is obtained from the always-running unified time reference.

15. The device for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 12, characterized in that, The filtering algorithm is a Kalman filter.

16. The apparatus for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 15, wherein, The step of correcting the timestamp of each sensor data based on the predicted timestamp includes: Determining whether the predicted timestamp is before the interrupt timestamp; Performing Kalman filter update to update the average interval between the predicted timestamp and the sensor data; and Correcting the timestamp of each sensor data based on the predicted timestamp and the average interval.

17. The device for high-precision timestamp stamping and synchronization in a low-power sensor system as claimed in claim 12, characterized in that, The apparatus further includes: An improved inter-integrated circuit controller coupled to the latch and the sensor; Wherein, when the in-band interrupt signal output from the improved inter-integrated circuit controller conforms to the setting in the improved inter-integrated circuit protocol, the interrupt timestamp is obtained from the always-running unified time reference.

18. The apparatus for high-precision timestamping and synchronization in a low-power sensor system according to claim 12, wherein The system time of the application processor, the system time of the sensor hub, and the system time of the latch are derived from the time of the always-running unified time reference.

19. The apparatus for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 12, wherein The apparatus further includes an application processor, and when the apparatus does not have the sensor hub, the application processor performs the steps performed by the sensor hub.

20. The apparatus for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 12, wherein The apparatus further includes: An application processor, coupled to the sensor hub; wherein, the sensor hub obtains the system time of the sensor hub and obtains a first time from the unified time reference that is always running, and the application processor further performs: Obtain the system time of the application processor according to the interrupt signal and obtain a second time from the unified time reference that is always running; According to the system time of the sensor hub and the system time of the application processor, use a filtering algorithm to calculate the time deviation between the system time of the sensor hub and the system time of the application processor; and Use the time deviation to recalibrate the timestamp of each data sent from the sensor hub.

21. The apparatus for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 12, wherein, The sensor hub and the latch further perform: By the sensor hub, obtain the system time of the sensor through a controller; When the controller discovers the time synchronization protocol, the latch obtains a first time from the time of the unified time reference that is always running; By the sensor hub, obtain a time offset by using a filtering algorithm according to the system time and the first time transmitted by the latch; And By the sensor hub, write the time offset into the sensor through the controller for time synchronization between the sensor hub and the sensor; wherein, the system time is based on a time reference different from the unified time reference that is always running.

22. The apparatus for high-precision timestamp stamping and synchronization in a low-power sensor system according to claim 21, wherein The sensor is a high-end sensor, and the controller is a communication interface of an internal integrated circuit / serial peripheral interface.

Images