Timestamp synchronization method, system, device and readable storage medium

Abstract

The application discloses a timestamp synchronization method, system, device and readable storage medium, relates to the technical field of wearable devices, and is applied to a wearable device. The wearable device comprises an MCU and an ISP. The MCU and the ISP are connected through GPIO pins. The method comprises the following steps: if the MCU detects a timestamp synchronization event trigger, a synchronization signal is sent to the ISP through the GPIO pin of the MCU; a first timestamp recorded when the MCU sends the synchronization signal and a second timestamp recorded when the ISP receives the synchronization signal are acquired; and sensor data collected by the MCU and image data collected by the ISP are synchronized according to the first timestamp and the second timestamp. The application improves the timestamp synchronization accuracy between the MCU and the ISP.

Description

Technical Field

[0001] This application relates to the field of wearable device technology, and in particular to a timestamp synchronization method, system, device and readable storage medium. Background Technology

[0002] With the rapid development of technology, wearable devices have gradually become an indispensable part of people's lives. In wearable devices, the microcontroller unit (MCU) and image signal processor (ISP) are two key components. The MCU is mainly responsible for the acquisition and processing of sensor data, such as data from the microphone and IMU (Inertial Measurement Unit). The ISP, on the other hand, focuses on processing image data acquired by the camera, providing users with clear, high-quality visual content. In various application scenarios of wearable devices, timestamp synchronization is a key technology to ensure data consistency and user experience. Synchronization accuracy has a significant impact on audio and video synchronization effects, as well as the accuracy of electronic image stabilization (EIS) algorithms.

[0003] Currently, a common method for achieving timestamp synchronization in wearable devices is based on timestamp transmission and calibration mechanisms. Specifically, when performing operations related to timestamp synchronization, the MCU transmits its own timestamp information to the ISP. After receiving the timestamp, the ISP calculates and records the difference Δt between its own time and the MCU's time, and then synchronizes the timestamps of relevant data between the MCU and the ISP based on this difference Δt.

[0004] However, the synchronization accuracy of the above scheme is limited by the inherent latency of the command transmission link. Due to communication delays during the process of the MCU sending commands to the ISP for receiving and responding, bus communication involves protocol encapsulation, transmission, parsing, and command queuing. The Δt value recorded by the ISP actually includes transmission delay errors. Especially in resource-constrained wearable devices such as smart glasses, such latency can be as high as milliseconds and is random, resulting in a significant deviation between the calculated Δt value and the actual time. This leads to low timestamp synchronization accuracy, which may in turn cause problems such as decreased EIS image stabilization and audio-visual synchronization distortion.

[0005] Therefore, improving the timestamp synchronization accuracy between the MCU and the ISP is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The main purpose of this application is to provide a timestamp synchronization method, system, device, and readable storage medium, aiming to solve how to improve the timestamp synchronization accuracy between MCU and ISP.

[0007] To achieve the above objectives, this application provides a timestamp synchronization method applied to a wearable device. The wearable device includes an MCU and an ISP, and the MCU and the ISP are interconnected by GPIO pins. The timestamp synchronization method includes:

[0008] If the MCU detects a timestamp synchronization event, it sends a synchronization signal to the ISP via the MCU's GPIO pin.

[0009] Obtain the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal;

[0010] Based on the first timestamp and the second timestamp, the sensor data collected by the MCU and the image data collected by the ISP are synchronized.

[0011] In one embodiment, the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp includes:

[0012] The data acquisition delay is obtained, wherein the data acquisition delay is the difference between the timestamp recorded when the image data is acquired by the ISP and the second timestamp;

[0013] The first corrected timestamp is obtained by calculating the sum between the data acquisition delay and the first timestamp. The acquisition timestamp of the image data is then updated to the first corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

[0014] In one embodiment, before the step of sending a synchronization signal to the ISP via the GPIO pin of the MCU if the MCU detects a timestamp synchronization event, the method further includes:

[0015] If the MCU detects that a data acquisition event has been triggered, it acquires the data to be acquired based on the data acquisition event.

[0016] If the data to be collected includes sensor data and image data, then the MCU is activated to collect sensor data and sends an image acquisition command to the ISP, so that the ISP can start collecting image data after receiving the image acquisition command;

[0017] After the ISP starts sending the acquired image data, the timestamp synchronization event is triggered once every preset time window.

[0018] In one embodiment, after the step of obtaining the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal, the method further includes:

[0019] If the MCU sends the image acquisition command to the ISP through the GPIO pin, then the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp is executed;

[0020] If the MCU sends the image acquisition command to the ISP via the serial port, the transmission delay between the MCU and the ISP is obtained, and the sensor data acquired by the MCU and the image data acquired by the ISP are synchronized based on the first timestamp, the second timestamp and the transmission delay.

[0021] The transmission delay is the data transmission delay between the MCU sending the image acquisition command and the ISP receiving the image acquisition command.

[0022] In one embodiment, the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp, the second timestamp, and the transmission delay includes:

[0023] The data acquisition delay is obtained, wherein the data acquisition delay is the difference between the timestamp recorded when the image data is acquired by the ISP and the second timestamp;

[0024] The first corrected timestamp is obtained by calculating the sum between the data acquisition delay and the first timestamp, and the second corrected timestamp is obtained by subtracting the transmission delay from the first corrected timestamp.

[0025] The acquisition timestamp of the image data is updated to the second corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

[0026] In one embodiment, the step of obtaining the transmission delay between the MCU and the ISP includes:

[0027] Obtain the receive-transmission time difference sent by the ISP, and obtain the transmit-transmission time difference of the MCU;

[0028] Calculate the difference between the transmission time difference and the reception time difference to obtain the transmission delay between the MCU and the ISP;

[0029] Wherein, the receiving transmission time difference is the difference between the timestamp recorded by the ISP when it receives the image acquisition command and the second timestamp, and the sending transmission time difference is the difference between the timestamp recorded by the MCU when it sends the image acquisition command and the first timestamp.

[0030] In one embodiment, before the step of obtaining the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal, the method further includes:

[0031] Obtain the first time difference observation value of the MCU and the second time difference observation value of the ISP;

[0032] The first time difference observation is input into the Kalman filter algorithm to obtain the first drift rate prediction value, and the second time difference observation is input into the Kalman filter algorithm to obtain the second drift rate prediction value.

[0033] The frequency division factor of the timer in the MCU is adjusted and / or the auto-reload register value of the timer in the MCU is reset based on the first drift rate prediction value. The frequency division factor of the timer in the ISP is adjusted and / or the auto-reload register value of the timer in the ISP is reset based on the second drift rate prediction value, so as to record a timestamp based on the adjusted timer.

[0034] Furthermore, to achieve the above objectives, this application also provides a timestamp synchronization system deployed in a wearable device. The wearable device includes an MCU and an ISP, with interconnected GPIO pins between the MCU and the ISP. The timestamp synchronization system includes:

[0035] The transmitting module is used to send a synchronization signal to the ISP through the GPIO pin of the MCU if the MCU detects that a timestamp synchronization event has been triggered.

[0036] The acquisition module is used to acquire a first timestamp recorded when the MCU sends the synchronization signal and a second timestamp recorded when the ISP receives the synchronization signal;

[0037] The synchronization module is used to synchronize the sensor data collected by the MCU and the image data collected by the ISP based on the first timestamp and the second timestamp.

[0038] In addition, to achieve the above objectives, this application also provides a wearable device, the wearable device including a processor, the processor including an MCU and an ISP, the MCU and the ISP having interconnected GPIO pins, the processor being used to execute the steps of the timestamp synchronization method described above.

[0039] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of the timestamp synchronization method described above.

[0040] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the timestamp synchronization method described above.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] This application interconnects the MCU and ISP of a wearable device via GPIO pins, meaning the MCU and ISP share GPIO pins. If the MCU detects a timestamp synchronization event, it sends a synchronization signal to the ISP through the MCU's GPIO pin. It acquires a first timestamp recorded when the MCU sends the synchronization signal and a second timestamp recorded when the ISP receives the synchronization signal. Based on the first and second timestamps, it synchronizes the sensor data collected by the MCU and the image data collected by the ISP. Thus, this embodiment transmits the synchronization signal through the GPIO pin directly connected between the MCU and ISP, avoiding the inherent delays caused by protocol encapsulation, transmission, parsing, and instruction queuing in traditional bus communication links. Because the GPIO pin uses a hardware-level triggered signal transmission method, its physical layer communication delay is extremely short and stable, eliminating the highly random and fluctuating protocol processing time in the path from the MCU to the ISP for the synchronization signal. When the MCU detects a synchronization event, the synchronization signal sent through the GPIO pin can be perceived by the ISP in almost real time. At this time, the independent recording of local timestamps by the MCU and ISP is limited only by the inherent microsecond-level signal propagation delay of the GPIO circuit, rather than the millisecond-level bus communication delay in traditional solutions. By directly comparing the local timestamps of the two devices at the moment the synchronization signal is triggered, the impact of transmission link delay on time difference calculation can be accurately isolated. It is easy to understand that based on the first and second timestamps, the time difference Δt value reflecting the actual clock deviation between the MCU and ISP can be obtained. The accuracy of this Δt value is no longer affected by the processing efficiency of the communication protocol stack or the instruction queue waiting time. This allows time stamp synchronization based on the first and second timestamps to ensure high-precision synchronization of sensor data and image data, thereby improving the time stamp synchronization accuracy between the MCU and ISP. This can effectively alleviate the degradation of EIS stabilization effect and audio-visual synchronization distortion caused by time stamp synchronization errors. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the first embodiment of the timestamp synchronization method of this application;

[0046] Figure 2 This is a schematic diagram of the timestamp synchronization process involved in one embodiment of the timestamp synchronization method of this application;

[0047] Figure 3 This is a flowchart illustrating the third embodiment of the timestamp synchronization method of this application;

[0048] Figure 4 This is a schematic diagram of the system architecture of the timestamp synchronization system in this application;

[0049] Figure 5 This is a schematic diagram of the hardware operating environment involved in the timestamp synchronization method apparatus in the embodiments of this application.

[0050] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] In wearable devices such as AI (Artificial Intelligence) glasses, the MCU is responsible for sensor data acquisition (such as data collected by microphones, IMUs, etc.), while the ISP module processes camera image data. The time synchronization accuracy of both directly affects the audio-visual synchronization and the accuracy of the EIS algorithm. Current technology transmits the timestamp from the MCU to the ISP when relevant instructions are executed. The ISP records the difference Δt between the two, and subsequently uses Δt to calibrate the relevant timestamps to synchronize the timestamps of related data. However, due to instruction transmission delays, there is a significant delay between the time the ISP receives the relevant instructions and the time the MCU sends them, resulting in a large Δt error and thus low timestamp synchronization accuracy.

[0053] Based on this, the main solution of this application is: to connect the MCU and ISP of the wearable device to each other through GPIO pins, that is, to share GPIO pins between the MCU and the ISP. If the MCU detects that a timestamp synchronization event is triggered, it sends a synchronization signal to the ISP through the GPIO pin of the MCU; to obtain the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal; and to synchronize the sensor data collected by the MCU and the image data collected by the ISP according to the first timestamp and the second timestamp.

[0054] This application transmits synchronization signals via GPIO pins that directly connect the MCU and ISP, avoiding the inherent delays caused by protocol encapsulation, transmission, parsing, and instruction queuing in traditional bus communication links. Because the GPIO pins use a hardware-level triggered signal transmission method, their physical layer communication delay is extremely short and stable. This eliminates the high-randomness and large-amplitude protocol processing time in the path from the MCU to the ISP for the synchronization signal. When the MCU detects a synchronization event, the synchronization signal sent via the GPIO pin can be perceived by the ISP almost in real time. At this point, the independent recording of local timestamps by both the MCU and ISP is limited only by the inherent microsecond-level signal propagation delay of the GPIO circuit, rather than the millisecond-level bus communication delay in traditional solutions. By directly comparing the local timestamps of the two devices at the moment the synchronization signal is triggered, the impact of transmission link delay on time difference calculation can be accurately isolated. It is easy to understand that based on the first and second timestamps, the time difference Δt value reflecting the deviation between the MCU and the ISP's real clock can be obtained. The accuracy of this Δt value is no longer affected by the processing efficiency of the communication protocol stack or the instruction queue waiting time. This allows time stamp synchronization based on the first and second timestamps to ensure high-precision synchronization of sensor data and image data, thereby improving the time stamp synchronization accuracy of the MCU and ISP. In turn, it can effectively alleviate the problems of EIS stabilization effect degradation and audio-visual synchronization distortion caused by time stamp synchronization errors.

[0055] It should be noted that the execution subject of each embodiment of the timestamp synchronization method of this application can be a wearable device capable of realizing the above functions, such as smart glasses, smartwatches, etc., and the embodiments of the timestamp synchronization method of this application do not impose specific limitations on this.

[0056] Based on this, this application proposes a timestamp synchronization method according to a first embodiment. The image acquisition method is applied to a wearable device, which includes an MCU and an ISP. The MCU and the ISP are interconnected by GPIO pins. (Refer to...) Figure 1 As shown, please refer to Figure 1 As shown, the timestamp synchronization method includes the following steps S10 to S30:

[0057] Step S10: If the MCU detects a timestamp synchronization event, it sends a synchronization signal to the ISP through the GPIO pin of the MCU.

[0058] The GPIO pin refers to a General Purpose Input / Output (GPIO) pin. As a programmable digital interface, it supports bidirectional level signal transmission and enables direct interaction between hardware modules without relying on complex communication protocols. There is at least one pair of interconnected GPIO pins between the MCU and the ISP. These interconnected GPIO pins communicate directly via hardware circuitry, enabling protocol-free level signal transmission. Furthermore, the GPIO pins of both devices can be connected through pre-defined traces on a printed circuit board (PCB) or physical wiring on a flexible printed circuit board (FPC), or in other ways, provided that hardware communication between the two devices is possible. This embodiment does not impose specific limitations on this.

[0059] After the MCU detects a timestamp synchronization event, it sends a synchronization signal to the ISP via its GPIO pin. This synchronization signal can be any signal that the GPIO pin can send to notify the ISP to record the timestamp. For example, upon detecting a timestamp synchronization event, a specific level signal can be output from the GPIO pin (e.g., switching the pin level from high to low). This type of signal directly acts on the corresponding GPIO input pin of the ISP, triggering it to immediately record the timestamp, thus avoiding delays introduced by software protocol interaction.

[0060] Step S20: Obtain the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal;

[0061] The first timestamp is the timestamp recorded when the MCU sends the synchronization signal, and the second timestamp is the timestamp recorded when the ISP receives the synchronization signal.

[0062] The first timestamp can be generated by the MCU within the same clock cycle that drives the GPIO pin level transition. The MCU's internal time base module (e.g., a 32-bit counter based on a crystal oscillator) can automatically latch the current count value during the hardware triggering phase of the synchronization signal output, thereby eliminating the instruction execution delay that may be introduced when the software records the timestamp. Similarly, the second timestamp on the ISP side can be directly triggered by the level transition event of the GPIO input pin to activate the hardware capture function of its timer register. This hardware-level timestamp recording mechanism ensures that the time base difference between the two devices reflects only the actual deviation of their internal clock sources, eliminating additional delays outside the signal transmission path.

[0063] Step S30: Synchronize the sensor data collected by the MCU and the image data collected by the ISP based on the first timestamp and the second timestamp.

[0064] After obtaining the first and second timestamps, the sensor data acquired by the MCU and the image data acquired by the ISP are synchronized based on these timestamps. It can be understood that the first and second timestamps reflect the time difference Δt between the MCU and the ISP; specifically, this time difference is the difference between the first and second timestamps. Therefore, based on the first and second timestamps, the sensor data and image data that need to be synchronized can be time-stamped. Specifically, the sensor data can be data acquired by sensors in the MCU, such as audio data or IMU (Inertial Measurement Unit) data. The image data can be data acquired by image sensors in the ISP, such as one or more frames of an image.

[0065] During synchronous processing, the timestamps of sensor data can be uniformly corrected to equivalent values ​​under the ISP time base, and the timestamps of image data can also be uniformly corrected to equivalent values ​​under the MCU time base.

[0066] It should be noted that after obtaining the first and second timestamps in this record, the timestamps of the sensor data and image data are synchronized based on the first and second timestamps until the next timestamp synchronization event is triggered. After updating the first and second timestamps, the timestamps of the sensor data and image data are synchronized again with the latest first and second timestamps.

[0067] This embodiment transmits synchronization signals via GPIO pins that directly connect the MCU and ISP, avoiding the inherent delays caused by protocol encapsulation, transmission, parsing, and instruction queuing in traditional bus communication links. Because the GPIO pins use a hardware-level triggered signal transmission method, their physical layer communication delay is extremely short and stable. This eliminates the high-randomness, high-fluctuation protocol processing time in the path from the MCU to the ISP for the synchronization signal. When the MCU detects a synchronization event, the synchronization signal sent via the GPIO pins can be perceived by the ISP almost in real time. At this point, the independent recording of local timestamps by both the MCU and ISP is limited only by the inherent microsecond-level signal propagation delay of the GPIO circuit, rather than the millisecond-level bus communication delay in traditional solutions. By directly comparing the local timestamps of the two devices at the moment the synchronization signal is triggered, the impact of transmission link delay on time difference calculation can be accurately isolated. It is easy to understand that based on the first and second timestamps, the time difference Δt value reflecting the deviation between the MCU and the ISP's real clock can be obtained. The accuracy of this Δt value is no longer affected by the processing efficiency of the communication protocol stack or the instruction queue waiting time. This allows time stamp synchronization based on the first and second timestamps to ensure high-precision synchronization of sensor data and image data, thereby improving the time stamp synchronization accuracy of the MCU and ISP. In turn, it can effectively alleviate the problems of EIS stabilization effect degradation and audio-visual synchronization distortion caused by time stamp synchronization errors.

[0068] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, before the step of sending a synchronization signal to the ISP through the GPIO pin of the MCU if the MCU detects a timestamp synchronization event, the method further includes:

[0069] Step A10: If the MCU detects that a data acquisition event has been triggered, then the data to be acquired is obtained based on the data acquisition event.

[0070] The triggering condition for this data acquisition event can be associated with user operation (such as pressing the record button) or system-level scheduling (such as the initiation of a periodic data acquisition task). When the MCU detects the event, it first parses the event payload to determine the type of data to be acquired, such as one or a combination of motion sensor data, ambient light data, audio data, and image data.

[0071] Step A20: If the data to be collected includes sensor data and image data, then the MCU is started to collect sensor data and send an image acquisition command to the ISP, so that the ISP starts collecting image data after receiving the image acquisition command;

[0072] If the data to be collected includes sensor data that needs to be collected by the MCU and image data that needs to be collected by the ISP, the MCU sends an image acquisition command to the ISP while starting its own sensor data acquisition thread, so that the ISP can start the image sensor to collect image data after receiving the image acquisition command.

[0073] Step A30: After the ISP starts sending the acquired image data, the timestamp synchronization event is triggered once every preset time window.

[0074] After the ISP starts sending the acquired image data to the MCU, the MCU can actively trigger a timestamp synchronization event every preset time window. When the timestamp synchronization event is triggered, the MCU and ISP record the first and second timestamps of this synchronization cycle through the GPIO synchronization mechanism, and use them to synchronize the timestamps of the sensor data and image data. The preset time window can be any pre-set time window greater than zero, such as one frame of image as the time window; or a 500-millisecond time window, etc. This embodiment does not impose specific limitations on this.

[0075] In this embodiment, a timestamp synchronization event is triggered every preset time window to update the first and second timestamps recorded by the MCU and ISP, thereby updating the time difference between the MCU and ISP accordingly. This "initial synchronization + periodic resynchronization" strategy avoids the resource consumption problem caused by continuous high-frequency synchronization and effectively suppresses the long-term cumulative error caused by clock drift, thus ensuring that the timestamp synchronization accuracy of the sensor and image data remains high even during long-term continuous recording.

[0076] In one possible implementation, after the step of obtaining the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal, the method further includes:

[0077] Step B10: If the MCU sends the image acquisition command to the ISP through the GPIO pin, then the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp is executed.

[0078] When the MCU sends an image acquisition command to the ISP via GPIO pins, the transmission delay between the MCU sending the command and the ISP receiving it is on the order of microseconds and can be ignored due to the direct hardware connection between the GPIO pins. This means the ISP immediately performs image acquisition after the MCU sends the command, and the MCU sending the command and the sensor data acquisition are synchronized. Therefore, the MCU and ISP can be considered to be performing data acquisition synchronously. In this case, the sensor data and image data can be directly synchronized based on the first and second timestamps. Specifically, the MCU synchronously starts the sensor data acquisition thread at the instant the GPIO level changes when issuing the image acquisition command, and the ISP immediately starts the image sensor exposure upon receiving the command. Since the GPIO command transmission delay is negligible, the only difference between the sensor data's starting timestamp (e.g., the time of the first sampling point of the IMU) and the image data's initial frame exposure timestamp is the clock source deviation. No additional compensation is needed for the time difference between the MCU and ISP; that is, the time difference between the MCU and ISP can be considered as the difference between the first and second timestamps.

[0079] Step B20: If the MCU sends the image acquisition command to the ISP via the serial port, then the transmission delay between the MCU and the ISP is obtained, and the sensor data acquired by the MCU and the image data acquired by the ISP are synchronized based on the first timestamp, the second timestamp and the transmission delay.

[0080] If the MCU sends image acquisition commands to the ISP via a serial port, the transmission delay introduced by the serial communication link must be considered. This transmission delay is the time between the MCU sending the image acquisition command and the ISP receiving the image acquisition command. For example, assuming the MCU sends the image acquisition command at timestamp A1, and the ISP receives the image acquisition command at timestamp A2, then the transmission delay is the difference between A1 and A2. Here, "serial port" refers to a physical interface that supports asynchronous or synchronous serial communication. Specifically, it can be a communication link implemented through I2C (Inter-Integrated Circuit), SPI (Serial Peripheral Interface), UART (Universal Asynchronous Receiver / Transmitter), etc. This embodiment does not impose specific limitations on this.

[0081] In this embodiment, through the above branch logic, the system can adaptively maintain the timestamp alignment accuracy between the sensor and image data, regardless of whether a low-latency GPIO pin or a general serial port is used as the command transmission channel. This ensures that even in complex communication scenarios (such as switching to serial port command transmission when GPIO pin resources are scarce), the stringent requirements for timestamp synchronization accuracy in applications such as EIS stabilization, audio-visual synchronization, and multimodal data fusion can still be met.

[0082] In one possible implementation, the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp includes:

[0083] Step C10, obtain the data acquisition delay, wherein the data acquisition delay is the difference between the timestamp recorded when the ISP acquires the image data and the second timestamp;

[0084] It should be noted that if the image data includes multiple frames, the data acquisition delay of each frame can be obtained separately, and the timestamp of that frame can be corrected based on the data acquisition delay. For each frame, the data acquisition delay is the delay between the timestamp recorded by the ISP when it acquires that frame and the second timestamp.

[0085] When acquiring image data, the ISP can record the data acquisition delay of each frame and send it to the MCU. For ease of explanation, let the first timestamp be t0_mcu, the second timestamp be t0_isp, the timestamp recorded by the ISP when acquiring image data be current_time1, and the data acquisition delay be delta_t_isp. Then delta_t_isp can be expressed by the formula: delta_t_isp = current_time1 - t0_isp.

[0086] Step C20: Calculate the sum between the data acquisition delay and the first timestamp to obtain the first corrected timestamp, and update the acquisition timestamp of the image data to the first corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

[0087] The calculation of the first corrected timestamp can be expressed by the formula: First corrected timestamp = t0_mcu + delta_t_isp = t0_mcu - t0_isp + current_time1.

[0088] It is understandable that the calculation of the first corrected timestamp is to superimpose delta_t_isp onto the MCU base time t0_mcu, thus mapping the start time of the image data to the MCU time axis. Therefore, after updating the timestamp of the image data to the first corrected timestamp, the timestamp of the image data has been synchronized to the MCU time axis base. For the sensor data acquired by the MCU, the MCU only needs to keep the timestamp recorded when the sensor data was acquired, and there is no need to modify the timestamp of the sensor data.

[0089] In one possible implementation, the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp, the second timestamp, and the transmission delay includes:

[0090] Step D10: Obtain the data acquisition delay, wherein the data acquisition delay is the difference between the timestamp recorded when the ISP acquires the image data and the second timestamp;

[0091] Step D20: Calculate the sum between the data acquisition delay and the first timestamp to obtain the first corrected timestamp, and subtract the transmission delay from the first corrected timestamp to obtain the second corrected timestamp;

[0092] Step D30: Update the acquisition timestamp of the image data to the second corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

[0093] The calculation of the first corrected timestamp is the same as described above, and will not be elaborated here. After obtaining the first timestamp, the second corrected timestamp is obtained by subtracting the transmission delay from the first corrected timestamp. Let the transmission delay be delta_tr, then the second corrected timestamp can be expressed by the formula: Second corrected timestamp = t0_mcu + delta_t_isp - delta_tr.

[0094] It should be noted that, similarly, after updating the timestamp of the image data to the second corrected timestamp, the timestamp of the image data has been synchronized to the MCU reference. For the sensor data acquired by the MCU, the MCU only needs to keep the timestamp recorded when the sensor data was acquired, and there is no need to modify the timestamp of the sensor data.

[0095] In one possible implementation, the step of obtaining the transmission delay between the MCU and the ISP includes:

[0096] Step E10: Obtain the receive-transmission time difference of the ISP and the transmit-transmission time difference of the MCU;

[0097] The receive transmission time difference is the difference between the timestamp recorded by the ISP when it receives the image acquisition command and the second timestamp, and the send transmission time difference is the difference between the timestamp recorded by the MCU when it sends the image acquisition command and the first timestamp.

[0098] It should be noted that the ISP can record the timestamp current_time when it receives the image acquisition command after receiving the image acquisition command, and record the second timestamp t0_isp when it receives the synchronization signal. Then it can calculate the reception and transmission time difference deltat1_isp. Specifically, the reception and transmission time difference can be expressed by the formula: deltat1_isp = current_time - t0_isp.

[0099] Based on a similar calculation method for the receive-transmission time difference, the MCU can calculate its corresponding transmit-transmission time difference deltat1_mcu. Specifically, if the timestamp recorded when the MCU sends the image acquisition command is current_time2, then the transmit-transmission time difference can be expressed by the formula: deltat1_mcu = current_time2 - t0_mcu.

[0100] Step E20: Calculate the difference between the transmission time difference and the reception time difference to obtain the transmission delay between the MCU and the ISP;

[0101] After obtaining the transmission time difference deltat1_mcu and the reception time difference deltat1_isp, the transmission delay delta_tr between the MCU and the ISP can be calculated. Specifically, the transmission delay can be expressed by the formula delta_tr = deltat1_mcu - deltat1_isp.

[0102] For example, to aid in understanding the technical concept or principle of the timestamp synchronization method combined with the first embodiment described above, a specific embodiment is provided below. In this specific embodiment, the image acquisition method is applied to smart glasses, and the MCU and ISP in the smart glasses share GPIO pins to send pulse signals. Based on this, referring to... Figure 2 As shown, the image acquisition process includes:

[0103] 1. MCU starts ISP.

[0104] 2. The MCU starts acquiring audio and IMU sensor data and sends a command (i.e., image acquisition command) to the ISP to notify the ISP to record, that is, to acquire image data.

[0105] 3. The MCU pulls the GPIO low to send a synchronization signal to the ISP. The ISP and the MCU simultaneously record the timestamps of receiving and sending the synchronization signal as t0_mcu and t0_isp.

[0106] 4. The ISP sends the receive transmission time difference deltat1_isp(current_time-t0_isp) to the MCU, where current_time is the timestamp recorded by the ISP when it receives the command in step 2.

[0107] 5. The MCU calculates the transmission delay delta_tr, delta_tr = deltat1_mcu - deltat1_isp, where deltat1_mcu = current_time2 - t0_mcu, and current_time2 is the timestamp recorded by the MCU when sending the command in step 2.

[0108] 6. The ISP sends the encoded data to the MCU. Specifically, the data acquisition delay of each frame is delta_t_isp(current_time1-t0_isp), and the image is encoded and sent to the MCU. Here, current_time1 is the timestamp recorded by the ISP when it acquires the image frame.

[0109] 7. The MCU corrects the timestamp of the frame image based on the received delta_t_isp and the calculated delta_tr. Specifically, the acquisition timestamp of the frame image is corrected to t0_mcu + delta_t_isp - delta_tr to complete the timestamp synchronization of the frame image.

[0110] 8. Using one frame of image as a preset time window, trigger a timestamp synchronization event after each frame of image is corrected, and return to step 3 until all image data is synchronized with the timestamp of the sensor data.

[0111] 9. When the MCU detects that recording needs to be stopped, it sends a stop recording command to the ISP to notify the ISP to stop recording, that is, to stop image acquisition.

[0112] 10. After the ISP stops recording and all images are sent to the MCU, notify the MCU that recording is complete.

[0113] 11. The MCU saves H.265 encoded video files, audio files, and IMU data. After each synchronization of timestamps, the synchronized timestamp information is placed in the H.265 encoded Supplemental Enhancement Information (SEI).

[0114] It should be noted that the above examples are only used to help understand this embodiment and do not constitute a limitation on the timestamp synchronization process of this embodiment. Any simple modifications based on this technical concept are within the protection scope of this application.

[0115] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. On this basis, refer to Figure 3 As shown, before the step of obtaining the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal, the method further includes:

[0116] Step F10: Obtain the first time difference observation value of the MCU and the second time difference observation value of the ISP;

[0117] The first time difference observation can specifically be the difference between the local timestamp recorded by the MCU and the reference timestamp recorded by the reference clock at the same time; the second time difference observation can specifically be the difference between the local timestamp recorded by the ISP and the reference timestamp recorded by the reference clock at the same time.

[0118] The reference clock refers to a high-reliability clock source inside or outside the system, such as a global system timer driven by a high-precision crystal oscillator in the MCU, or a stable clock module synchronized by a phase-locked loop inside the ISP. Such clock sources are directly connected to the ISP timer in hardware to maintain periodic synchronization. Other examples include the 1PPS pulse signal of the GPS module, the NTP (Network Time Protocol) calibration clock transmitted by the smartphone via Bluetooth or Wi-Fi, or the global clock output by an independent temperature-compensated crystal oscillator module.

[0119] Furthermore, a reference time interval can be obtained. This reference time interval refers to the standard time length between two adjacent time difference observations, such as the 1-second pulse interval of GPS or the millisecond-level timer trigger interval of a mobile phone clock. This reference time interval, along with the time difference observations, is input into the Kalman filter algorithm. The inherent stability of the reference clock source ensures the reference time interval. As an input parameter of the Kalman filter algorithm, it models the dynamic change of the clock drift rate over time, thereby improving the accuracy of the drift rate prediction and the system's ability to suppress long-term drift.

[0120] Step F20: Input the first time difference observation value into the Kalman filter algorithm to obtain the first drift rate prediction value, and input the second time difference observation value into the Kalman filter algorithm to obtain the second drift rate prediction value;

[0121] The Kalman filter algorithm is a state estimation model where the state equation models the dynamic process of clock drift rate (the change in time difference per unit time). The observation equation correlates the measured time difference with the drift rate linearly. By iteratively updating the state covariance matrix and Kalman gain, it outputs the optimal estimate of the timer drift rate, i.e., the predicted drift rate, which is used to quantify the deviation trend of the timer frequency, such as the number of microseconds faster / slower per second.

[0122] Step F30: Adjust the division factor of the timer in the MCU and / or reset the auto-reload register value of the timer in the MCU according to the first drift rate prediction value, and adjust the division factor of the timer in the ISP and / or reset the auto-reload register value of the timer in the ISP according to the second drift rate prediction value, so as to record a timestamp based on the adjusted timer.

[0123] It should be noted that for timers in MCUs and ISPs, the adjusted frequency divider is negatively correlated with the predicted drift rate, while the auto-reload register value is positively correlated with the predicted drift rate.

[0124] The frequency division factor of the ISP / MCU timer is dynamically adjusted based on the predicted drift rate. For example, if the predicted timer frequency is too high, the frequency division factor is increased to reduce the counting frequency, and / or the auto-reload register value is adjusted. For example, if the timing period needs to be extended, the reload value is increased to prolong the single-cycle duration. After adjustment, the actual frequency of the ISP / MCU timer approaches the global timing reference, thereby suppressing clock drift and improving the accuracy of the timestamp. Specifically, the adjusted frequency division factor is negatively correlated with the predicted drift rate (the larger the drift rate, the larger the adjustment of the frequency division factor to compensate for high-frequency deviations), and the auto-reload register value is positively correlated with the predicted drift rate (when the drift rate is positive, the reload value is increased to prolong the cycle).

[0125] In addition, it should be noted that this method can also be used to adjust the divider coefficient of the timer in the MCU and / or reset the auto-reload register value of the timer, which will not be elaborated here.

[0126] By constructing an adaptive clock synchronization mechanism based on Kalman filtering, the problem of timestamp inaccuracy caused by clock drift between the ISP and MCU in wearable devices is effectively solved. By reusing multiple reliable clock references from various sources, such as the MCU's high-precision crystal oscillator, the ISP's phase-locked loop synchronization module, or an external GPS / mobile phone clock, the system can dynamically acquire the observed difference between the ISP / MCU's local time and the reference time. Combined with the stability input of the reference time interval, the system uses a Kalman filtering algorithm to model the dynamic changes in clock drift in real time, accurately predicting the frequency deviation trend per unit time. Based on the predicted drift rate, the system automatically adjusts the division factor and automatic reload register value of the ISP / MCU timer in reverse. For example, when a timer frequency is detected to be too high, the division factor is increased to reduce the counting rate, or the reload value is extended to prolong the counting period when a positive deviation accumulates. This brings the actual frequency of the ISP / MCU timer to the reference clock reference, suppressing microsecond-level timestamp errors caused by long-term drift. This closed-loop calibration process achieves high-precision time synchronization between the ISP and the sensor data acquisition end by dynamically tuning existing timer parameters without requiring additional hardware modules. Its adaptability is compatible with various scenarios, such as indoor reliance on the MCU's internal clock and outdoor switching to GPS / mobile phone clocks, while avoiding the additional power consumption caused by frequent bus communication or complex protocol parsing in traditional solutions. Through the coordinated optimization of hardware-level timer calibration and software algorithms, the system's clock stability and spatiotemporal consistency of multimodal data are significantly improved in complex environments, providing reliable underlying time reference support for high-speed image acquisition and motion compensation.

[0127] Furthermore, this application also proposes a timestamp synchronization system deployed in a wearable device. The wearable device includes an MCU and an ISP, and the MCU and the ISP are interconnected by GPIO pins. (Refer to...) Figure 4 As shown, the timestamp synchronization system includes:

[0128] The sending module 10 is used to send a synchronization signal to the ISP through the GPIO pin of the MCU if the MCU detects that a timestamp synchronization event has been triggered.

[0129] The acquisition module 20 is used to acquire a first timestamp recorded when the MCU sends the synchronization signal and a second timestamp recorded when the ISP receives the synchronization signal;

[0130] The synchronization module 30 is used to synchronize the sensor data collected by the MCU and the image data collected by the ISP based on the first timestamp and the second timestamp.

[0131] In one embodiment, the synchronization module 30 is further configured to:

[0132] The data acquisition delay is obtained, wherein the data acquisition delay is the difference between the timestamp recorded when the image data is acquired by the ISP and the second timestamp;

[0133] The first corrected timestamp is obtained by calculating the sum between the data acquisition delay and the first timestamp. The acquisition timestamp of the image data is then updated to the first corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

[0134] In one embodiment, the sending module 10 is further configured to:

[0135] If the MCU detects that a data acquisition event has been triggered, it acquires the data to be acquired based on the data acquisition event.

[0136] If the data to be collected includes sensor data and image data, then the MCU is activated to collect sensor data and sends an image acquisition command to the ISP, so that the ISP can start collecting image data after receiving the image acquisition command;

[0137] After the ISP starts sending the acquired image data, the timestamp synchronization event is triggered once every preset time window.

[0138] In one embodiment, the synchronization module 30 is further configured to:

[0139] If the MCU sends the image acquisition command to the ISP through the GPIO pin, then the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp is executed;

[0140] If the MCU sends the image acquisition command to the ISP via the serial port, the transmission delay between the MCU and the ISP is obtained, and the sensor data acquired by the MCU and the image data acquired by the ISP are synchronized based on the first timestamp, the second timestamp and the transmission delay.

[0141] The transmission delay is the data transmission delay between the MCU sending the image acquisition command and the ISP receiving the image acquisition command.

[0142] In one embodiment, the synchronization module 30 is further configured to:

[0143] The data acquisition delay is obtained, wherein the data acquisition delay is the difference between the timestamp recorded when the image data is acquired by the ISP and the second timestamp;

[0144] The first corrected timestamp is obtained by calculating the sum between the data acquisition delay and the first timestamp, and the second corrected timestamp is obtained by subtracting the transmission delay from the first corrected timestamp.

[0145] The acquisition timestamp of the image data is updated to the second corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

[0146] In one embodiment, the synchronization module 30 is further configured to:

[0147] Obtain the receive-transmission time difference sent by the ISP, and obtain the transmit-transmission time difference of the MCU;

[0148] Calculate the difference between the transmission time difference and the reception time difference to obtain the transmission delay between the MCU and the ISP;

[0149] Wherein, the receiving transmission time difference is the difference between the timestamp recorded by the ISP when it receives the image acquisition command and the second timestamp, and the sending transmission time difference is the difference between the timestamp recorded by the MCU when it sends the image acquisition command and the first timestamp.

[0150] In one embodiment, the timestamp synchronization system further includes a dynamic compensation module, used for:

[0151] Obtain the first time difference observation value of the MCU and the second time difference observation value of the ISP;

[0152] The first time difference observation is input into the Kalman filter algorithm to obtain the first drift rate prediction value, and the second time difference observation is input into the Kalman filter algorithm to obtain the second drift rate prediction value.

[0153] The frequency division factor of the timer in the MCU is adjusted and / or the auto-reload register value of the timer in the MCU is reset based on the first drift rate prediction value. The frequency division factor of the timer in the ISP is adjusted and / or the auto-reload register value of the timer in the ISP is reset based on the second drift rate prediction value, so as to record a timestamp based on the adjusted timer.

[0154] Furthermore, this application also proposes a wearable device, which includes a processor, comprising an MCU and an ISP, wherein the MCU and the ISP are interconnected by GPIO pins, and the processor is used to execute the steps of the image acquisition method described above.

[0155] refer to Figure 5The diagram illustrates a structural schematic suitable for implementing wearable devices according to embodiments of this application. Wearable devices in embodiments of this application may also include, but are not limited to, smart glasses, smartwatches, virtual reality glasses, virtual reality helmets, extended reality glasses, extended reality helmets, headphones, etc. Figure 5 The wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0156] like Figure 5 As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the wearable device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show wearable devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0157] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0158] The wearable device provided in this application, employing the timestamp synchronization method described in the above embodiments, can solve the problem of improving the timestamp synchronization accuracy between the MCU and the ISP. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the timestamp synchronization method provided in the above embodiments, and other technical features of the wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0159] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0161] In addition, to achieve the above objectives, this application also provides a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the timestamp synchronization method described above.

[0162] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0163] The aforementioned computer-readable storage medium may be included in a wearable device; or it may exist independently and not incorporated into a wearable device.

[0164] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by a wearable device, the wearable device causes the following: if the MCU detects a timestamp synchronization event, it sends a synchronization signal to the ISP through the GPIO pin of the MCU; it acquires a first timestamp recorded when the MCU sends the synchronization signal and a second timestamp recorded when the ISP receives the synchronization signal; and it synchronizes the sensor data collected by the MCU and the image data collected by the ISP based on the first timestamp and the second timestamp.

[0165] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0167] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0168] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described timestamp synchronization method, which can solve the problem of how to improve the timestamp synchronization accuracy between the MCU and the ISP. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the timestamp synchronization method provided in the above embodiments, and will not be repeated here.

[0169] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the timestamp synchronization method described above.

[0170] The specific implementation of the computer program product in this application is basically the same as the embodiments of the timestamp synchronization method described above, and will not be repeated here.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0174] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A timestamp synchronization method, characterized in that, The timestamp synchronization method is applied to a wearable device, which includes an MCU and an ISP, with GPIO pins interconnected between the MCU and the ISP. The timestamp synchronization method includes: If the MCU detects a timestamp synchronization event, it sends a synchronization signal to the ISP via the MCU's GPIO pin. Obtain the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal; Based on the first timestamp and the second timestamp, the sensor data collected by the MCU and the image data collected by the ISP are synchronized.

2. The timestamp synchronization method as described in claim 1, characterized in that, The step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp includes: The data acquisition delay is obtained, wherein the data acquisition delay is the difference between the timestamp recorded when the image data is acquired by the ISP and the second timestamp; The first corrected timestamp is obtained by calculating the sum between the data acquisition delay and the first timestamp. The acquisition timestamp of the image data is then updated to the first corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

3. The timestamp synchronization method as described in claim 1, characterized in that, Before the step of sending a synchronization signal to the ISP via the GPIO pin of the MCU if the MCU detects a timestamp synchronization event, the method further includes: If the MCU detects that a data acquisition event has been triggered, it acquires the data to be acquired based on the data acquisition event. If the data to be collected includes sensor data and image data, then the MCU is activated to collect sensor data and sends an image acquisition command to the ISP, so that the ISP can start collecting image data after receiving the image acquisition command; After the ISP starts sending the acquired image data, the timestamp synchronization event is triggered once every preset time window.

4. The timestamp synchronization method as described in claim 3, characterized in that, After the step of obtaining the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal, the method further includes: If the MCU sends the image acquisition command to the ISP through the GPIO pin, then the step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp and the second timestamp is executed; If the MCU sends the image acquisition command to the ISP via the serial port, the transmission delay between the MCU and the ISP is obtained, and the sensor data acquired by the MCU and the image data acquired by the ISP are synchronized based on the first timestamp, the second timestamp and the transmission delay. The transmission delay is the data transmission delay between the MCU sending the image acquisition command and the ISP receiving the image acquisition command.

5. The timestamp synchronization method as described in claim 4, characterized in that, The step of synchronizing the sensor data acquired by the MCU and the image data acquired by the ISP based on the first timestamp, the second timestamp, and the transmission delay includes: The data acquisition delay is obtained, wherein the data acquisition delay is the difference between the timestamp recorded when the image data is acquired by the ISP and the second timestamp; The first corrected timestamp is obtained by calculating the sum between the data acquisition delay and the first timestamp, and the second corrected timestamp is obtained by subtracting the transmission delay from the first corrected timestamp. The acquisition timestamp of the image data is updated to the second corrected timestamp to complete the synchronization of the sensor data acquired by the MCU and the image data acquired by the ISP.

6. The timestamp synchronization method as described in claim 4, characterized in that, The step of obtaining the transmission delay between the MCU and the ISP includes: Obtain the receive-transmission time difference sent by the ISP, and obtain the transmit-transmission time difference of the MCU; Calculate the difference between the transmission time difference and the reception time difference to obtain the transmission delay between the MCU and the ISP; Wherein, the receiving transmission time difference is the difference between the timestamp recorded by the ISP when it receives the image acquisition command and the second timestamp, and the sending transmission time difference is the difference between the timestamp recorded by the MCU when it sends the image acquisition command and the first timestamp.

7. The timestamp synchronization method according to any one of claims 1 to 6, characterized in that, Before the step of obtaining the first timestamp recorded when the MCU sends the synchronization signal and the second timestamp recorded when the ISP receives the synchronization signal, the method further includes: Obtain the first time difference observation value of the MCU and the second time difference observation value of the ISP; The first time difference observation is input into the Kalman filter algorithm to obtain the first drift rate prediction value, and the second time difference observation is input into the Kalman filter algorithm to obtain the second drift rate prediction value. The frequency division factor of the timer in the MCU is adjusted and / or the auto-reload register value of the timer in the MCU is reset based on the first drift rate prediction value. The frequency division factor of the timer in the ISP is adjusted and / or the auto-reload register value of the timer in the ISP is reset based on the second drift rate prediction value, so as to record a timestamp based on the adjusted timer.

8. A timestamp synchronization system, characterized in that, The timestamp synchronization system is deployed in a wearable device, which includes an MCU and an ISP. The MCU and the ISP are interconnected via GPIO pins. The timestamp synchronization system includes: The transmitting module is used to send a synchronization signal to the ISP through the GPIO pin of the MCU if the MCU detects that a timestamp synchronization event has been triggered. The acquisition module is used to acquire a first timestamp recorded when the MCU sends the synchronization signal and a second timestamp recorded when the ISP receives the synchronization signal; The synchronization module is used to synchronize the sensor data collected by the MCU and the image data collected by the ISP based on the first timestamp and the second timestamp.

9. A wearable device, characterized in that, The wearable device includes a processor, which includes an MCU and an ISP, with GPIO pins interconnected between the MCU and the ISP, and the processor is used to perform the steps of the timestamp synchronization method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium includes a computer-readable storage medium storing a timestamp synchronization program, which, when executed by a processor, implements the steps of the timestamp synchronization method as described in any one of claims 1 to 7.

Images