Indicating clock drift of internal clock in sensor device

Abstract

A clock drift indicating an internal clock in a sensor device is provided. A method for indicating a clock drift of an internal clock comprised in a sensor device, the method comprising: capturing (S302), by the sensor device, a first data frame; associating (S304), by the sensor device, the first data frame with a first point in time indicating a point in time of the internal clock when the first data frame is captured; receiving (S306), by the sensor device, time synchronization data via a network protocol for clock synchronization; determining (S308), by the sensor device, a reference time point using the time synchronization data, the reference time point indicating a time point of the reference clock when the first data frame is captured; determining (S310), by the sensor device, an offset between the first point in time and a reference point in time; and associating, by the sensor device, the first data frame with data indicative of the offset when the offset exceeds a threshold offset (S312).

Description

Technical Field

[0001] The present invention relates to networking systems and time synchronization protocols, and more particularly, to methods, apparatus, and systems for indicating clock drift in an internal clock in a sensor device. Background Technology

[0002] In modern networked environments, sensor devices such as cameras and microphones, as well as other network-based sensor devices, rely heavily on accurate timing to ensure synchronized operation. These devices typically utilize network protocols for clock synchronization, which allows them to periodically adjust their internal clocks to match a central time reference. This synchronization is crucial for applications that depend on precisely added timestamps, such as video surveillance and event monitoring.

[0003] A widely used protocol for clock synchronization is the Network Time Protocol (NTP). NTP enables devices to periodically adjust their clock frequency to gradually synchronize with the time of an NTP server, thus ensuring that their internal clock time remains accurate over time (i.e., achieving the same time value and average time-ticking speed as the NTP server). NTP and other similar protocols typically have specific rules regarding how devices should quickly adjust their clocks to avoid abrupt time drift after detecting a difference. For example, when devices connect to a synchronization server, they can employ mechanisms such as slew adjustment to periodically make small adjustments to their internal clocks, where the clock is gradually corrected over time to synchronize with the server's time.

[0004] This can cause problems when a device temporarily loses its connection to a synchronization server (which could be due to network issues, server downtime, or other interruptions). During this offline period, the sensor's internal clock may begin to deviate from the correct time provided by the server. This drift can occur if the sensor's real-time clock (RTC) malfunctions, causing it to lose accuracy or precision over time. This problem is particularly problematic in applications where precise time alignment is critical, such as video surveillance. For example, in a multi-camera setup, if one or more cameras experience clock drift during the period they are disconnected from the synchronization server, their clocks may continue to show different times even after reconnection.

[0005] Therefore, improvements are needed in this area. Summary of the Invention

[0006] In view of the foregoing, it would be beneficial to address or at least reduce one or more of the disadvantages discussed above, as set forth in the appended independent patent claims.

[0007] According to a first aspect of this disclosure, a method for indicating clock drift of an internal clock included in a sensor device is provided, the method comprising: capturing a first data frame by the sensor device; associating the first data frame with a first time point, the first time point indicating the time point of the internal clock when the first data frame is captured; receiving time synchronization data by the sensor device via a network protocol for clock synchronization; determining a reference time point by the sensor device using the time synchronization data, the reference time point indicating the time point of a reference clock when the first data frame is captured; determining an offset between the first time point and the reference time point by the sensor device; and associating the first data frame with data indicating the offset when the offset exceeds a threshold offset.

[0008] This disclosure offers several advantages in indicating clock drift in the internal clock of a sensor device. For example, using the techniques described herein, when the drift exceeds a predefined threshold (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 6 seconds, etc.), it facilitates recording clock drift information and correlating it with sensor data. This ensures that systems implementing the techniques described herein remain aware of any significant time differences, which can later be used to reconstruct the actual time of data frames, for example, for event detection. Therefore, this recorded drift information can be used when synchronizing sensor data and / or detected events between different sensor devices in which clock drift may differ. By accessing this drift data, operators, analysis systems, and post-processing tools (such as those used by law enforcement or security agencies) can adjust for time differences and align events across multiple sensor devices, thereby ensuring accurate time-based event reconstruction.

[0009] Furthermore, using the techniques described herein, data processing can be improved by avoiding the inclusion of unnecessary drift information when the drift is minimal and below a threshold. If the drift is small and insignificant, embedding the information may not provide any meaningful value, and the system can skip recording it. This selective recording reduces the overhead of both data storage and transmission, thereby utilizing resources more efficiently without sacrificing performance. As a result, the system can operate with reduced data load, thus improving efficiency while still providing the necessary time correction capabilities when needed.

[0010] In the context of this disclosure, the term "sensor device" should be understood as any device capable of capturing and collecting sensor data (data frames) (e.g., any device related to event detection and monitoring). This includes devices such as cameras that capture visual data, microphones that record audio, motion sensors that detect body movement, and environmental sensors that measure conditions such as temperature, humidity, or pressure. Furthermore, lidar and radar systems that collect spatial data by using lasers or radio waves to measure distance or detect objects are also considered sensor devices.

[0011] In the context of this disclosure, the term "data frame" should be understood as a unit of data captured by a sensor device during a specific time period or at a specific point in time. Depending on the sensor, a data frame may consist of different types of information, such as image or video frames from a camera, audio frames from a microphone, a set of spatial measurements from a lidar or radar system, or readings from environmental sensors. Each data frame represents a snapshot of data captured at a specific moment or over a time span.

[0012] In the context of this disclosure, the term "network protocol for clock synchronization" should be understood as a protocol used to synchronize the internal clocks of devices within a network with a reference clock. Examples include protocols such as Network Time Protocol (NTP), Precision Time Protocol (PTP), and Network Time Security (NTS). NTS can be used to ensure secure synchronization by protecting the integrity of time data. These protocols periodically transmit "time synchronization data," which refers to the exchanged information (such as timestamps or other time-related details) that enables devices to correct clock drift and maintain accurate synchronization with a reference clock accessible via the NTP / NTS / PTP protocol.

[0013] In some examples, the sensor device is a webcam, wherein the first data frame is a first image frame, and wherein associating the first image frame with data indicating an offset by the webcam includes adding graphical data that visualizes the data indicating the offset to the overlay associated with the first image frame.

[0014] Advantageously, using overlays to visualize clock drift data can provide immediate, intuitive feedback to users or systems analyzing captured data, allowing them to quickly identify the extent of any time discrepancies. This visual representation can improve the efficiency of monitoring and post-analysis because drift information is directly embedded as an overlay into the image frame. Furthermore, this overlay can be toggled on the operator side, allowing users to enable or disable the display of clock drift information as needed.

[0015] In some examples, associating a first data frame with data indicating an offset by a sensor device includes adding the data indicating the offset as first metadata associated with the first data frame.

[0016] Advantageously, by associating the first data frame with offset data as metadata, client-side applications can be given the flexibility to use the information in any way suited to the specific needs of the application. By embedding clock drift data as metadata, the integrity of the original sensor data frame can be maintained, allowing drift information to be processed independently or in combination with the data frame.

[0017] In some examples, the first metadata associated with the first data frame is provided as at least one of the following: data stream metadata, metadata added to the header of the first data frame, and a separate metadata stream. Data stream metadata is metadata directly embedded within the stream of the data frame (such as a video or audio stream). For example, for a video stream, metadata such as OBU metadata in AV1, Supplemental Enhancement Information (SEI) in the H.26x standard, or metadata of user data in MPEG-2 can be used. For an audio stream, Event Message (EMSG) boxes of the MPEG-DASH audio stream can be used. Metadata can also be provided in the header of the first data frame. In this case, clock drift information is included in the header portion of the data frame. The header is part of the data frame and contains information about the frame itself (such as timing, encoding details, or other attributes). Adding metadata to the header ensures that it is closely related to a specific data frame, allowing the application to easily read this information before processing the actual content of the frame. Metadata can also be provided as a separate metadata stream. This approach is common in systems like ONVIF, a protocol for networked sensor devices that allows metadata to be streamed along with the main data stream, rather than being directly embedded in the sensor data.

[0018] In some examples, the method further includes: determining a signature using first metadata and sensor data from a first data frame; and integrating the signature into a record that includes the first data frame and the first metadata.

[0019] Combining signatures based on both sensor data and metadata can improve data integrity by ensuring that any tampering with the data is detectable. This can be advantageous for applications such as security and forensic analytics. Furthermore, this example can improve traceability by allowing systems to verify the authenticity and origin of data, ensuring it comes from a trusted source at a given time. Such signatures can also support legal compliance and verification, providing tamper-proof records usable in legal settings to confirm the accuracy and integrity of the data.

[0020] In some examples, associating a first data frame with a first time point by a sensor device includes adding the first time point as second metadata associated with the first data frame. Alternatively or additionally, the sensor device is a webcam, where the first data frame is a first image frame, and where associating the first image frame with the first time point by the webcam includes adding graphical data visualizing the first time point to the overlay associated with the first image frame. By providing a first time point (i.e., the time point of the internal clock when the first data frame is captured), time reference for the data frame can be achieved even in the absence of an offset between the internal clock and a reference clock. This ensures that the timestamp is always available, regardless of clock drift (offset) and how the data indicating the offset is represented. Whether the offset is provided as an actual timestamp or as a calculated difference from the first timestamp, these examples facilitate accurate and consistent timing information.

[0021] In some examples, the method further includes: continuously capturing data frames by a sensor device; and for each captured data frame, associating the captured data frame with another time point, which indicates the time point of the internal clock when the captured data frame was captured.

[0022] While continuously capturing data frames at predetermined intervals, the method may include: receiving time synchronization data by a sensor device via a network protocol for clock synchronization; using the time synchronization data by the sensor device to determine another reference time point, which indicates the time point of a reference clock when the latest captured data frame was captured; determining another offset between the other time point associated with the latest captured image frame and the other reference time point; and when the other offset exceeds a threshold offset, associating the latest captured data frame with data indicating the other offset by the sensor device.

[0023] In these examples, embedding information about the current clock drift (offset) into the recorded video is done at regular intervals, such as per group of pictures (GOP) or every 10 seconds. The method continuously captures data frames and checks for drift between the sensor device's internal clock and a reference clock at predetermined intervals via a network protocol. If a significant offset exceeding a predefined threshold is detected between the internal clock and the reference clock, this offset is recorded along with the most recently captured data frame, as explained above. Advantageously, time synchronization data does not need to be continuously exchanged, which reduces the bandwidth required for communication between the sensor device and the reference clock. By performing clock synchronization only at predetermined intervals, network resources are saved, and data load is reduced. Furthermore, the processing burden on the sensor device is reduced because it does not need to constantly calculate and indicate clock drift.

[0024] In some examples, the method includes adjusting the internal clock to reduce skew, wherein the magnitude of the adjustment is limited by the specifications of the network protocol used for clock synchronization. Advantageously, by adjusting the internal clock to reduce skew (where the magnitude of the adjustment is limited by the specifications of the network protocol used for clock synchronization), gradual synchronization of the internal clock to reduce skew is facilitated. This prevents abrupt changes in the internal clock that could cause data inconsistencies or jitter in time-sensitive applications such as video or audio recording. Gradual synchronization ensures smooth time adjustment, reducing the risk of disrupting normal operation of the sensor device or affecting the continuity of data frames. By adhering to the protocol's limitations on clock adjustment, the method also maintains compatibility with industry standards, thereby ensuring reliable and stable performance across a variety of devices and systems.

[0025] In some examples, the method further includes determining the accurate time point at which the first data frame was captured using a first time point and data indicating the offset at a device separate from the sensor device. This allows a separate device (such as a video management system (VMS) or forensic analysis workstation) to reconstruct the true capture time of the data frame even when the sensor device's internal clock is out of sync with a reference clock.

[0026] In some examples, the method further includes, at a device separate from the sensor device, using the following to determine the accurate time point at which the second data frame was captured: a second time point associated with the second data frame, indicating the time point of the sensor device's internal clock when the second data frame was captured; and data indicating an offset associated with the first data frame. Even if not every data frame carries synchronization data, the method can facilitate the determination of accurate timestamps for all frames (i.e., the second frame) based on the offset from the first frame. This can facilitate continuity and precise time alignment across data frames without requiring time synchronization to be determined for each data frame at the sensor device.

[0027] According to a second aspect of the invention, the above objective is achieved by a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed on a camera with processing capabilities, are used to implement the method according to the first aspect.

[0028] According to a third aspect of the invention, the above objective is achieved by a sensor device configured to indicate clock drift of an internal clock included in the sensor device, the sensor device being configured to: capture a first data frame; associate the first data frame with a first time point indicating the time point of the internal clock when the first data frame is captured; receive time synchronization data via a network protocol for clock synchronization; determine a reference time point using the time synchronization data, the reference time point indicating the time point of a reference clock when the first data frame is captured; determine an offset between the first time point and the reference time point; and associate the first data frame with data indicating the offset when the offset exceeds a threshold offset.

[0029] According to a fourth aspect of the invention, the above objective is achieved by a system comprising a sensor device of the third aspect and means separate from the sensor device, the means separate from the sensor device being configured to: determine the accurate time point at which the sensor device captures the first data frame using a first time point and data indicating an offset.

[0030] According to some examples, the device separate from the sensor device is further configured to: determine the accurate time point at which the second data frame is captured by the sensor device using: a second time point associated with the second data frame, which indicates the time point of the internal clock of the sensor device when the second data frame is captured; and data indicating the offset associated with the first data frame.

[0031] The second, third, and fourth aspects may generally have the same features and advantages as the first aspect. It should further be noted that this disclosure relates to all possible combinations of features, unless otherwise expressly stated. Attached Figure Description

[0032] Referring to the accompanying drawings, the above and additional objects, features, and advantages of the present invention will be better understood from the following illustrative and non-limiting detailed description of embodiments of the present disclosure, wherein the same reference numerals will be used for similar elements in the drawings:

[0033] Figure 1 A graph showing clock drift in a sensor device according to an example is shown;

[0034] Figure 2 This illustrates a system configured to indicate clock drift in a sensor device, according to an example.

[0035] Figure 3 A flowchart illustrating a method for indicative clock drift of an internal clock included in a sensor device, according to an embodiment, is shown. Detailed Implementation

[0036] In time-sensitive applications such as video surveillance, event detection, and multi-sensor systems, maintaining accurate time synchronization across devices is crucial. Sensor devices typically rely on internal clocks, which can drift over time relative to a reference clock (such as one provided by a Network Time Protocol (NTP) server). When time synchronization is unavailable, this drift can lead to significant time discrepancies in recorded data, potentially impacting event reconstruction, forensic analysis, and multi-sensor synchronization.

[0037] To address these challenges, this disclosure provides a method and system for managing clock drift in a sensor device by embedding clock drift information in the recorded data. The method involves capturing data frames using the sensor device and associating the data frames with an offset between an internal clock and a reference clock when the drift exceeds a predefined threshold. (See below for details.) Figures 1 to 3 The techniques described herein can be used to indicate clock drift after a period of time when a sensor device has lost connection with a reference clock, thereby allowing accurate reconstruction of event timing even after a period of time when synchronization is temporarily unavailable.

[0038] Figure 1 The diagram 100 illustrates the clock drift in the sensor device, where the vertical axis 112 represents the clock offset or drift in seconds (both positive and negative), and the horizontal axis represents the time 104 of the reference clock (which is obviously perfectly synchronized in time and has no offset). The dashed line 102 indicates the time of the sensor device's internal clock, which experiences drift relative to the reference clock's time 104 over time (as indicated by the vertical axis 112). At any given time, the vertical distance between the internal clock's time 102 and the reference clock's time 104 represents the clock drift or offset at that moment. This offset can be positive or negative, depending on whether the internal clock is ahead or behind the reference clock. Therefore, the term "offset" encompasses both cases where the internal clock is faster or slower than the reference clock.

[0039] Figure 1 The diagram also illustrates data frames 140 captured by the sensor device during the time period represented by the graph. The alignment of each data frame 140 with the horizontal axis indicates the time of a reference clock at the moment each data frame 140 was captured.

[0040] On the left side of the graph, the sensor device is connected to the reference clock via a network protocol (NTP, NTS, PTP, etc.) for clock synchronization. In this way, the internal clock time 102 is substantially synchronized with the reference clock time 104.

[0041] During the time period depicted in the graph, the sensor device captures data frames 140, and associates each frame with the time 102 of the internal clock. As long as the time 102 of the internal clock is substantially synchronized with the time 104 of the reference clock, the time 102 of the internal clock is sufficient for downstream analysis (such as event reconstruction, forensic analysis, multi-sensor synchronization, etc.).

[0042] At a certain point in time 106, a disconnection occurs between the sensor device and the reference clock. This could happen if the NTP server goes offline or if the sensor device loses its connection to the NTP server for other reasons, such as a Wi-Fi outage or network instability. This causes the sensor device to operate independently of the network protocol used for clock synchronization. As a result, the internal clock time 102 begins to drift from the reference clock time 104 over time. Because the sensor device is not connected to the NTP server, it does not perceive the drift and thus maintains association between each frame and the internal clock time 102.

[0043] At a later time point 108, the reference clock becomes available to the sensor device again. Therefore, the sensor device receives time synchronization data via a network protocol used for clock synchronization. The time synchronization data may include a timestamp indicating the time of the reference clock, delay information considering network latency, a clock identifier ensuring synchronization with the correct reference source, etc. Using the time synchronization data, the sensor device can then determine a reference time point 114, indicating the time of the reference clock when the current / most recent data frame 142 is captured. Then, when the current / most recent data frame 142 is captured, the reference time point 114 can be compared with the (first) time point 116 indicated by the internal clock 102. Using both timestamps 114 and 116, the sensor device can determine the offset between the first time point 116 and the reference time point 114. If this offset exceeds a threshold offset 110, the current / most recent data frame 142 is associated with data indicating the magnitude and sign of the offset, regardless of whether the offset is positive (indicating the internal clock is ahead) or negative (indicating the internal clock is behind). In other words, when the offset exceeds a predefined threshold, whether positive or negative, the system begins to associate the captured data frames with metadata indicating the magnitude and sign or direction of the drift.

[0044] Threshold offsets can be 5 seconds, 1 second, or even less. Even such small time offsets can be detrimental to time-sensitive applications, such as those used in law enforcement investigations, where precise timing is essential for accurate event correlation, decision-making, and synchronization across multiple devices. For example, in law enforcement, even minute time differences can lead to inconsistent evidence, impair the ability to accurately reconstruct events, and potentially affect the outcome of investigations or legal proceedings.

[0045] Threshold-based association ensures that only significant (environment-dependent) drifts are recorded, thereby reducing unnecessary data overhead when the offset is minimal.

[0046] When reconnected to the reference clock, the internal clock's time 102 is adjusted (in this case, the clock frequency is increased) to reduce the offset (e.g., a cyclic adjustment), where the magnitude of the adjustment is limited by the specifications of the network protocol used for clock synchronization. The magnitude of the adjustment can be intentionally small (e.g., 0.8 ms per second) to ensure that synchronization occurs gradually, thus avoiding sudden time jumps that could interrupt ongoing operations such as video recording or data analysis. This gradual adjustment helps maintain the continuity of the data stream and prevents interruptions or inconsistencies that could be caused by sudden clock corrections, thus ensuring smooth performance for time-sensitive applications. However, this small adjustment means that even a tiny offset caused by a break in the connection to the reference clock can take a considerable amount of time to fully correct. For example, at a rate of 0.8 ms per second, it would take more than 6000 seconds (or approximately 1 hour and 40 minutes) to fully correct a 5-second clock drift.

[0047] The sensor device can continuously capture data frames 140. For each frame (such as data frames 142, 144, and 146), the sensor device can associate the captured frame with a corresponding time point based on the time 102 of its internal clock at the capture time. However, the actual determination of the clock offset may only occur at predetermined intervals, rather than for each data frame 140. During these intervals (in... Figure 1At each data frame 140 (corresponding to the previous one), as described above, the sensor device can receive time synchronization data via a network protocol, allowing it to identify clock drift. For example, the sensor device can use the time synchronization data to determine another reference time point 118, indicating the time of the reference clock when the latest captured data frame 146 was captured. The sensor device can also associate the same captured data frame 146 with another time point 120, indicating the time of the internal clock when the captured data frame 146 was captured. Using these two timestamps, the sensor device can determine another offset between the other time point 120 associated with the latest captured image frame 146 and the other reference time point 118. As described above, when this other offset exceeds a threshold offset 110, the sensor device can associate the latest captured data frame 146 with data indicating this other offset. However, it should be noted that the offset associated with the captured data frame 146 (possibly in combination with the offset associated with the previous data frame 142) can be used to determine the offset of the data frame 144 preceding data frame 146. This means that even if not every data frame 140 has a directly measurable offset, the downstream analysis device can interpolate or calculate the offset of the intermediate frame 144 based on known offsets from one or more adjacent frames 146, 142. By doing so, the technique described herein ensures accurate time correction across all frames 140 without requiring the recording of synchronization data for each individual frame.

[0048] Figure 2 A system 200 configured to indicate clock drift in sensor device 204 is shown. The functions and techniques implemented in sensor device 204 will now be described in conjunction with a flowchart of a method 300 for indicating clock drift of an internal clock 208 included in sensor device 204.

[0049] exist Figure 2 In the example, sensor device 204 is a webcam that captures scene 202, but in other examples, sensor device 204 may be a microphone, radar, environmental sensor, etc.

[0050] Sensor device 204 is configured to capture S302 data frames. Figure 2 In the example, sensor device 204 captures image frames depicting scene 202 in S302. In other embodiments, depending on the capabilities of sensor device 204, the sensor device may capture audio frames, lidar / radar frames, or any other suitable type of data frames.

[0051] Sensor device 204 includes an internal clock 208. The sensor device, for example, implements a function in timestamp processing unit 205 to associate a captured data frame (hereinafter referred to as the first data frame) with a first time point S304, which indicates the time point of the internal clock 208 when the first data frame was captured. Timestamp processing unit 205 may process this process, for example, by reading the internal clock 208 when the first data frame is captured, and optionally embed the timestamp as metadata or as an overlay associated with the first data frame, thereby ensuring that each data frame has a clear reference to the time of capture according to the internal clock 208.

[0052] Sensor device 204 is connected to reference clock 214 via network protocol 212 for clock synchronization. Examples of network protocol 212 include NTS, NTP, and PTP as previously discussed. The connection can be wired or wireless. For example, a wired connection can utilize Ethernet or fiber optics, while wireless options can include technologies such as Wi-Fi, 3G, 4G, or 5G. Sensor device receives time synchronization data 210 in S306 via network protocol 212 for clock synchronization.

[0053] Sensor device 204 (e.g., timestamp processing unit 205) can be configured to use time synchronization data 210 to determine a reference time point for S308, which indicates the time point of reference clock 214 when the first data frame is captured.

[0054] The sensor device 204 (e.g., timestamp processing unit 205) may be further configured to determine the offset between the first time point of S310 and the reference time point.

[0055] For example, the timestamp processing unit can be configured to calculate the difference between a first time point and a time point of reference clock 214 determined from the time synchronization data. It is important to note that the time synchronization data 210 may not arrive at exactly the same moment as when the first data frame was captured. To handle this, the timestamp processing unit 205 can be designed to consider the timing difference between the time the first data frame was captured and the time the synchronization data was received, thereby ensuring accurate time alignment. Furthermore, the timestamp processing unit 205 can be configured to consider any delay information included in the time synchronization data 210 to account for network latency. By taking this latency into account, the timestamp processing unit 205 can perform the necessary calculations to determine the reference time point, thereby ensuring that network latency or arrival time does not distort the accuracy of synchronization. Once the reference time point is determined, the timestamp processing unit 205 can proceed to determine the offset between the first time point of S310 (from the internal clock 208) and the reference time point (from the reference clock 214).

[0056] Sensor devices (e.g., timestamp processing unit 205) can be configured to adjust internal clock 208 to reduce skew, wherein the magnitude of the adjustment is limited by the specifications of the network protocol for clock synchronization as described above.

[0057] Such as combination Figure 1 The discussed sensor device (e.g., timestamp processing unit 205) can be configured to compare the offset with a threshold offset. If the offset exceeds the threshold offset, the timestamp processing unit 205 can be configured to associate a first data frame with data indicating the offset S312. The offset data can represent a signed actual offset, or it can represent an actual timestamp of a reference clock 214 used to calculate the magnitude of the offset, as determined. The signed actual offset indicates the deviation of the sensor's internal clock 208 from the reference clock 214 (having a positive or negative sign depending on whether the sensor's internal clock 208 is leading or lagging).

[0058] When the first data frame is a first image frame, the sensor device 204 (e.g., the timestamp processing unit 205) can be configured to add graphical data that visualizes the data indicating the offset to the overlay associated with the first image frame, S314. Alternatively or additionally, and regardless of the format of the first data frame, the timestamp processing unit 205 can be configured to associate the first data frame with the data indicating the offset, S312, by adding the data indicating the offset as first metadata associated with the first data frame, S316.

[0059] In some embodiments, the first metadata may be stored on the sensor device 204, having a reference (e.g., an ID number, hash code, or similar identifier) ​​that links it to an associated data frame or is directly embedded within the data frame itself. The stored metadata can be retrieved from the sensor device 204 when needed (such as when sensor data is required for legal or law enforcement purposes, or in other scenarios where verifying the accurate capture time of the data frame is critical).

[0060] In some embodiments, the first metadata associated with the first data frame may be provided as data stream metadata in data stream 216 transmitted from sensor device 204. Depending on the codec used for encoding, this data stream metadata may include OBU metadata as defined in the AV1 codec, registered or unregistered Supplemental Enhancement Information (SEI) as specified in the H.26x standard, user data in MPEG-2, VP9 metadata, or any other suitable video stream metadata format. Furthermore, if the data frame is an audio frame (i.e., audio data), the metadata may be provided in an audio-specific data stream format, such as in-band metadata supported by a streaming protocol like the Extensible Metadata Platform (XMP). The choice of metadata format may vary based on the implementation of the data frame structure used, the codec, or specific requirements.

[0061] In some embodiments, the first metadata associated with the first data frame can be provided as metadata added to the header of the first data frame by directly embedding the necessary information (such as timestamps, offsets, etc.) into the header structure of the data frame. This can be achieved using standard encoding techniques supported by the specific media format or protocol used (e.g., adding metadata to the header portion of video frames using formats like MPEG or H.264).

[0062] In some embodiments, the first metadata associated with the first data frame can be provided as a separate metadata stream 226, such as using the ONVIF (Open Network Video Interface Forum) standard. This means that the sensor device 204 can transmit metadata to the data frame stream 216 in a parallel stream 226, rather than embedding the metadata directly within the data frame itself (e.g., in the header). ONVIF, typically used in IP-based video surveillance systems, allows metadata to be transmitted separately from the video or sensor data stream 216, thus providing flexibility for handling synchronization, timestamps, or event marking.

[0063] In some examples, sensor device 204 can implement one or both of two methods: either adding the time point of the internal clock as metadata associated with the first data frame when it is captured, or, in the case that sensor device 204 is a network camera, visualizing the time point of the internal clock when the first data frame is captured as graphical data overlaid on the first image frame. For example, in the first method, the timestamp of the internal clock 208 can be embedded as metadata in the frame header, just like offset data, allowing downstream systems or applications to extract and use the time information for synchronization and analysis. Similarly, in the second method, the timestamp of the internal clock 208 can be visually represented as graphical data overlaid on the image frame itself, making it immediately visible to the operator or analyst without requiring a custom application.

[0064] In some embodiments, the captured data frame and its associated metadata can be signed by sensor device 204 (e.g., using data signing component 206). This signing process may involve generating a digital signature that verifies the authenticity and integrity of both the data frame and its associated metadata (such as timestamps or offset information). By applying an encrypted signature, the system ensures that any tampering or alteration to the data after its capture can be detected. This is particularly important in applications such as surveillance, law enforcement, or security data collection, where the integrity of timestamped data is crucial for legal or analytical purposes. The signature can be embedded in the metadata or transmitted as part of a separate data stream, providing a secure means of verifying the authenticity of the data during transmission or storage. For example, data signing component 206 can be configured to determine the S318 signature using the first metadata and sensor data from the first data frame; and to integrate that signature into a record that includes the first data frame and the first metadata (which can be transmitted 216 or stored). Techniques used to determine the S318 signature using the first metadata and sensor data from the first data frame typically involve cryptographic hashing or signature algorithms. Sensor data and metadata can be combined and processed, for example, using a secure hash algorithm such as SHA-256, to generate a unique hash representing the combined data. This hash can then be signed using a private key with a digital signature algorithm (e.g., RSA or ECDSA), ensuring that any tampering with the data or metadata will invalidate the signature.

[0065] In some embodiments, the metadata may also be separately signed (using the techniques described above) and uploaded to a different server or database, rather than being included directly in media record 216.

[0066] In some examples, if a protocol such as Network Time Security (NTS) is used to provide time synchronization data 210, the timestamps in the time synchronization data 210 can be signed by that protocol. By using NTS to sign the timestamps and time offsets, sensor device 204 provides an additional layer of security, thereby verifying not only the integrity of the offset data determined at sensor device 204, but also the integrity of the time synchronization data 210 used by sensor device 204 to determine the offset.

[0067] System 200 may further include a separate device 218 from sensor device 204, which may receive data frames and associated offset data from sensor device 204 via streams 216, 226 or as part of a log file. For example, such a device may be a video management system (VMS), a data analytics server, or a cloud-based processing unit. These devices may use the received data 216, 226 for further processing and analysis (such as time alignment across multiple sensor feeds, forensic analysis, or event reconstruction). The separate device 218 may also be responsible for verifying the authenticity of the received data by checking embedded digital signatures, adjusting clock drift using offset metadata, and performing any additional post-processing tasks (such as generating alarms).

[0068] The separate device 218 may, for example, include a display for displaying the data frame 220. This display may further display data indicating the offset and / or data indicating the timestamp from the internal clock 208. For example, as described above, the display may use overlays 222 and / or overlays 224 associated with the first image frame to display these data points. Alternatively, the separate device 218 may be implemented to implement custom applications that display this data using metadata associated with the first data frame. In any event, the separate device 218 may be configured to use a first time point (i.e., the timestamp from the internal clock 208) and data indicating the offset to determine the exact time point at which the first data frame was captured. (As in combination...) Figure 1 The individual device 218 discussed can be configured to use a second time point associated with the second data frame and data indicating an offset associated with the first data frame to determine the exact time point at which the second data frame was captured, the second time point indicating the time point of the sensor device's internal clock when the second data frame was captured.

[0069] It should be noted that, as mentioned above, Figure 2 The described functional divisions (such as data signature component 206 and timestamp processing unit 205) are provided by way of example only. Any suitable division or architectural arrangement of the functions explained herein may be adopted without departing from the scope of the invention. Depending on the specific system architecture, hardware capabilities, or application requirements, the described components and their functions may be combined, distributed across multiple units, or implemented in various ways.

[0070] The above-described techniques can be implemented on a non-transitory computer-readable storage medium having instructions stored thereon for performing the methods discussed herein. When these instructions are executed on one or more devices with processing capabilities, such as a general-purpose processor (CPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other dedicated processing unit, methods for instructing time synchronization, clock drift correction, metadata association, and data integrity verification are performed.

[0071] The above embodiments should be understood as illustrative examples of the invention. Further embodiments of this disclosure are conceivable. It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment, or in any combination of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of this disclosure as defined in the appended claims.

Claims

1. A method for indicating clock drift of an internal clock included in a sensor device, the method comprising: The first data frame is captured by the sensor device; The sensor device associates the first data frame with a first time point, the first time point indicating the time of the internal clock when the first data frame is captured; The sensor device receives time synchronization data via a network protocol for clock synchronization; The sensor device uses the time synchronization data to determine a reference time point, which indicates the time point of a reference clock when the first data frame is captured. The offset between the first time point and the reference time point is determined by the sensor device; as well as When the offset exceeds a threshold offset, the sensor device associates the first data frame with data indicating the offset.

2. The method according to claim 1, wherein, The sensor device is a network camera, wherein the first data frame is a first image frame, and wherein associating the first image frame with data indicating the offset by the network camera includes: Graphical data that visualizes the data indicating the offset is added to the overlay associated with the first image frame.

3. The method according to claim 1, wherein, Associating the first data frame with data indicating the offset by the sensor device includes adding the data indicating the offset as first metadata associated with the first data frame.

4. The method according to claim 3, wherein, The first metadata associated with the first data frame is provided as at least one of the following: Data stream metadata, metadata added to the header of the first data frame, and a separate metadata stream.

5. The method of claim 3, further comprising: The signature is determined using the first metadata and the sensor data from the first data frame; as well as The signature is integrated into a record that includes the first data frame and the first metadata.

6. The method according to claim 1, wherein, Associating the first data frame with the first time point by the sensor device includes: Add the first time point as second metadata associated with the first data frame.

7. The method according to claim 1, wherein, The sensor device is a network camera, wherein the first data frame is a first image frame, and wherein associating the first image frame with the first time point by the network camera includes: Graphical data that visualizes the first time point is added to the overlay associated with the first image frame.

8. The method of claim 1, further comprising: Data frames are continuously captured by the sensor device; For each captured data frame, the sensor device associates the captured data frame with another time point, which indicates the time of the internal clock when the captured data frame was captured; While continuously capturing the data frames at predetermined intervals: The sensor device receives time synchronization data via the network protocol for clock synchronization; The sensor device uses the time synchronization data to determine another reference time point, which indicates the time point of the reference clock when the latest captured data frame was captured. Determine another offset between the other time point associated with the latest captured image frame and the other reference time point; When the other offset exceeds the threshold offset, the sensor device associates the latest captured data frame with the data indicating the other offset.

9. The method of claim 1, further comprising: The internal clock is adjusted to reduce the offset, wherein the magnitude of the adjustment is limited by the specifications of the network protocol used for clock synchronization.

10. The method of claim 1, further comprising: At a device separate from the sensor device, the first time point and the data indicating the offset are used to determine the exact time point at which the first data frame was captured.

11. The method of claim 10, further comprising: At the device detached from the sensor device, the following are used to determine the exact time point at which the second data frame was captured: A second time point associated with the second data frame, the second time point indicating the time point of the internal clock of the sensor device when the second data frame was captured; as well as The data indicating the offset associated with the first data frame.

12. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed on one or more means having processing capabilities, are used to implement the method according to any one of claims 1 to 11.

13. A sensor device configured to indicate clock drift of an internal clock included in the sensor device, the sensor device being configured to: Capture the first data frame; Associate the first data frame with a first time point, the first time point indicating the time point of the internal clock when the first data frame is captured; Receive time synchronization data via a network protocol used for clock synchronization; The time synchronization data is used to determine a reference time point, which indicates the time point of a reference clock when the first data frame is captured. Determine the offset between the first time point and the reference time point; as well as When the offset exceeds a threshold offset, the first data frame is associated with data indicating the offset.

14. A system comprising a sensor device as claimed in claim 13 and means separate from said sensor device, said means separate from said sensor device being configured to: The first time point and the data indicating the offset are used to determine the exact time point at which the sensor device captured the first data frame.

15. The system according to claim 14, wherein, The device, separate from the sensor device, is further configured to: The following items are used to determine the exact time point at which the second data frame was captured by the sensor device: A second time point associated with the second data frame, the second time point indicating the time point of the internal clock of the sensor device when the second data frame was captured; as well as The data indicating the offset associated with the first data frame.

Images