Camera device timestamp verification

Abstract

A method for determining image capture timestamp offset or error includes generating an optical flash at an optical flash rate. Capturing a set of images of the optical flash at an image capture rate. The image capture rate is different from the optical flash rate, and each image includes an associated image timestamp. A signal associated with the generation of the optical flash is also timestamped. An intensity of each image in the set of images is determined, and the image in the set having the greatest intensity is identified. The timestamp offset or error is determined as the difference between the timestamp of the image having the greatest intensity and the timestamp of the corresponding optical flash.

Description

[0001] Related application data

[0002] This application claims priority to U.S. patent application serial number 17 / 845,478, filed on June 21, 2022. Technical Field

[0003] The present disclosure relates generally to image capture devices and, more particularly, but not exclusively, to image capture devices for augmented reality. Background Art

[0004] Augmented reality (AR) devices allow a user to view a surrounding environment (the "real world") and also to see objects generated for display that appear to be part of and / or superimposed on the surrounding environment (e.g., virtual objects such as 3D renderings, images, videos, text, etc.). In some examples, the AR device takes the form of glasses with a display through which the user can view the real world and on which virtual objects can be displayed. In other examples, the AR device may take the form of a smartphone or tablet. In such cases, a video feed of the real world captured by a camera in the smartphone or tablet is displayed on the smartphone or tablet's display, with AR objects or effects included in or applied to the real-world video feed.

[0005] An AR device relies on accurate timing information from one or more associated cameras to ensure accurate tracking of the AR device as it moves, and to ensure correct placement of AR objects and effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head-mounted device according to some examples.

[0008] Figure 2 is shown according to some examples including Figure 1 Block diagram of networking system 200 with details of a head mounted device.

[0009] Figure 3 is a diagram illustrating a system for performing camera timestamp verification or camera timestamp shifting according to some examples.

[0010] Figure 4 is shown according to some examples Figure 3 A diagram of the sequence of events in the operation of a system.

[0011] Figure 5 yes Figure 4 A magnified version of the region of maximum mean intensity is shown.

[0012] Figure 6 is a graph showing sequential average intensity values for images captured by a camera according to some examples.

[0013] Figure 7 is a flow chart illustrating a method of determining an error or offset in a camera timestamp according to some examples.

[0014] Figure 8 is a flow chart illustrating a method of determining an error or offset in a camera timestamp, according to some examples.

[0015] Figure 9 is a diagram illustrating alternative methods of determining camera timestamp offsets according to some examples.

[0016] Figure 10 is a flow chart illustrating a method of determining an error or offset in a camera timestamp, according to some examples.

[0017] Figure 11 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0018] Figure 12 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to some examples.

[0019] Figure 13 is a diagrammatic representation of a machine in the form of a computer system, according to some examples, within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0020] Some head-mounted AR devices, such as AR glasses, include a transparent or translucent display, which enables the user to see through the transparent or translucent display to view the surrounding environment. Additional information or objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) are displayed on the display and appear as part of the surrounding environment and / or superimposed on the surrounding environment to provide the user with an augmented reality (AR) experience. The display may, for example, include a waveguide that receives a light beam from a projector, but any suitable display for presenting augmented content or virtual content to the wearer may be used.

[0021] The correct positioning of virtual objects and other AR effects, and the synchronization of their movement or updating with what is happening in the real world, depends on accurate timing information received from the camera used to capture images or video feeds of the real world. The captured images are typically used by the AR device to represent the real world, such as device positioning and tracking using image processing techniques. The accuracy of the timing of image capture is relative to other events occurring in the AR device, so that two different events occurring at the same moment will have the same timestamp assigned by the AR device.

[0022] To determine a shift or error in the timestamps of images captured by a camera, a set of time-stamped images of an optical flash is captured at an image capture rate. The image capture rate is different from the optical flash rate. A signal associated with the generation of the optical flash is also time-stamped. The intensity of each image in the set of images is determined, and the image with the maximum intensity in the set of images is identified. The camera's timestamp shift or error is determined as the difference between the timestamp of the actual or theoretical image with the maximum intensity and the timestamp of the corresponding optical flash.

[0023] The timestamp offset can then be used to adjust the camera timestamps so that the image timestamps more accurately align with other timestamps generated by the AR device. This provides improved alignment and synchronization of visual content with real-world objects.

[0024] In some examples, a computer-implemented method of determining an image capture timestamp offset is provided, comprising: generating a plurality of optical flashes at an optical flash rate, capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different from the optical flash rate, and each image of the optical flashes including an associated image timestamp, timestamping a signal associated with the generation of the plurality of optical flashes, determining an intensity of each image in the set of images, determining an image in the set of images having a maximum intensity, and determining a difference between the timestamp of the image having the maximum intensity and the timestamp of the corresponding optical flash to generate an image capture timestamp offset. The optical flash duration may be approximately equal to the image exposure duration.

[0025] The method may further include determining a first line based on the intensity and timestamp of an image in the set of images that precedes the image having the maximum intensity in the set of images, determining a second line based on the intensity and timestamp of an image in the set of images that follows the image having the maximum intensity in the set of images, determining an intersection of the first line and the second line, and using the time value of the intersection of the first line and the second line as the timestamp of the image having the maximum intensity when determining the difference between the timestamp of the image having the maximum intensity and the timestamp of the corresponding optical flash.

[0026] A signal associated with the generation of the optical flash may be received at a port of a computer processor associated with a camera that captured a set of images with the optical flash, the computer processor time-stamping receipt of the signal associated with the generation of the optical flash.

[0027] The method may include setting a timer having a duration sufficient to ensure that an image having maximum intensity is captured, and capturing a set of images of the optical flash during the duration of the timer. The image intensity of each image may be an average image intensity.

[0028] The method may include adjusting a camera timestamping parameter based on the image capture timestamp offset.

[0029] In some examples, a non-transitory computer-readable storage medium is provided that includes instructions that, when executed by a computer, cause the computer to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate according to any of the above methods and limitations, the operations including but not limited to the following operations: generating multiple optical flashes at the optical flash rate, capturing a set of images of the multiple optical flashes at an image capture rate, the image capture rate being different from the optical flash rate, and each image of the optical flash including an associated image timestamp, timestamping a signal associated with the generation of the multiple optical flashes, determining an intensity of each image in the set of images, determining an image with a maximum intensity in the set of images, and determining a difference between the timestamp of the image with the maximum intensity and the timestamp of the corresponding optical flash to generate an image capture timestamp offset.

[0030] In some examples, a computing device is provided that includes a processor and a memory storing instructions that, when executed by the processor, configure the device to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate according to any of the above methods and limitations, the operations including, but not limited to, generating multiple optical flashes at the optical flash rate, capturing a set of images of the multiple optical flashes at an image capture rate, the image capture rate being different from the optical flash rate, and each image of the optical flash including an associated image timestamp, timestamping a signal associated with the generation of the multiple optical flashes, determining an intensity of each image in the set of images, determining an image with a maximum intensity in the set of images, and determining a difference between the timestamp of the image with the maximum intensity and the timestamp of the corresponding optical flash to generate an image capture timestamp offset.

[0031] Figure 11 is a perspective view of a head-mounted AR device (e.g., glasses 100) according to some examples. Glasses 100 can include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112.

[0032] A first or left optical element 108 and a second or right optical element 110 can be disposed within the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 can be lenses, displays, display components, or combinations thereof. Any suitable display component can be disposed within the eyeglasses 100.

[0033] Frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, frame 102 can be formed from a single piece of material to have a unitary or integrated construction.

[0034] The glasses 100 can include a computing device such as a computer 120, which can be of any suitable type to be carried by the frame 102 and, in one or more examples, can be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or 124. The computer 120 can include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes low-power circuitry, high-speed circuitry, and a display processor. Various other examples can include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 can be implemented as shown in the data processor 202 discussed below.

[0035] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter, or transceiver (not shown), or a combination of such devices.

[0036] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single camera or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, and the like.

[0037] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from a real-world scene.

[0038] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left and right temple pieces 122, 124. The touchpad 126 is generally vertically aligned, in some examples approximately parallel to the user's temple. As used herein, generally vertically aligned means the touchpad is more vertical than horizontal, although potentially more vertical than depicted. Additional user input may be provided via one or more buttons 128, which, in the example shown, are located on the outer upper edges of the left and right optic holders 104, 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from the user of the glasses 100.

[0039] In use, information, content, and various 3D user interfaces will be presented to the user of the glasses 100 on the near-eye displays in the left optical element 108 and / or the right optical element 110. The user can then use the touchpad 126 and / or buttons 128, associated devices (e.g., Figure 2 The user device 228 shown in FIG. 2 may be used to interact with the glasses 100 through voice input or touch input on the user device 228 and / or hand movements, position, and location detected by the glasses 100.

[0040] Figure 22 is a block diagram illustrating a networked system 200 including details of the glasses 100 according to some examples. The networked system 200 includes the glasses 100, a user device 228, and a server system 232. The user device 228 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 236 and / or a high-speed wireless connection 234. The user device 228 is connected to the server system 232 via a network 230. The network 230 may include any combination of wired and wireless connections. The server system 232 may be one or more computing devices that are part of a service or network computing system. The server system 232 may be a server system that is connected to the glasses 100 using a network 230. Figure 12 and Figure 13 The details of the software architecture 1204 or machine 1300 described in the embodiment of the present invention implement the user device 228 and any elements of the server system 232 and network 230.

[0041] The glasses 100 include a data processor 202, a display 210, one or more cameras 208, and additional input / output elements 216. The input / output elements 216 may include a microphone, an audio speaker, a biometric sensor, additional sensors, or additional display elements integrated with the data processor 202. Figure 12 and Figure 13 Examples of input / output elements 216 are further discussed. For example, input / output elements 216 can include any I / O components 1306, including output components 1328, motion components 1336, etc. In the particular example described herein, display 210 includes a display for each of the user's left and right eyes.

[0042] Data processor 202 includes image processor 206 (eg, video processor), GPU and display driver 238, tracking module 240, interface 212, low power circuitry 204, and high speed circuitry 220. The components of data processor 202 are interconnected by bus 242.

[0043] Interface 212 refers to any source of user commands provided to data processor 202. In one or more examples, interface 212 is a physical button that, when pressed, transmits a user input signal from interface 212 to low-power processor 214. Low-power processor 214 can process pressing such a button and then immediately releasing it as a request to capture a single image, or vice versa. Low-power processor 214 can process pressing such a button for a first period of time as a request to capture video data while the button is pressed and to stop video capture when the button is released, wherein the video captured while the button is pressed is stored as a single video file. Alternatively, pressing the button for a longer period of time can capture a still image. In some examples, interface 212 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera 208. In other examples, interface 212 can have a software component or be associated with a command received wirelessly from another source, such as from user device 228.

[0044] Image processor 206 includes circuitry for receiving signals from camera 208 and processing those signals from camera 208 into a format suitable for storage in memory 224 or transmission to user device 228. In one or more examples, image processor 206 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 208, and volatile memory used by the microprocessor during operation.

[0045] The low power circuitry 204 includes a low power processor 214 and a low power wireless circuitry 218. These elements of the low power circuitry 204 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 214 includes logic for managing the other elements of the glasses 100. As described above, for example, the low power processor 214 can accept user input signals from the interface 212. The low power processor 214 can also be configured to receive input signals or command communications from the user device 228 via the low power wireless connection 236. The low power wireless circuitry 218 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 218. In other examples, other low power communication systems may be used.

[0046] High-speed circuitry 220 includes a high-speed processor 222, memory 224, and high-speed wireless circuitry 226. High-speed processor 222 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 202. High-speed processor 222 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 234 using high-speed wireless circuitry 226. In some examples, high-speed processor 222 executes an operating system such as the LINUX operating system or a program such as the UNIX operating system. Figure 12 The high-speed processor 222, which executes the software architecture of the data processor 202, manages data transmission with the high-speed wireless circuit system 226, in addition to any other responsibilities. In some examples, the high-speed wireless circuit system 226 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 226 can implement other high-speed communication standards.

[0047] Memory 224 includes any storage device capable of storing camera data generated by camera 208 and image processor 206. Although memory 224 is shown as being integrated with high-speed circuitry 220, in other examples, memory 224 may be a separate, independent element of data processor 202. In some such examples, electrical wiring may provide a connection from image processor 206 or low-power processor 214 to memory 224 through a chip that includes high-speed processor 222. In other examples, high-speed processor 222 may manage addressing of memory 224 so that low-power processor 214 will direct high-speed processor 222 whenever a read or write operation involving memory 224 is desired.

[0048] The tracking module 240 estimates the position and orientation ("pose") of the glasses 100. For example, the tracking module 240 uses image data and corresponding inertial data from the camera 208 and the positioning component 1340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world environment). The tracking module 240 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 240 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 210.

[0049] The GPU and display driver 238 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 210 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 238 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world environment.

[0050] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the user device 228 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1206, such as the messaging application 1246.

[0051] Figure 3 1 is a schematic diagram illustrating a system 300 for performing camera timestamp verification or camera timestamp shifting according to some examples. The system includes a microcontroller 302, a transistor 304 (such as a metal oxide semiconductor field effect transistor), a light emitting diode (LED 306), and a camera 208 and a high-speed circuit system 220 that form part of an AR device such as glasses 100 or a user device 228. Figure 3 It is a conceptual diagram that is not intended to illustrate a literal circuit.

[0052] Microcontroller 302 provides an on / off signal that supplies power to / from LED 306 via transistor 304. The on / off signal is simultaneously provided to general purpose input / output port 308 in high-speed circuitry 220, where its "on" arrival is time-stamped to give an accurate measurement of the activation time of LED 306. Frames from camera 208, time-stamped by camera 208, are provided to high-speed circuitry 220 via camera serial interface 310. The timestamp is generated by the processor that processes the camera image.

[0053] The high-speed circuit system 220 includes an application having program instructions that, when executed, captures a timestamp generated when a signal from the microcontroller 302 arrives at the port 308 and captures a corresponding image of the flash of the LED 306 using the camera 208. The image of the flash will include the camera timestamp. The sequence of images and timestamps will be captured and saved to the memory 224. Additional program instructions may be included in the high-speed circuit system 220 to analyze the set of images and data in whole or in part, or the set of images and data may be uploaded for analysis locally (such as on the user device 228 or another local computing device) or remotely for analysis at the server system 232. The program instructions will perform the following with reference to Figure 7Described method.

[0054] As described below, microcontroller 302 also includes program instructions that instruct microcontroller 302 to provide a periodic signal to flash LED 306. The period of the signal is adjustable.

[0055] In use, the camera 208 is set to a fixed exposure, and the LED 306 is flashed by the microcontroller 302 for a duration approximately equal to the image exposure time, but at a frequency slightly different from the camera frame rate. When the flashing of the LED 306 is exactly in phase with the camera exposure, the average intensity of the image captured by the camera 208 will be at its maximum, and the timestamp of the corresponding image and the timestamp generated when the signal is received at the port 308 should be the same. By capturing the image stream and identifying the brightest image, and comparing the LED timestamp generated by the general purpose input / output (GPIO) with the camera timestamp, any error in the camera timestamp can be determined, as discussed in more detail below.

[0056] For simplicity of implementation, the intensity of a single global optical flash is measured over the complete image. A more general approach could direct any number of optical flashes (possibly with known temporal extension) to specific areas on the image sensor surface to achieve the same result and / or increase the robustness of the measurement process. The optical flash could also come from a laser diode or other light source.

[0057] Whether the image data captured by the camera 208 is stored as a video or as a single image is a matter of encoding. The camera 208 can generate an image stream (output images at a constant frequency) or provide repeated on-demand image capture. Either method can be used in the methods described herein, but image streaming is more convenient.

[0058] The most information is obtained from the image data if the exposure time is half the duration of the frame period (the inverse of the frame rate). If the exposure time is greater than half the frame period, the peaks of the average intensity plot are flattened, and if the exposure time is less than half the frame period, the troughs of the average intensity plot are flattened. To obtain optimal data, the exposure is preferably balanced with the gain on the image sensor to ensure that the maximum average intensity is as bright as possible while avoiding oversaturated pixels. The exposure duration should not result in oversaturation, but should preferably be close to half the frame rate. Such optimization improves the quality of the data, but if enough data is captured, any errors due to deviations from ideal conditions should be eliminated.

[0059] All other things being equal, a faster frame rate will capture all the required data faster, but in practice this isn’t a significant issue. The frame rate also depends on the frame rate the camera is capable of at the requested resolution, which for this article doesn’t need to be high resolution.

[0060] Figure 4 It shows Figure 3 4 is a diagram of a sequence of events in the operation of the system 300. The diagram shows the passage of time 402 from left to right in the drawing, with associated LED flashes (corresponding to the LED "on" state), exposures, time stamps, etc.

[0061] As can be seen in the figure, camera exposure 404 and "on" LED state 406 have approximately the same duration, but the start of each LED "on" state occurs at a frequency slightly less than the camera exposure frequency. Thus, LED states 406 initially begin slightly after the start of the corresponding camera exposure (LED start 408 and exposure start 410), progress to coincide with the start of the corresponding camera exposure (LED start 412 and exposure start 414), and progress to before the start of the camera exposure (LED start 416 and exposure start 418).

[0062] The amount of overlap 420 between the two states corresponds to the time that camera 208 captures the light emitted by LED 306, and therefore increases to a peak value and then decreases. Consequently, average intensity 424 also increases to a maximum average intensity peak 432, which represents the image frame whose exposure time most closely matches the time of the LED's "on" state. The corresponding timestamp 426 is shown at the bottom of the graph, with camera timestamp 428 shown above the corresponding GPIO timestamp 430. Assuming the camera timestamp represents the start of the exposure period, the timestamp at the maximum average intensity 432 should be equal.

[0063] Figure 5 yes Figure 4 , to illustrate the offset or error between a particular LED timestamp 504 and a particular camera timestamp 502 corresponding to a particular maximum intensity 506. As can be seen, a difference 508 exists between the particular camera timestamp 502 and the particular LED timestamp 504, indicating that the particular LED timestamp 504 is before the particular camera timestamp 502. The magnitude of the difference can be obtained by subtracting one timestamp from the other, where the sign of the difference indicates whether one timestamp is before or after the other. This difference can then be used to adjust the camera's timestamp function to advance or retard the camera timestamp as needed so that the camera and LED timestamps match. This can be done as part of a calibration process.

[0064] Figure 6 6 is a graph 600 showing sequential average intensity values 602 for images captured by camera 208. As expected, average intensity values 602 increase to a peak and then decrease from the peak as the LED flash is aligned with the camera exposure period and then becomes misaligned. As can be seen from graph 600, the increase and decrease in average intensity are not perfectly linear, and the precise location of the maximum average intensity has some uncertainty due to the width or time spent in the exposure and the proximity of the two maximum average intensity values at the peak. There is no guarantee that a single camera exposure will be perfectly aligned with the LED flash.

[0065] To improve the timestamps recorded by camera 208, a first line 604 is fitted to increasing average intensity values 602, and a second line 606 is fitted to decreasing average intensity values 602. The set of values used to determine lines 604 and 606 may include a common maximum intensity value, if one exists. An intersection point 608 of first line 604 and second line 606 is determined. The value on the time axis of intersection point 608 provides a calculated timestamp 610 that may be more accurate than the camera timestamp for the peak average intensity value. Therefore, calculated timestamp 610 is used as the timestamp for the theoretical image with the maximum average intensity (i.e., the theoretical image that is optimally aligned with the LED flash).

[0066] Figure 7 700 is a flowchart illustrating a method for determining an error or offset in a camera timestamp according to some examples. For illustrative purposes, the operations of flowchart 700 are described herein as occurring serially or linearly. However, multiple operations of flowchart 700 may occur in parallel. Additionally, the operations of flowchart 700 need not be performed in the order shown and / or one or more blocks of flowchart 700 need not be performed and / or may be replaced by other operations.

[0067] Figure 7 The operations shown in the figure will generally be performed on the glasses 100. In other examples, the operations are performed jointly between an application running on the user device 228 and the data processor 202 and related hardware in or associated with the glasses 100. Various implementations are certainly possible in which some of the operations occur in the server system 232, or in which one application calls another application or SDK to obtain the desired functionality.

[0068] The method begins at operation 702, where the microcontroller 302 begins executing instructions to provide periodic LED flashes having a duration approximately equal to the camera exposure time as described above. In operation 704, the high-speed circuitry 220 begins executing instructions to provide periodic image capture by the camera 208 at a specific exposure time, as described above.

[0069] In operation 706, data processor 202 sets a timer for image capture. The duration of the timer is at least long enough to ensure that there will be a complete overlap of the LED flash and exposure time periods so that at least one peak will be captured. If the interval between exposures is of approximately the same duration as the exposure time, and the LED flash has a duration approximately the same as the exposure time "E", but the interval between LED flashes is shorter than the interval between exposures by a time "T", then a timer of duration D = (4*E^2 - 2*E*T) / T will provide sufficient time for the LED flash to sweep through the exposure and provide a guarantee that a valid maximum average intensity value will be captured. If the interval between exposures is not of the same duration as the exposure time, then the time period between subsequent exposures is used in place of (2*E) in the above equation.

[0070] In operation 708, the data processor 202 receives a signal corresponding to the LED flash at the port 308 and timestamps the arrival of the signal. In operation 710, the camera 208 captures an image including at least a portion of the LED flash. The captured image includes a timestamp generated by the camera 208.

[0071] If the data processor 202 determines in operation 712 that the timer has not expired, the flowchart 700 returns to operation 708 to further capture the arrival of the signal corresponding to the next (e.g., subsequent) LED flash, and the method continues therefrom. Operations 708 and 710 are not sequential, but run in parallel.

[0072] When it is determined in operation 712 that the timer has expired, the data processor 202 determines the average intensity of each of the captured images in operation 714. This may be accomplished, for example, by averaging the intensity values of the pixels constituting each image. In operation 716, the data processor 202 determines the captured image with the greatest average intensity. This may be accomplished, for example, by sorting the average image intensities from greatest to least.

[0073] In operation 718, the data processor 202 subtracts the timestamp of the image with the maximum average intensity from the arrival timestamp of the corresponding LED flash. The resulting timestamp offset is output in operation 720. In some examples, in operation 722, the data processor 202 corrects the timestamps performed by the camera by subtracting the timestamp offset determined in operation 720 based on relevant timestamp parameters and an algorithm for camera timestamps.

[0074] Figure 8 FIGURE 8 is a flow chart 800 illustrating alternative steps for determining an error or offset in a camera timestamp. Flow chart 800 begins at operation 802, which corresponds to Figure 7 Operation 714.

[0075] As about Figure 7 As explained in operation 714 of FIG, data processor 202 determines the average intensity of the captured images in operation 802. This can be done, for example, by averaging the intensity values of the pixels comprising each image. In operation 804, the captured image with the maximum average intensity is determined. In operation 806, a straight line is fitted to the average intensity and time stamp values on either side of the image with the maximum average intensity.

[0076] In operation 808, an intersection point of the lines determined in 806 is determined. In operation 810, a time value corresponding to the value of the intersection point in operation 808 is subtracted from the arrival timestamp of the corresponding LED flash.

[0077] The resulting timestamp offset is output in operation 812. In some examples, the data processor 202 corrects the timestamps performed by the camera in operation 722 by subtracting the timestamp offset determined in operation 810 according to relevant timestamp parameters and algorithms used for camera timestamps.

[0078] Figure 9 900 is a diagram illustrating an alternative method of determining a camera timestamp offset according to some examples. The diagram shows the passage of time 902 from left to right in the drawing, along with associated exposure 904, LED state 906, timestamps 910 (including camera timestamp 428 and GPIO timestamp 430), and average intensity 908. Figure 9 Also shown in FIG. 9 is a graph 924 of the expected time difference between the GPIO timestamp 430 and the camera timestamp 428 as a function of time.

[0079] The difference in duration between the LED flash and the exposure is exaggerated in graph 900 to illustrate that there may not be perfect overlap between a single LED flash and a single exposure, and therefore there will not be a perfect maximum average intensity, as described above with reference to FIG. Figure 6 As discussed. Figure 6 , lines 912 and 914 have been fitted to the increasing and decreasing average intensities 908 to determine the timestamp of the theoretical exposure having the calculated maximum average intensity 916 .

[0080] It can be seen that after the first calculated maximum average intensity 916, the interval between the camera device timestamp 428 and the GPIO timestamp 430 gradually becomes larger and larger until the next peak in intensity, at which time the GPIO timestamp 430 has lagged behind the camera device timestamp 428 by a full frame period (for example, 2*E).

[0081] Graph 924 shows the expected offset between camera timestamp 428 and GPIO timestamp 430 based on the fact that if the expected offset is zero at the first calculated maximum average intensity 916 , then it should be the frame period (2*E) at the second calculated maximum average intensity 918 .

[0082] The expected offset can then be compared to the measured offset for each set of measured GPIO timestamps 430 and camera timestamps 428. For example, for a particular camera timestamp 928 and associated LED timestamp 930, the measured time difference 922 can be determined by subtracting the value of the LED timestamp 930 from the value of the camera timestamp 928. The midpoint 932 of these two values can then be used in the graph 924 to determine the expected time difference 934. If the measured time difference 922 and the expected time difference 934 are the same, then there is no error in the camera timestamp 428. If these values are not the same, then the one variable with the expected error is the camera timestamp 928, and therefore the camera timestamp 928 must be in error by the difference between the measured time difference 922 and the expected time difference 934.

[0083] The error or offset in the camera timestamp can be determined for each LED flash between two peaks (such as the calculated maximum average intensity 916 and the calculated maximum average intensity 918) as the difference between the measured time difference and the expected time difference, and averaged to determine a more accurate camera timestamp offset value. Additional processing, such as outlier removal, can be performed on the obtained value. The sign of the error or offset will depend on the specific situation, such as whether the camera exposure time is longer or shorter than the LED flash period.

[0084] Figure 101 is a flowchart 1000 illustrating alternative steps for determining an error or offset in a camera timestamp. For illustrative purposes, the operations of flowchart 1000 are described herein as occurring serially or linearly. However, multiple operations of flowchart 1000 may occur in parallel. Additionally, the operations of flowchart 1000 need not be performed in the order shown and / or one or more blocks of flowchart 1000 need not be performed and / or may be replaced by other operations.

[0085] Flowchart 1000 begins at operation 1002, which corresponds to Figure 7 Operation 714 of FIG. The steps in flowchart 1000 are performed by a data processor, such as data processor 202, and may also be implemented in a non-transitory computer-readable medium.

[0086] As about Figure 7 As explained in operation 714 of FIG, the data processor 202 determines the average intensity of the captured images in operation 1002. For example, this may be done by averaging the intensity values of the pixels comprising each image.

[0087] In operation 1004, a straight line is fitted to two consecutive sets of increasing and decreasing intensity values. Figure 8 , the first set of increasing and decreasing intensity values are before and after, respectively, a calculated maximum average intensity 916 (not yet determined), and the resulting fitted lines are line 912 and line 914. The second (and consecutive) set of increasing and decreasing values are on either side of a calculated maximum average intensity 918 (also not yet determined).

[0088] In operation 1008, the intersection of the line fitted to the first set of intensity values is determined to identify the time value of the first calculated maximum average intensity (such as Figure 9 916 in ), and determining the intersection of the line fitted to the second set of intensity values to identify the time value of the second calculated maximum average intensity (such as Figure 9 918 in the ).

[0089] In operation 1010, the time value of the second calculated maximum average intensity is subtracted from the time value of the first calculated maximum average intensity to determine a period during which the average intensity value cycles between the two maximum values. Then, in operation 1012, a linear relationship between the expected time difference and time is determined using the expected value of zero time difference at the first calculated maximum average intensity at the beginning of the period and the value of the camera exposure frame period (the inverse of the exposure frame rate) at the second calculated maximum average intensity at the end of the period.

[0090] The actual offset between the first camera timestamp and the first LED timestamp after the first calculated maximum average intensity is determined in operation 1014. In subsequent cycles, this will be the next camera timestamp and the next LED timestamp in a direction away from the first calculated maximum average intensity.

[0091] In operation 1016, an expected offset for the first / next set of camera and LED timestamps is determined using the relationship determined in operation 1012 and the time value of the related set of camera and LED timestamps, such as the average of the two timestamp values.

[0092] The difference between the expected offset determined in operation 1016 and the actual offset determined in operation 1014 is then determined in operation 1018. In operation 1020, it is determined whether the set of camera and LED timestamps is the last set before the second calculated maximum average intensity. If not, the method returns to operation 1014 and continues from there.

[0093] If it is the last group, the method proceeds to operation 1022 where a camera timestamp offset is determined using the differences determined in operation 1018. In some examples, the camera timestamp offset is determined as an average of the differences determined in operation 1018.

[0094] The resulting timestamp offset is then output.In some examples, the data processor 202 corrects the timestamps performed by the camera by subtracting the timestamp offset determined in operation 1022 according to relevant timestamp parameters and / or algorithms used for camera timestamps.

[0095] Figure 11 1 is a block diagram illustrating an example messaging system 1100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1100 includes multiple instances of user devices 228 that host several applications, including a messaging client 1102 and other applications 1104. The messaging client 1102 is communicatively coupled to other instances of the messaging client 1102 (e.g., hosted on respective other user devices 228), a messaging server system 1106, and a third-party server 1108 via a network 230 (e.g., the Internet). The messaging client 1102 can also communicate with the locally hosted applications 1104 using an application programming interface (API).

[0096] The messaging clients 1102 are able to communicate and exchange data with other messaging clients 1102 and with a messaging server system 1106 via the network 230. The data exchanged between the messaging clients 1102 and between the messaging clients 1102 and the messaging server system 1106 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0097] The messaging server system 1106 provides server-side functionality to a particular messaging client 1102 via the network 230. Although some functionality of the messaging system 1100 is described herein as being performed by either the messaging client 1102 or the messaging server system 1106, whether some functionality resides within the messaging client 1102 or within the messaging server system 1106 may be a design choice. For example, it may be technically preferable to initially deploy some technology and functionality within the messaging server system 1106 but later migrate the technology and functionality to a messaging client 1102 where the user device 228 has sufficient processing power.

[0098] The messaging server system 1106 supports various services and operations provided to the messaging clients 1102. Such operations include transmitting data to the messaging clients 1102, receiving data from the messaging clients 1102, and processing data generated by the messaging clients 1102. By way of example, this data may include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 1100 is activated and controlled by functionality available through the user interface (UI) of the messaging clients 1102.

[0099] Turning now specifically to the messaging server system 1106, an application program interface (API) server 1110 is coupled to an application server 1114 and provides a programming interface to the application server 1114. The application server 1114 is communicatively coupled to a database server 1116, which facilitates access to a database 1120 that stores data associated with messages processed by the application server 1114. Similarly, a web server 1124 is coupled to the application server 1114 and provides a web-based interface to the application server 1114. To this end, the web server 1124 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0100] The application program interface (API) server 1110 receives and transmits message data (e.g., commands and message payloads) between the user device 228 and the application server 1114. Specifically, the application program interface (API) server 1110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1102 to activate the functionality of the application server 1114. The application program interface (API) server 1110 exposes various functions supported by the application server 1114, including: account registration; login functionality; sending messages from a particular messaging client 1102 to another messaging client 1102 via the application server 1114, sending media files (e.g., images or videos) from the messaging client 1102 to the messaging server 1112, and possible access by another messaging client 1102; setting up collections of media data (e.g., stories); retrieving a friend list of a user of the user device 228; retrieving such collections; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1102).

[0101] The application server 1114 hosts several server applications and subsystems, including, for example, a messaging server 1112, an image processing server 1118, and a social network server 1122. The messaging server 1112 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1102. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1102. Given the hardware requirements for processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1112.

[0102] The application server 1114 also includes an image processing server 1118 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1112.

[0103] The social network server 1122 supports various social networking functions and services and makes these functions and services available to the messaging server 1112. To do so, the social network server 1122 maintains and accesses an entity graph within the database 1120. Examples of functions and services supported by the social network server 1122 include identifying other users in the messaging system 1100 with whom a particular user has relationships or who the particular user is "following," and also identifying interests and other entities of a particular user.

[0104] The messaging client 1102 may notify the user of the user device 228 or other users related to such user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1102 may provide participants in a conversation (e.g., a chat session) in the messaging client 1102 with notifications related to current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0105] Figure 12 12 is a block diagram 1200 illustrating a software architecture 1204 that can be installed on any one or more of the devices described herein. The software architecture 1204 is supported by hardware, such as a machine 1202 including a processor 1220, memory 1226, and I / O components 1238. In this example, the software architecture 1204 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1204 includes layers such as an operating system 1212, libraries 1208, frameworks 1210, and applications 1206. In operation, the applications 1206 invoke API calls 1250 through the software stack and receive messages 1252 in response to the API calls 1250.

[0106] The operating system 1212 manages hardware resources and provides common services. The operating system 1212 includes, for example, a kernel 1214, services 1216, and drivers 1222. The kernel 1214 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1214 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1216 can provide other common services to other software layers. Drivers 1222 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1222 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, etc.

[0107] Libraries 1208 provide low-level common infrastructure used by applications 1206. Libraries 1208 may include system libraries 1218 (eg, C standard library) that provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1208 may include an API library 1224, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting graphical content in two dimensions (2D) and three dimensions (3D) on a display, GLMotif for implementing a 3D user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1208 may also include various other libraries 1228 to provide many other APIs to the application 1206.

[0108] The framework 1210 provides a high-level common infrastructure used by the applications 1206. For example, the framework 1210 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1210 can provide a wide range of other APIs that can be used by the applications 1206, some of which may be specific to a particular operating system or platform.

[0109] In an example, the applications 1206 may include a home application 1236, a contacts application 1230, a browser application 1232, a shopping application 1234, a location application 1242, a media application 1244, a messaging application 1246, a game application 1248, and a variety of other applications such as third-party applications 1240. An application 1206 is a program that performs the functions defined in the program. Various programming languages may be used to create one or more of the applications 1206 structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application 1240 (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID) may be used to create a third-party application 1240. TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOSTM ANDROID TM 、 Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 1240 can call API calls 1250 provided by the operating system 1212 to facilitate the functions described herein.

[0110] Figure 13 1300 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed on the machine 1300 to cause the machine 1300 to perform any one or more of the methodologies discussed herein. For example, the instructions 1310 may cause the machine 1300 to perform any one or more of the methodologies described herein. The instructions 1310 transform a general-purpose, unprogrammed machine 1300 into a specialized machine 1300 that is programmed to perform the functions described and illustrated in the manner described. The machine 1300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1300 may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 1310 specifying actions to be taken by the machine 1300. Furthermore, while a single machine 1300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute the instructions 1310 to perform any one or more of the methodologies discussed herein.

[0111] The machine 1300 may include a processor 1302, a memory 1304, and an I / O component 1306 that may be configured to communicate with each other via a bus 1344. In an example, the processor 1302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1308 that executes instructions 1310 and a processor 1312. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 13 Multiple processors 1302 are shown, but the machine 1300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0112] The memory 1304 includes a main memory 1314, a static memory 1316, and a storage unit 1318, all of which are accessible by the processor 1302 via a bus 1344. The main memory 1304, the static memory 1316, and the storage unit 1318 store instructions 1310 that implement any one or more of the methods or functions described herein. During execution of the instructions 1310 by the networked system 200, the instructions 1310 may also reside, in whole or in part, within the main memory 1314, within the static memory 1316, within the machine-readable medium 1320, within the storage unit 1318, within one or more of the processors 1302 (e.g., within a processor's cache), or within any suitable combination thereof.

[0113] The I / O components 1306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is understood that the I / O components 1306 may include Figure 13Many other components are not shown in the figure. In various examples, the I / O components 1306 may include output components 1328 and input components 1332. The output components 1328 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1332 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0114] In another example, the I / O component 1306 may include a biometric component 1334, a motion component 1336, an environmental component 1338, or a positioning component 1340, as well as various other components. For example, the biometric component 1334 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1336 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1338 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1340 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure that can provide altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0115] A variety of technologies can be used to achieve communication. I / O components 1306 also include communication components 1342 that are operable to couple networked system 200 to network 1322 or device 1324 via coupling 1330 and coupling 1326, respectively. For example, communication components 1342 may include a network interface component or another suitable device that interfaces with network 1322. In other examples, communication components 1342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Components and other communication components that provide communication via other modalities. Device 1324 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0116] In addition, the communication component 1342 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF417, a Hypercode, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1342, such as a location obtained via Internet Protocol (IP) geolocation, a location obtained via Location obtained by signal triangulation, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0117] Various memories (e.g., memory 1304, main memory 1314, static memory 1316, and / or memory of processor 1302) and / or storage unit 1318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1310) when executed by processor 1302 cause various operations to implement the disclosed examples.

[0118] Instructions 1310 may be transmitted or received over network 1322 via a network interface device (e.g., a network interface component included in communications component 1342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1310 may be transmitted or received to device 1324 via coupling 1326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0119] "Carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0120] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0121] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0122] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed to be used together with other components and a part of a program for a specific function that generally performs related functions. A component can constitute a software component (e.g., a code implemented on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., an independent computer system, a client computer system or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured to operate to perform some operations as described herein by software (e.g., an application or an application part). Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component can be a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is customized to perform the configured function, rather than a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven due to cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any one moment. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as a different special-purpose processor (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. A hardware component can provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components can be considered to be communicatively coupled.When there are multiple hardware components at the same time, communication can be achieved by signal transmission (for example, by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example that multiple hardware components are configured or instantiated at different times, the communication between such hardware components can be achieved, for example, by storing information in a memory structure that multiple hardware components can access and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to its communication ground. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (for example, by software) or permanently configured to perform one or more processors for performing related operations. Whether it is temporary configuration or permanent configuration, such a processor can constitute a component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein specific one or more processors are examples of hardware. For example, some operations in the operation of the method can be performed by the components implemented by one or more processors or processors. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or the components implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server farm). In other examples, a processor or the components implemented by a processor can be distributed across multiple geographical locations.

[0123] "Computer-readable media" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and are used interchangeably in this disclosure.

[0124] “Machine storage medium” refers to a single or multiple storage devices and / or media (e.g., a centralized or distributed database, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term “signal media.”

[0125] A "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0126] "Signal medium" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0127] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

[0128] "Carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0129] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0130] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0131] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed to be used together with other components and a part of a program for a specific function that generally performs related functions. A component can constitute a software component (e.g., a code implemented on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., an independent computer system, a client computer system or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured to operate to perform some operations as described herein by software (e.g., an application or an application part). Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component can be a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is customized to perform the configured function, rather than a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven due to cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any one moment. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as a different special-purpose processor (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. A hardware component can provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components can be considered to be communicatively coupled.When there are multiple hardware components at the same time, communication can be achieved by signal transmission (for example, by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example that multiple hardware components are configured or instantiated at different times, the communication between such hardware components can be achieved, for example, by storing information in a memory structure that multiple hardware components can access and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to its communication ground. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (for example, by software) or permanently configured to perform one or more processors for performing related operations. Whether it is temporary configuration or permanent configuration, such a processor can constitute a component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein specific one or more processors are examples of hardware. For example, some operations in the operation of the method can be performed by the components implemented by one or more processors or processors. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or the components implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server farm). In other examples, a processor or the components implemented by a processor can be distributed across multiple geographical locations.

[0132] "Computer-readable media" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and are used interchangeably in this disclosure.

[0133] “Machine storage medium” refers to a single or multiple storage devices and / or media (e.g., a centralized or distributed database, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term “signal media.”

[0134] A "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0135] "Signal medium" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0136] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

Claims

1. A computer-implemented method for determining an image capture timestamp offset, the method comprising: generating a plurality of optical flashes at an optical flash rate; capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different than the optical flash rate, and each image of the optical flash including an associated image timestamp; time-stamping a signal associated with generation of the plurality of optical flashes; determining an intensity of each image in the set of images; determining an image having a maximum intensity in the set of images; as well as A difference between a timestamp of the image having the maximum intensity and a timestamp of a corresponding optical flash is determined to generate an image capture timestamp offset.

2. The computer-implemented method of claim 1 , wherein: The optical flash duration is approximately equal to the image exposure duration.

3. The computer-implemented method of claim 1 , further comprising: determining a first line based on an intensity and a timestamp of an image in the set of images that precedes the image in the set of images having the maximum intensity; determining a second line based on the intensity and the time stamp of an image in the set of images subsequent to the image in the set of images having the maximum intensity; determining an intersection point of the first line and the second line; as well as When determining the difference between the time stamp of the image with the maximum intensity and the time stamp of the corresponding optical flash, the time value of the intersection of the first line and the second line is used as the time stamp of the image with the maximum intensity.

4. The computer-implemented method of claim 1 , further comprising: A camera timestamp parameter is adjusted based on the image capture timestamp offset.

5. The computer-implemented method of claim 1 , wherein: The signal associated with the generation of the optical flash is received at a port of a computer processor associated with a camera that captured the set of images of the optical flash, the computer processor timestamping receipt of the signal associated with the generation of the optical flash.

6. The computer-implemented method of claim 1 , further comprising: Setting a timer having a duration sufficient to ensure that an image having maximum intensity is captured; as well as The set of images of the optical flash is captured within the duration of the timer.

7. The computer-implemented method of claim 1 , wherein: The image intensity of each image is the average image intensity.

8. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate, the operations comprising: generating a plurality of optical flashes at an optical flash rate; capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different than the optical flash rate, and each image of the optical flash including an associated image timestamp; time-stamping a signal associated with generation of the plurality of optical flashes; determining an intensity of each image in the set of images; determining an image having a maximum intensity in the set of images; as well as A difference between a timestamp of the image having the maximum intensity and a timestamp of a corresponding optical flash is determined to generate an image capture timestamp offset.

9. The non-transitory computer-readable storage medium of claim 8, wherein: The optical flash duration is approximately equal to the image exposure duration.

10. The non-transitory computer-readable storage medium of claim 8, wherein: The operations further include: determining a first line based on an intensity and a timestamp of an image in the set of images that precedes the image in the set of images having the maximum intensity; determining a second line based on the intensity and the time stamp of an image in the set of images subsequent to the image in the set of images having the maximum intensity; determining an intersection point of the first line and the second line; and When determining the difference between the time stamp of the image with the maximum intensity and the time stamp of the corresponding optical flash, the time value of the intersection of the first line and the second line is used as the time stamp of the image with the maximum intensity.

11. The non-transitory computer-readable storage medium of claim 8, wherein: The operations further include: A camera timestamp parameter is adjusted based on the image capture timestamp offset.

12. The non-transitory computer-readable storage medium of claim 8, wherein: The signal associated with the generation of the optical flash is received at a port of a computer processor associated with a camera that captured the set of images of the optical flash, the computer processor timestamping receipt of the signal associated with the generation of the optical flash.

13. The non-transitory computer-readable storage medium of claim 8, wherein: The image intensity of each image is the average image intensity.

14. The non-transitory computer-readable storage medium of claim 8, wherein: The operations further include: setting a timer having a duration sufficient to ensure that an image having maximum intensity is captured; and The set of images of the optical flash is captured within the duration of the timer.

15. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate, the operations comprising: generating a plurality of optical flashes at an optical flash rate; capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different than the optical flash rate, and each image of the optical flash including an associated image timestamp; time-stamping a signal associated with generation of the plurality of optical flashes; determining an intensity of each image in the set of images; determining an image in the set of images having a maximum intensity; and A difference between a timestamp of the image having the maximum intensity and a timestamp of a corresponding optical flash is determined to generate an image capture timestamp offset.

16. The computing device of claim 15, wherein: The optical flash duration is approximately equal to the image exposure duration.

17. The computing device of claim 15, wherein: The operations further include: determining a first line based on an intensity and a timestamp of an image in the set of images that precedes the image in the set of images having the maximum intensity; determining a second line based on the intensity and the time stamp of an image in the set of images subsequent to the image in the set of images having the maximum intensity; determining an intersection point of the first line and the second line; and When determining the difference between the time stamp of the image with the maximum intensity and the time stamp of the corresponding optical flash, the time value of the intersection of the first line and the second line is used as the time stamp of the image with the maximum intensity.

18. The computing device of claim 15, wherein: The operations further include: A camera timestamp parameter is adjusted based on the image capture timestamp offset.

19. The computing device of claim 15, wherein: The signal associated with generation of the optical flash is received at a port of the processor, the processor timestamping receipt of the signal associated with generation of the optical flash.

20. The computing device of claim 15, wherein: The image intensity of each image is the average image intensity.

Images