Camera timestamp verification

Abstract

A method of determining an image capture timestamp offset or error includes generating an optical flash at an optical flash rate. A set of images of the optical flash is captured at an image capture rate. The image capture rate is different from the optical flash rate, and each image includes an associated image timestamp. Signals associated with the generation of the optical flash are also timestamped. An intensity of each image in the set of images is determined, and an image of the set of images having a maximum intensity is identified. The timestamp offset or error is determined as a difference between the timestamp of the image having the maximum intensity and the timestamp of the corresponding optical flash.

Description

[0001] Related application data

[0002] This application claims priority to U.S. patent application serial number 17 / 845,478 filed on June 21, 2022. Technical Field

[0003] The present disclosure relates generally to image capture devices and, more particularly, but not exclusively, to image capture devices for augmented reality. Background Art

[0004] Augmented reality (AR) devices allow a user to view a surrounding environment (the "real world") and also to see objects generated for display that appear to be part of and / or superimposed on the surrounding environment (e.g., virtual objects such as 3D renderings, images, videos, text, etc.). In some examples, the AR device takes the form of glasses with a display through which the user can see the real world and on which virtual objects can be displayed. In other examples, the AR device may take the form of a smartphone or tablet. In such a case, a video feed of the real world captured by a camera in the smartphone or tablet is displayed on the display of the smartphone or tablet, with AR objects or effects included in or applied to the video feed of the real world.

[0005] An AR device relies on accurate timing information from one or more associated cameras to ensure accurate tracking of the AR device as it moves, and to ensure correct placement of AR objects and effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 It is shown that according to some examples, Figure 1 A block diagram of a networking system 200 showing details of a head mounted device.

[0009] Figure 3 is a schematic diagram illustrating a system for performing camera timestamp verification or camera timestamp shifting according to some examples.

[0010] Figure 4 is shown according to some examples Figure 3 A diagram of the sequence of events in the operation of a system.

[0011] Figure 5 yes Figure 4 A magnified version of the region of maximum mean intensity is shown in .

[0012] Figure 6 is a graph showing sequential average intensity values ​​for images captured by a camera according to some examples.

[0013] Figure 7 is a flow chart illustrating a method of determining an error or offset in a camera timestamp according to some examples.

[0014] Figure 8 is a flow chart illustrating a method of determining an error or offset in a camera timestamp according to some examples.

[0015] Fig. 9 is a diagram illustrating alternative methods of determining camera timestamp offsets according to some examples.

[0016] Fig.10 is a flow chart illustrating a method of determining an error or offset in a camera timestamp according to some examples.

[0017] Fig.11 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0018] Fig.12 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to some examples.

[0019] Fig.13 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0020] Some head-mounted AR devices, such as AR glasses, include a transparent or translucent display, which enables the user to see through the transparent or translucent display to view the surrounding environment. Additional information or objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) are displayed on the display and appear as part of the surrounding environment and / or superimposed on the surrounding environment to provide an augmented reality (AR) experience for the user. The display may, for example, include a waveguide that receives a beam from a projector, but any suitable display for presenting augmented content or virtual content to the wearer may be used.

[0021] The correct positioning of virtual objects and other AR effects, and the synchronization of their movement or updating with what is happening in the real world, depends on accurate timing information received from the camera used to capture images or video feeds of the real world. The captured images are typically used by the AR device to represent the real world, such as device positioning and tracking using image processing techniques. The accuracy of the timing of image capture is relative to other events occurring in the AR device, so that two different events occurring at the same moment will have the same timestamp assigned by the AR device.

[0022] To determine an offset or error in the timestamps of images captured by a camera, a set of time-stamped images of an optical flash are captured at an image capture rate. The image capture rate is different from the optical flash rate. A signal associated with the generation of the optical flash is also time-stamped. The intensity of each image in the set of images is determined, and the image in the set of images having the maximum intensity is identified. The timestamp offset or error of the camera is determined as the difference between the timestamp of a real or theoretical image having the maximum intensity and the timestamp of the corresponding optical flash.

[0023] The timestamp offset can then be used to adjust the camera timestamping so that the image timestamps more accurately align with other timestamps generated by the AR device. This provides improved alignment and synchronization of visual content with real-world objects.

[0024] In some examples, a computer-implemented method of determining an image capture timestamp offset is provided, comprising: generating a plurality of optical flashes at an optical flash rate, capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different from the optical flash rate, and each image of the optical flashes including an associated image timestamp, timestamping a signal associated with the generation of the plurality of optical flashes, determining an intensity of each image in the set of images, determining an image in the set of images having a maximum intensity, and determining a difference between the timestamp of the image having the maximum intensity and the timestamp of the corresponding optical flash to generate an image capture timestamp offset. The optical flash duration may be approximately equal to the image exposure duration.

[0025] The method may also include determining a first line based on the intensity and timestamp of an image in the group of images that precedes the image with maximum intensity in the group of images, determining a second line based on the intensity and timestamp of an image in the group of images that follows the image with maximum intensity in the group of images, determining an intersection of the first line and the second line, and, when determining the difference between the timestamp of the image with maximum intensity and the timestamp of the corresponding optical flash, using the time value of the intersection of the first line and the second line as the timestamp of the image with maximum intensity.

[0026] A signal associated with the generation of the optical flash may be received at a port of a computer processor associated with a camera that captured a set of images with the optical flash, the computer processor time stamping receipt of the signal associated with the generation of the optical flash.

[0027] The method may include setting a timer having a duration sufficient to ensure that an image having maximum intensity is captured, and capturing a set of images of the optical flash during the duration of the timer. The image intensity of each image may be an average image intensity.

[0028] The method may include adjusting a camera timestamping parameter based on the image capture timestamp offset.

[0029] In some examples, a non-transitory computer-readable storage medium is provided, which includes instructions that, when executed by a computer, cause the computer to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate according to any of the above methods and limitations, the operations including but not limited to the following operations: generating multiple optical flashes at an optical flash rate, capturing a set of images of the multiple optical flashes at an image capture rate, the image capture rate being different from the optical flash rate, and each image of the optical flash including an associated image timestamp, timestamping a signal associated with the generation of the multiple optical flashes, determining an intensity of each image in the set of images, determining an image with a maximum intensity in the set of images, and determining a difference between the timestamp of the image with the maximum intensity and the timestamp of the corresponding optical flash to generate an image capture timestamp offset.

[0030] In some examples, a computing device is provided that includes a processor and a memory storing instructions that, when executed by the processor, configure the device to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate according to any of the above methods and limitations, the operations including, but not limited to, generating multiple optical flashes at an optical flash rate, capturing a set of images of the multiple optical flashes at an image capture rate, the image capture rate being different from the optical flash rate, and each image of the optical flash including an associated image timestamp, timestamping a signal associated with the generation of the multiple optical flashes, determining an intensity of each image in the set of images, determining an image with a maximum intensity in the set of images, and determining a difference between the timestamp of the image with the maximum intensity and the timestamp of the corresponding optical flash to generate an image capture timestamp offset.

[0031] Figure 11 is a perspective view of a head-mounted AR device (e.g., glasses 100) according to some examples. Glasses 100 may include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112.

[0032] A first optical element or left optical element 108 and a second optical element or right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination thereof. Any suitable display component may be disposed in the glasses 100.

[0033] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or integrated construction.

[0034] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 202 discussed below.

[0035] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0036] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0037] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from a real-world scene.

[0038] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, generally vertically aligned means that the touchpad is more vertical than horizontal, although potentially more vertical than described. Additional user input may be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from a user of the glasses 100.

[0039] In use, information, content, and various 3D user interfaces will be presented to a user of the glasses 100 on a near-eye display in the left optical element 108 and / or the right optical element 110. The user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 2 The user device 228 shown in FIG. 2 may be used to interact with the glasses 100 through voice input or touch input on the glasses 100 and / or hand movements, positions, and locations detected by the glasses 100.

[0040] Figure 2is a block diagram showing a networked system 200 including details of the glasses 100 according to some examples. The networked system 200 includes the glasses 100, a user device 228, and a server system 232. The user device 228 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low power wireless connection 236 and / or a high speed wireless connection 234. The user device 228 is connected to the server system 232 via a network 230. The network 230 may include any combination of wired and wireless connections. The server system 232 may be one or more computing devices that are part of a service or network computing system. The server system 232 may be a server system 232 that is a networked computing system that may be used separately. Fig.12 and Fig.13 The details of the software architecture 1204 or machine 1300 described in the implementation of the user device 228 and any elements of the server system 232 and network 230.

[0041] The glasses 100 include a data processor 202, a display 210, one or more cameras 208, and additional input / output elements 216. The input / output elements 216 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 202. Fig.12 and Fig.13 Examples of input / output elements 216 are further discussed. For example, input / output elements 216 may include any I / O components 1306 including output components 1328, motion components 1336, etc. In the particular example described herein, display 210 includes a display for each of the left and right eyes of a user.

[0042] Data processor 202 includes image processor 206 (eg, video processor), GPU and display driver 238, tracking module 240, interface 212, low power circuitry 204, and high speed circuitry 220. Components of data processor 202 are interconnected by bus 242.

[0043] Interface 212 refers to any source of user commands provided to data processor 202. In one or more examples, interface 212 is a physical button that, when pressed, sends a user input signal from interface 212 to low-power processor 214. Low-power processor 214 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. Low-power processor 214 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing a button for a long period of time can capture a still image. In some examples, interface 212 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera 208. In other examples, interface 212 can have a software component, or can be associated with a command received wirelessly from another source, such as from user device 228.

[0044] Image processor 206 includes circuitry for receiving signals from camera 208 and processing those signals from camera 208 into a format suitable for storage in memory 224 or for transmission to user device 228. In one or more examples, image processor 206 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 208, and volatile memory used by the microprocessor in operation.

[0045] The low power circuit system 204 includes a low power processor 214 and a low power wireless circuit system 218. These elements of the low power circuit system 204 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 214 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 214 can accept user input signals from the interface 212. The low power processor 214 can also be configured to receive input signals or command communications from a user device 228 via a low power wireless connection 236. The low power wireless circuit system 218 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 218. In other examples, other low power communication systems may be used.

[0046] High-speed circuit system 220 includes high-speed processor 222, memory 224, and high-speed wireless circuit system 226. High-speed processor 222 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 202. High-speed processor 222 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 234 using high-speed wireless circuit system 226. In some examples, high-speed processor 222 executes an operating system such as the LINUX operating system or a program such as a UNIX operating system. Fig.12 The high-speed processor 222, which executes the software architecture of the data processor 202, manages data transmission with the high-speed wireless circuit system 226, in addition to any other duties. In some examples, the high-speed wireless circuit system 226 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 226 can implement other high-speed communication standards.

[0047] The memory 224 includes any storage device capable of storing camera data generated by the camera 208 and the image processor 206. Although the memory 224 is shown as being integrated with the high-speed circuitry 220, in other examples, the memory 224 may be a separate, independent element of the data processor 202. In some such examples, electrical wiring may provide a connection from the image processor 206 or the low-power processor 214 to the memory 224 through a chip that includes the high-speed processor 222. In other examples, the high-speed processor 222 may manage the addressing of the memory 224 so that the low-power processor 214 will direct the high-speed processor 222 whenever a read or write operation involving the memory 224 is desired.

[0048] The tracking module 240 estimates the position and orientation ("pose") of the glasses 100. For example, the tracking module 240 uses image data and corresponding inertial data from the camera 208 and the positioning component 1340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world environment). The tracking module 240 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100 that indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 240 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 210.

[0049] The GPU and display driver 238 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 210 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 238 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world environment.

[0050] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the user device 228 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1206, such as the messaging application 1246.

[0051] Figure 3 is a schematic diagram showing a system 300 for performing camera timestamp verification or camera timestamp shifting according to some examples. The system includes a microcontroller 302, a transistor 304 (such as a metal oxide semiconductor field effect transistor), a light emitting diode (LED 306), and a camera 208 and a high-speed circuit system 220 that form part of an AR device such as glasses 100 or a user device 228. Figure 3 It is a conceptual diagram that is not intended to illustrate a literal circuit.

[0052] Microcontroller 302 provides an on / off signal which provides / removes power to LED 306 via transistor 304. The on / off signal is simultaneously provided to general purpose input / output port 308 in high speed circuitry 220 where its "on" arrival is time stamped to give an accurate measurement of the activation time of LED 306. Frames from camera 208, time stamped by camera 208, are provided to high speed circuitry 220 via camera serial interface 310. The timestamp is generated by the processor that processes the camera image.

[0053] The high speed circuit system 220 includes an application with program instructions that, when executed, captures a timestamp generated when a signal from the microcontroller 302 arrives at the port 308, and captures a corresponding image of the flash of the LED 306 using the camera 208. The image of the flash will include the camera timestamp. The sequence of images and timestamps will be captured and saved to the memory 224. Additional program instructions may be included in the high speed circuit system 220 to analyze the set of images and data in whole or in part, or the set of images and data may be uploaded for analysis locally (such as on the user device 228 or another local computing device), or remotely for analysis at the server system 232. The program instructions will perform the following with reference to Figure 7Describe the method.

[0054] As described below, microcontroller 302 also includes program instructions that instruct microcontroller 302 to provide a periodic signal to flash LED 306. The period of the signal may be adjustable.

[0055] In use, the camera 208 is set to a fixed exposure, and the LED 306 is flashed by the microcontroller 302 for a duration approximately equal to the image exposure time, but at a frequency slightly different from the camera frame rate. When the flashing of the LED 306 is exactly in phase with the camera exposure, then the average intensity of the image captured by the camera 208 will be at its maximum, and the timestamp of the corresponding image and the timestamp generated when the signal is received at the port 308 should be the same. By capturing a stream of images and identifying the brightest image, and comparing the LED timestamp generated by the general purpose input / output (GPIO) to the camera timestamp, any error in the camera timestamp can be determined, as discussed in more detail below.

[0056] For simplicity of implementation, the intensity of a single global optical flash is measured over the complete image. A more general approach may direct any number of optical flashes (possibly with known temporal extension) to specific areas on the image sensor surface to achieve the same result and / or increase the robustness of the measurement process. The optical flash may also come from a laser diode or other light source.

[0057] Whether the image data captured by the camera 208 is stored as a video or a single image is a matter of coding. The camera 208 can generate an image stream (output images at a constant frequency) or provide repeated on-demand image capture. Either can be used in the methods described herein, but image streaming is more convenient.

[0058] The most information is obtained from the image data if the exposure time is half the duration of the frame period (the inverse of the frame rate). If the exposure time is greater than half the frame period, the peaks of the average intensity plot are flattened, and if the exposure time is less than half the frame period, the troughs of the average intensity plot are flattened. To obtain optimal data, the exposure is preferably balanced with the gain on the image sensor to ensure that the maximum average intensity is as bright as possible while avoiding oversaturated pixels. The exposure duration should not result in oversaturation, but should preferably be close to half the frame rate. Such optimization improves the quality of the data, but if enough data is captured, any errors due to deviations from ideal conditions should be eliminated.

[0059] All other things being equal, the faster the frame rate, the faster all the required data can be captured, but in practice this is not an important issue. The frame rate also depends on the frame rate that the camera is capable of at the requested resolution, which for the purposes of this article does not need to be high resolution.

[0060] Figure 4 It is shown Figure 3 4 is a diagram of a sequence of events in the operation of the system 300. The diagram shows the passage of time 402 from left to right in the figure, and the associated LED flashes (corresponding to the LED "on" state), exposures, time stamps, etc.

[0061] As can be seen in the figure, camera exposure 404 and "on" LED state 406 have approximately the same duration, but the start of each LED "on" state occurs at a frequency slightly less than the camera exposure frequency. Thus, LED states 406 progress from initially starting slightly after the start of the corresponding camera exposure (LED Start 408 and Exposure Start 410), to coinciding with the start of the corresponding camera exposure (LED Start 412 and Exposure Start 414), to before the start of the camera exposure (LED Start 416 and Exposure Start 418).

[0062] The amount of overlap 420 between the two states corresponds to the time that the camera 208 captures the light emitted by the LED 306, and therefore increases to a peak value and then decreases. Therefore, the average intensity 424 also increases to a maximum average intensity peak 432, which represents the image frame whose exposure time most closely matches the time of the "on" state of the LED. The corresponding timestamp 426 is shown at the bottom of the graph, and the camera timestamp 428 is shown above the corresponding GPIO timestamp 430. Assuming that the camera timestamp represents the beginning of the exposure period, the timestamp at the maximum average intensity 432 should be equal.

[0063] Figure 5 yes Figure 4 , to illustrate the offset or error between a particular LED timestamp 504 and a particular camera timestamp 502 corresponding to a particular maximum intensity 506. As can be seen, a difference 508 exists between the particular camera timestamp 502 and the particular LED timestamp 504, indicating that the particular LED timestamp 504 is before the particular camera timestamp 502. The magnitude of the difference can be obtained by subtracting one timestamp from the other, where the sign of the difference indicates whether one timestamp is before or after the other. This difference can then be used to adjust the camera's timestamp function to advance or retard the camera timestamp as needed so that the camera and LED timestamps match. This can be done as part of a calibration process.

[0064] Figure 6 6 is a graph 600 showing sequential average intensity values ​​602 for images captured by camera 208. As expected, average intensity values ​​602 increase to a peak and then decrease from the peak as the LED flash is aligned with the camera exposure period and then becomes misaligned. It can be seen from graph 600 that the increase and decrease in average intensity is not perfectly linear and the exact location of the maximum average intensity has some uncertainty due to the width or time spent in the exposure and the closeness of the values ​​of the two maximum average intensities at the peak. There is no guarantee that there will be a single camera exposure that is perfectly aligned with the LED flash.

[0065] To improve the timestamps recorded by camera 208, a first line 604 is fit to increasing average intensity values ​​602, and a second line 606 is fit to decreasing average intensity values ​​602. The set of values ​​used to determine line 604 and line 606 may include a common maximum intensity value, if one exists. An intersection 608 of first line 604 and second line 606 is determined. The value on the time axis of intersection 608 provides a calculated timestamp 610 that may be more accurate than the camera timestamp for the peak average intensity value. Thus, the calculated timestamp 610 is used as the timestamp for the theoretical image with the maximum average intensity (i.e., the theoretical image that is best aligned with the LED flash).

[0066] Figure 7 700 is a flowchart illustrating a method of determining an error or offset in a camera timestamp according to some examples. For purposes of illustration, the operations of flowchart 700 are described herein as occurring serially or linearly. However, multiple operations of flowchart 700 may occur in parallel. Additionally, the operations of flowchart 700 need not be performed in the order shown and / or one or more blocks of flowchart 700 need not be performed and / or may be replaced by other operations.

[0067] Figure 7 The operations shown in the figure will generally be performed on the glasses 100. In other examples, the operations are performed jointly between an application running on the user device 228 and the data processor 202 and related hardware in or associated with the glasses 100. Various implementations are certainly possible in which some of the operations occur in the server system 232, or in which one application calls another application or SDK to obtain the desired functionality.

[0068] The method begins at operation 702, where the microcontroller 302 begins executing instructions to provide periodic LED flashes having a duration of approximately the camera exposure time as described above. In operation 704, the high-speed circuit system 220 begins executing instructions to provide periodic image capture by the camera 208 at a specific exposure time, as described above.

[0069] In operation 706, the data processor 202 sets a timer for image capture. The duration of the timer is at least long enough to ensure that there will be a complete overlap of the LED flash and exposure time periods so that at least one peak will be captured. If the interval between exposures is of approximately the same duration as the exposure time, and the LED flash has approximately the same duration as the exposure time "E", but the interval between LED flashes is shorter than the interval between exposures by a time "T", then a timer of duration D = (4*E^2 - 2*E*T) / T will provide sufficient time for the LED flash to sweep across the exposure and provide a guarantee that a valid maximum average intensity value will be captured. If the interval between exposures is not of the same duration as the exposure time, then the time period between subsequent exposures is used in place of (2*E) in the above equation.

[0070] In operation 708, the data processor 202 receives a signal corresponding to the LED flash at the port 308 and timestamps the arrival of the signal. At operation 710, the camera 208 captures an image including at least a portion of the LED flash. The captured image includes a timestamp generated by the camera 208.

[0071] If the data processor 202 determines in operation 712 that the timer has not expired, the flowchart 700 returns to operation 708 to further capture the arrival of the signal corresponding to the next (eg, subsequent) LED flash, and the method continues therefrom. Operations 708 and 710 are not sequential, but run in parallel.

[0072] When it is determined in operation 712 that the timer has expired, the data processor 202 determines the average intensity of each of the captured images in operation 714. For example, this can be done by averaging the intensity values ​​of the pixels constituting each image. In operation 716, the data processor 202 determines the captured image with the largest average intensity. This can be done, for example, by sorting the average image intensities from largest to smallest.

[0073] In operation 718, the data processor 202 subtracts the timestamp of the image with the maximum average intensity from the arrival timestamp of the corresponding LED flash. The resulting timestamp offset is output in operation 720. In some examples, the data processor 202 corrects the timestamp performed by the camera in operation 722 by subtracting the timestamp offset determined in operation 720 according to relevant timestamp parameters and algorithms for camera timestamps.

[0074] Figure 8 8 is a flowchart 800 showing alternative steps for determining an error or offset in a camera timestamp. Flowchart 800 begins at operation 802, which corresponds to Figure 7 Operation 714.

[0075] As about Figure 7 As explained in operation 714 of , the data processor 202 determines the average intensity of the captured images in operation 802. For example, this can be done by averaging the intensity values ​​of the pixels that make up each image. In operation 804, the captured image with the maximum average intensity is determined. In operation 806, a straight line is fit to the average intensity and time stamp values ​​on either side of the image with the maximum average intensity.

[0076] In operation 808, the intersection of the lines determined in 806 is determined. In operation 810, the time value corresponding to the value of the intersection in operation 808 is subtracted from the arrival timestamp of the corresponding LED flash.

[0077] The resulting timestamp offset is output in operation 812. In some examples, the data processor 202 corrects the timestamps performed by the camera in operation 722 by subtracting the timestamp offset determined in operation 810 according to relevant timestamp parameters and algorithms used for camera timestamping.

[0078] Fig. 9 900 is a diagram illustrating an alternative method of determining a camera timestamp offset according to some examples. The diagram shows the passage of time 902 from left to right in the figure, and the associated exposure 904, LED state 906, timestamps 910 (including camera timestamp 428 and GPIO timestamp 430), and average intensity 908. Fig. 9 Also shown in FIG. 9 is a graph 924 of the expected time difference between the GPIO timestamp 430 and the camera timestamp 428 as a function of time.

[0079] The difference in duration between the LED flash and the exposure is exaggerated in graph 900 to illustrate that there may not be perfect overlap between a single LED flash and a single exposure, and therefore there will not be a perfect maximum average intensity, as described above with reference to FIG. Figure 6 As discussed. Figure 6 , lines 912 and 914 have been fit to the increasing and decreasing average intensities 908 to determine the timestamp of the theoretical exposure having the calculated maximum average intensity 916.

[0080] It can be seen that after the first calculated maximum average intensity 916, the interval between the camera timestamp 428 and the GPIO timestamp 430 gradually becomes larger and larger until the next peak of the intensity, when the GPIO timestamp 430 has lagged behind the camera timestamp 428 by a full frame period (e.g., 2*E).

[0081] Graph 924 shows the expected offset between camera timestamp 428 and GPIO timestamp 430 based on the fact that if the expected offset is zero at the first calculated maximum average intensity 916 , then it should be a frame period (2*E) at the second calculated maximum average intensity 918 .

[0082] The expected offset can then be compared to the measured offset for each set of measured GPIO timestamps 430 and camera timestamps 428. For example, for a particular camera timestamp 928 and associated LED timestamp 930, the measured time difference 922 can be determined by subtracting the value of the LED timestamp 930 from the value of the camera timestamp 928. The midpoint 932 of these two values ​​can then be used in the graph 924 to determine the expected time difference 934. If the measured time difference 922 and the expected time difference 934 are the same, then there is no error in the camera timestamp 428. If these values ​​are not the same, then the one variable with the expected error is the camera timestamp 928, and therefore the camera timestamp 928 must be wrong by the difference between the measured time difference 922 and the expected time difference 934.

[0083] The error or offset in the camera timestamp may be determined for each LED flash between two peaks (such as the calculated maximum average intensity 916 and the calculated maximum average intensity 918) as the difference between the measured time difference and the expected time difference and averaged to determine a more accurate camera timestamp offset value. Additional processing may be performed on the obtained value, such as outlier removal. The sign of the error or offset will depend on the specific circumstances, such as whether the camera exposure time is longer or shorter than the LED flash period.

[0084] Fig.101 is a flowchart 1000 showing alternative steps for determining an error or offset in a camera timestamp. For purposes of illustration, the operations of flowchart 1000 are described herein as occurring serially or linearly. However, multiple operations of flowchart 1000 may occur in parallel. Additionally, the operations of flowchart 1000 need not be performed in the order shown and / or one or more blocks of flowchart 1000 need not be performed and / or may be replaced by other operations.

[0085] Flowchart 1000 begins at operation 1002, which corresponds to Figure 7 Operation 714 of . The steps in flowchart 1000 are performed by a data processor, such as data processor 202, and may also be implemented in a non-transitory computer-readable medium.

[0086] As about Figure 7 As explained in operation 714 of FIG. 1 , the data processor 202 determines the average intensity of the captured images in operation 1002. For example, this may be accomplished by averaging the intensity values ​​of the pixels that make up each image.

[0087] In operation 1004, a straight line is fitted to two consecutive sets of increasing and decreasing intensity values. Figure 8 , the first set of increasing and decreasing intensity values ​​are before and after, respectively, the calculated maximum average intensity 916 (not yet determined), and the resulting fitted lines are line 912 and line 914. The second (and consecutive) set of increasing and decreasing values ​​are on either side of the calculated maximum average intensity 918 (which is also not yet determined).

[0088] In operation 1008, the intersection of the line fitted from the first set of intensity values ​​is determined to identify the time value of the first calculated maximum average intensity (such as Fig. 9 916 in ), and determining the intersection of the line fitted according to the second set of intensity values ​​to identify the time value of the second calculated maximum average intensity (such as Fig. 9 918 in the above).

[0089] In operation 1010, the time value of the second calculated maximum average intensity is subtracted from the time value of the first calculated maximum average intensity to determine a period during which the average intensity value cycles between the two maximum values. Then, in operation 1012, a linear relationship between the expected time difference and time is determined using the expected value of zero time difference at the first calculated maximum average intensity at the beginning of the period and the value of the camera exposure frame period (the inverse of the exposure frame rate) at the second calculated maximum average intensity at the end of the period.

[0090] The actual offset between the first camera timestamp and the first LED timestamp after the first calculated maximum average intensity is determined in operation 1014. In subsequent loops, this will be the next camera timestamp and the next LED timestamp in a direction away from the first calculated maximum average intensity.

[0091] In operation 1016, an expected offset for a first / next set of camera and LED timestamps is determined using the relationship determined in operation 1012 and a time value of a related set of camera and LED timestamps, such as an average of the two timestamp values.

[0092] The difference between the expected offset determined in operation 1016 and the actual offset determined in operation 1014 is then determined in operation 1018. In operation 1020, it is determined whether the set of camera and LED timestamps is the last set before the second calculated maximum average intensity. If not, the method returns to operation 1014 and continues from there.

[0093] If it is the last group, the method proceeds to operation 1022 where a camera timestamp offset is determined using the differences determined in operation 1018. In some examples, the camera timestamp offset is determined as an average of the differences determined in operation 1018.

[0094] The resulting timestamp offset is then output.In some examples, the data processor 202 corrects the timestamps performed by the camera by subtracting the timestamp offset determined in operation 1022 according to relevant timestamp parameters and / or algorithms used for camera timestamping.

[0095] Fig.11 1 is a block diagram illustrating an example messaging system 1100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1100 includes multiple instances of user devices 228 that host several applications including a messaging client 1102 and other applications 1104. The messaging client 1102 is communicatively coupled to other instances of the messaging client 1102 (e.g., hosted on respective other user devices 228), a messaging server system 1106, and a third-party server 1108 via a network 230 (e.g., the Internet). The messaging client 1102 can also communicate with the locally hosted applications 1104 using an application program interface (API).

[0096] The messaging clients 1102 are able to communicate and exchange data with other messaging clients 1102 and with a messaging server system 1106 via the network 230. The data exchanged between the messaging clients 1102 and between the messaging clients 1102 and the messaging server system 1106 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0097] The messaging server system 1106 provides server-side functionality to a particular messaging client 1102 via the network 230. Although some functions of the messaging system 1100 are described herein as being performed by the messaging client 1102 or by the messaging server system 1106, it may be a design choice whether some functions are located within the messaging client 1102 or within the messaging server system 1106. For example, it may be technically preferred to initially deploy some technologies and functions within the messaging server system 1106, but then migrate the technologies and functions to the messaging client 1102 where the user device 228 has sufficient processing power.

[0098] The messaging server system 1106 supports various services and operations provided to the messaging clients 1102. Such operations include transmitting data to the messaging clients 1102, receiving data from the messaging clients 1102, and processing data generated by the messaging clients 1102. As examples, the data may include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1100 is activated and controlled by functions available via the user interface (UI) of the messaging client 1102.

[0099] Turning now specifically to the messaging server system 1106, an application program interface (API) server 1110 is coupled to and provides a programming interface to an application server 1114. The application server 1114 is communicatively coupled to a database server 1116, which facilitates access to a database 1120 that stores data associated with messages processed by the application server 1114. Similarly, a web server 1124 is coupled to and provides a web-based interface to the application server 1114. To this end, the web server 1124 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0100] The application program interface (API) server 1110 receives and transmits message data (e.g., commands and message payloads) between the user device 228 and the application server 1114. Specifically, the application program interface (API) server 1110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1102 to activate the functions of the application server 1114. The application program interface (API) server 1110 exposes various functions supported by the application server 1114, including: account registration; login functionality; sending messages from a particular messaging client 1102 to another messaging client 1102 via the application server 1114, sending media files (e.g., images or videos) from the messaging client 1102 to the messaging server 1112, and possible access by another messaging client 1102; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the user device 228; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1102).

[0101] The application server 1114 hosts several server applications and subsystems, including, for example, a messaging server 1112, an image processing server 1118, and a social network server 1122. The messaging server 1112 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1102. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1102. In view of the hardware requirements for processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1112.

[0102] The application server 1114 also includes an image processing server 1118 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1112.

[0103] The social network server 1122 supports various social networking functions and services and makes these functions and services available to the messaging server 1112. To this end, the social network server 1122 maintains and accesses an entity graph within the database 1120. Examples of functions and services supported by the social network server 1122 include identifying other users in the messaging system 1100 with whom a particular user has a relationship or who the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0104] The messaging client 1102 may notify the user of the user device 228 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1102 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 1102 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0105] Fig.12 1200 is a block diagram illustrating a software architecture 1204 that may be installed on any one or more of the devices described herein. The software architecture 1204 is supported by hardware, such as a machine 1202 including a processor 1220, a memory 1226, and an I / O component 1238. In this example, the software architecture 1204 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1204 includes layers such as an operating system 1212, a library 1208, a framework 1210, and an application 1206. In operation, the application 1206 invokes an API call 1250 through the software stack and receives a message 1252 in response to the API call 1250.

[0106] The operating system 1212 manages hardware resources and provides public services. The operating system 1212 includes, for example, a kernel 1214, a service 1216, and a driver 1222. The kernel 1214 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1214 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The service 1216 can provide other public services to other software layers. The driver 1222 is responsible for controlling or interfacing with the underlying hardware. For example, the driver 1222 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0107] The libraries 1208 provide low-level common infrastructure used by the applications 1206. The libraries 1208 may include system libraries 1218 (eg, C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1208 may include an API library 1224, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting graphics content in two dimensions (2D) and three dimensions (3D) on a display, GLMotif for implementing a 3D user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1208 may also include various other libraries 1228 to provide many other APIs to the application 1206.

[0108] The framework 1210 provides a high-level common infrastructure used by the applications 1206. For example, the framework 1210 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1210 can provide a wide range of other APIs that can be used by the applications 1206, some of which can be specific to a particular operating system or platform.

[0109] In an example, applications 1206 may include a home application 1236, a contacts application 1230, a browser application 1232, a shopping application 1234, a location application 1242, a media application 1244, a messaging application 1246, a game application 1248, and a variety of other applications such as third-party applications 1240. Applications 1206 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1206 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1240 (e.g., applications created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOSTM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1240 can call the API call 1250 provided by the operating system 1212 to facilitate the functions described in this article.

[0110] Fig.13 1300 in which instructions 1310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 1300 to perform any one or more of the methodologies discussed herein. For example, the instructions 1310 may cause the machine 1300 to perform any one or more of the methodologies described herein. The instructions 1310 transform a general purpose, unprogrammed machine 1300 into a specific machine 1300 that is programmed to perform the functions described and illustrated in the manner described. The machine 1300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1300 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1310 that specify actions to be taken by the machine 1300, either sequentially or otherwise. In addition, while a single machine 1300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 1310 to perform any one or more of the methodologies discussed herein.

[0111] The machine 1300 may include a processor 1302, a memory 1304, and an I / O component 1306, which may be configured to communicate with each other via a bus 1344. In an example, the processor 1302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1308 that executes instructions 1310 and a processor 1312. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.13 Multiple processors 1302 are shown, but the machine 1300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0112] The memory 1304 includes a main memory 1314, a static memory 1316, and a storage unit 1318, all of which are accessible by the processor 1302 via the bus 1344. The main memory 1304, the static memory 1316, and the storage unit 1318 store instructions 1310 that implement any one or more of the methods or functions described herein. During execution of the instructions 1310 by the networked system 200, the instructions 1310 may also reside, in whole or in part, within the main memory 1314, within the static memory 1316, within the machine-readable medium 1320, within the storage unit 1318, within one or more of the processors 1302 (e.g., within a cache memory of a processor), or within any suitable combination thereof.

[0113] I / O components 1306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 1306 may include Fig.131306. In various examples, the I / O components 1306 may include output components 1328 and input components 1332. The output components 1328 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1332 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0114] In another example, the I / O component 1306 may include: a biometric component 1334, a motion component 1336, an environmental component 1338, or a positioning component 1340, as well as various other components. For example, the biometric component 1334 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1336 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1338 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1340 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0115] A variety of technologies may be used to achieve communication. I / O components 1306 also include communication components 1342 that are operable to couple networked system 200 to network 1322 or device 1324 via coupling 1330 and coupling 1326, respectively. For example, communication components 1342 may include a network interface component or another suitable device that interfaces with network 1322. In other examples, communication components 1342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Device 1324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0116] In addition, the communication component 1342 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1342, such as a location obtained via Internet Protocol (IP) geolocation, ... Location obtained by signal triangulation, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0117] Various memories (e.g., memory 1304, main memory 1314, static memory 1316, and / or memory of processor 1302) and / or storage unit 1318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1310) when executed by processor 1302 cause various operations to implement the disclosed examples.

[0118] The instructions 1310 may be transmitted or received via a network interface device (e.g., a network interface component included in the communication component 1342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)) over the network 1322. Similarly, the instructions 1310 may be transmitted or received to the device 1324 via a coupling 1326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0119] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0120] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0121] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0122] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0123] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0124] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are covered by the term "signal media".

[0125] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0126] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0127] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

[0128] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0129] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0130] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0131] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0132] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0133] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are covered by the term "signal media".

[0134] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0135] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0136] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method of determining an image capture timestamp offset, the method comprising: generating a plurality of optical flashes at an optical flash rate; capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different than the optical flash rate, and each image of the optical flashes including an associated image timestamp; time stamping a signal associated with generation of the plurality of optical flashes; determining an intensity of each image in the set of images; determining an image having a maximum intensity in the set of images; as well as A difference between a timestamp of the image having the maximum intensity and a timestamp of a corresponding optical flash is determined to generate an image capture timestamp offset.

2. The computer-implemented method of claim 1 , wherein: The optical flash duration is approximately equal to the image exposure duration.

3. The computer-implemented method of claim 1 , further comprising: determining a first line based on an intensity and a timestamp of an image in the set of images that precedes the image in the set of images having the maximum intensity; determining a second line based on the intensity and the time stamp of an image in the set of images subsequent to the image in the set of images having the maximum intensity; determining an intersection point of the first line and the second line; as well as When determining the difference between the time stamp of the image with the maximum intensity and the time stamp of the corresponding optical flash, the time value of the intersection of the first line and the second line is used as the time stamp of the image with the maximum intensity.

4. The computer-implemented method of claim 1 , further comprising: A camera timestamp parameter is adjusted based on the image capture timestamp offset.

5. The computer-implemented method of claim 1 , wherein: The signal associated with the generation of the optical flash is received at a port of a computer processor associated with a camera that captures the set of images of the optical flash, and the computer processor timestamps the receipt of the signal associated with the generation of the optical flash.

6. The computer-implemented method of claim 1 , further comprising: Setting a timer having a duration sufficient to ensure that an image having maximum intensity is captured; as well as The set of images of the optical flash is captured during the duration of the timer.

7. The computer-implemented method of claim 1 , wherein: The image intensity for each image is the average image intensity.

8. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate, the operations comprising: generating a plurality of optical flashes at an optical flash rate; capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different than the optical flash rate, and each image of the optical flashes including an associated image timestamp; time stamping a signal associated with generation of the plurality of optical flashes; determining an intensity of each image in the set of images; determining an image having a maximum intensity in the set of images; as well as A difference between a timestamp of the image having the maximum intensity and a timestamp of a corresponding optical flash is determined to generate an image capture timestamp offset.

9. The non-transitory computer-readable storage medium of claim 8, wherein: The optical flash duration is approximately equal to the image exposure duration.

10. The non-transitory computer-readable storage medium of claim 8, wherein: The operations also include: determining a first line based on an intensity and a timestamp of an image in the set of images that precedes the image in the set of images having the maximum intensity; determining a second line based on the intensity and the time stamp of an image in the set of images subsequent to the image in the set of images having the maximum intensity; determining an intersection point of the first line and the second line; and When determining the difference between the time stamp of the image with the maximum intensity and the time stamp of the corresponding optical flash, the time value of the intersection of the first line and the second line is used as the time stamp of the image with the maximum intensity.

11. The non-transitory computer-readable storage medium of claim 8, wherein: The operations also include: A camera timestamp parameter is adjusted based on the image capture timestamp offset.

12. The non-transitory computer-readable storage medium of claim 8, wherein: The signal associated with the generation of the optical flash is received at a port of a computer processor associated with a camera that captures the set of images of the optical flash, and the computer processor timestamps the receipt of the signal associated with the generation of the optical flash.

13. The non-transitory computer-readable storage medium of claim 8, wherein: The image intensity for each image is the average image intensity.

14. The non-transitory computer-readable storage medium of claim 8, wherein: The operations also include: setting a timer having a duration sufficient to ensure that an image having maximum intensity is captured; and The set of images of the optical flash is captured during the duration of the timer.

15. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for determining an image capture timestamp offset from an optical flash generated at an optical flash rate, the operations comprising: generating a plurality of optical flashes at an optical flash rate; capturing a set of images of the plurality of optical flashes at an image capture rate, the image capture rate being different than the optical flash rate, and each image of the optical flashes including an associated image timestamp; time stamping a signal associated with generation of the plurality of optical flashes; determining an intensity of each image in the set of images; determining an image in the set of images having a maximum intensity; and A difference between a timestamp of the image having the maximum intensity and a timestamp of a corresponding optical flash is determined to generate an image capture timestamp offset.

16. The computing device of claim 15, wherein: The optical flash duration is approximately equal to the image exposure duration.

17. The computing device of claim 15, wherein: The operations also include: determining a first line based on an intensity and a timestamp of an image in the set of images that precedes the image in the set of images having the maximum intensity; determining a second line based on the intensity and the time stamp of an image in the set of images subsequent to the image in the set of images having the maximum intensity; determining an intersection point of the first line and the second line; and When determining the difference between the time stamp of the image with the maximum intensity and the time stamp of the corresponding optical flash, the time value of the intersection of the first line and the second line is used as the time stamp of the image with the maximum intensity.

18. The computing device of claim 15, wherein: The operations also include: A camera timestamp parameter is adjusted based on the image capture timestamp offset.

19. The computing device of claim 15, wherein: The signal associated with generation of the optical flash is received at a port of the processor, the processor time stamping receipt of the signal associated with generation of the optical flash.

20. The computing device of claim 15, wherein: The image intensity for each image is the average image intensity.

Images