Rolling shutter camera pipeline exposure time stamp error determination

Abstract

A method and system for determining a camera pipeline exposure timestamp error in a rolling shutter camera system is disclosed. The exposure timestamp error determination system includes a rolling shutter image sensor configured to capture an image and a processor. The processor determines a timestamp error by capturing the image using the rolling shutter image sensor, where the image includes a barcode encoded with a barcode timestamp. The processor then obtains a system exposure timestamp corresponding to when the rolling shutter image sensor captured the image and obtains the barcode timestamp by decoding the barcode. The pipeline exposure timestamp error for the rolling shutter camera system is then determined by comparing the obtained barcode timestamp to the system exposure timestamp.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 046,383, filed June 30, 2020, and U.S. Patent Application No. 17 / 204,076, filed March 17, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This topic relates to imaging systems, such as cameras used in eye-wearing devices, and more specifically to determining the camera pipeline exposure timestamp error in rolling shutter camera systems. Background Technology

[0004] Rolling shutter camera systems capture images by scanning the entire scene onto an imaging sensor (e.g., a CMOS sensor). With a rolling shutter, the top row of the imaging sensor begins exposure before the bottom row. When the exposure time is reached, the top row is read out while the other rows are still being exposed. All rows are exposed for the same duration, but they begin and end their exposures at different times. This allows the imaging sensor to collect photons in some rows during the acquisition process while reading out other rows, effectively increasing sensitivity.

[0005] Stereoscopic imaging systems utilize two or more imaging sensors to capture images from different viewpoints in order to create, for example, three-dimensional (3D) images. The captured images are synchronized in real time to produce realistic 3D images. Attached Figure Description

[0006] The accompanying drawings depict one or more specific embodiments by way of example only and not by way of limitation. In these drawings, similar reference numerals refer to the same or similar features, and letter markings are added to distinguish them. Letter markings may be omitted when referring collectively to the same or similar features, or when referring to non-specific features among the same or similar elements.

[0007] Figure 1A is a side view of an exemplary hardware configuration of an eye-wearing device used in a camera pipeline exposure timestamp error determination system.

[0008] Figure 1B is a top cross-sectional view of the right electronic housing of the temple of the eye-wearing device of Figure 1A, depicting the right visible light camera of the depth capture camera and the circuit board.

[0009] Figure 1C is a left-side view of an exemplary hardware configuration of the eye-wearing device of Figure 1A, showing the left visible light camera 1A of the depth capture camera.

[0010] Figure 1D is a top cross-sectional view of the left electronic housing of the eye-wearing device in Figure 1C, depicting the left visible light camera of the depth capture camera and the circuit board.

[0011] Figure 2A is a side view of another example hardware configuration of an eye-worn device used in a camera pipeline exposure timestamp error determination system, showing the right visible light camera and depth sensor of the depth capture camera used to generate (e.g., in the initial video) an initial depth image sequence.

[0012] Figures 2B and 2C are rear views of exemplary hardware configurations for eye-wearing devices that include two different types of image displays.

[0013] Figure 3 shows a rear cross-sectional perspective view of the eye-wearing device of Figure 2A, which depicts the infrared camera of the depth sensor, the front of the frame, the rear of the frame, and the circuit board.

[0014] Figure 4 is a cross-sectional view taken through the infrared camera and frame of the eye-wearing device in Figure 3.

[0015] Figure 5 shows a rear perspective view of the eye-wearing device of Figure 2A, depicting the infrared emitter of the depth sensor, the infrared camera of the depth sensor, the front of the frame, the rear of the frame, and the circuit board.

[0016] Figure 6 is a cross-sectional view taken through the infrared emitter and frame of the eye-wearing device in Figure 5.

[0017] Figure 7 is a block diagram depicting an exemplary camera pipeline for processing images including barcodes to determine exposure timestamp errors in a camera pipeline.

[0018] Figure 8 shows a barcode used in conjunction with the camera production line in Figure 7 to determine exposure timestamp errors.

[0019] Figure 9 is a high-level functional block diagram of an exemplary camera pipeline exposure timestamp error determination system, which includes eye-wearing devices, mobile devices, and server systems connected via various networks.

[0020] Figure 10 is a block diagram illustrating an exemplary hardware configuration of a mobile device for determining the camera pipeline exposure timestamp error of Figure 9.

[0021] Figures 11A, 11B, 11C, and 11D are flowcharts of a method for determining the exposure timestamp error in a rolling shutter camera system pipeline. Detailed Implementation

[0022] Visible light communication (VLC) is used in camera systems to determine exposure timestamp errors within the camera pipeline of the device under test (DUT, for example, due to a rolling shutter image sensor, image signal processor (ISP), operating system (OS), or imaging application). The Test Generation System (TGS) determines the current time to be transmitted to the DUT via VLC. The TGS's light source (light-emitting diode; LED) is positioned in front of the image sensor of the DUT's rolling shutter camera system. Electronics control the light source using very short exposure times (e.g., 10 microseconds) at a frequency matched to the rolling shutter rate of the rolling shutter image sensor to create an image including white and black lines. By varying the flash length, a barcode is generated by the rolling shutter image sensor as it is read; this barcode is encoded as the current time the TGS presents the flash.

[0023] The difference between the value corresponding to the time embedded in the image (i.e., the time encoded in the barcode; TGS(1)) and the value corresponding to the exposure time of the image by the rolling shutter image sensor (e.g., determined by a component of the rolling shutter camera system; DUT(1)) represents the exposure timestamp error in the rolling shutter camera pipeline. When the clocks of the DUT and TGS are in the same time domain (i.e., starting operation from the same clock or a synchronized clock; DUT(2) = TGS(2)), the pipeline exposure timestamp error is the difference between the time embedded in the image and the exposure time of the image (e.g., TGS(1) - DUT(1)). When the clocks of the DUT and TGS are in different time domains (i.e., DUT(2) <> TGS(2)), the pipeline exposure timestamp error is the difference between the first difference at TGS (TGS(2) - TGS(1)) and the second difference at DUT (DUT(2) - DUT(1)). Generally, computer vision (CV) and augmented reality systems improve as errors decrease. Furthermore, in stereo systems with multiple cameras, more realistic effects can be achieved by reducing exposure timestamp errors and by ensuring similar exposure timestamp error times for each of the multiple cameras. Therefore, identifying such errors in the system pipeline during the design phase is useful for component selection and for compensating for delays / errors in the final product.

[0024] In this specific embodiment, numerous specific details are illustrated by way of example to provide a thorough understanding of the relevant teachings. However, it will be apparent to those skilled in the art that the teachings can be practiced without such details. In other instances, well-known methods, processes, components, and circuits are described at a higher level without detail to avoid unnecessarily obscuring various aspects of the teachings.

[0025] As used herein, the terms “coupled” or “connected” refer to any logical, optical, physical, or electrical connection, link, etc., through which electrical or magnetic signals generated or provided by one system element are transmitted to another coupled or connected system element. Unless otherwise described, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media that can be modified, manipulated, or carry signals. The term “on” means that the element is directly supported by the element or indirectly supported by another element integrated into or supported by the element.

[0026] For purposes of illustration and discussion, the orientation of eye-wearing devices, associated components, and any complete device incorporating a depth capture camera, as shown in any of the accompanying figures, is given by way of example only. In operation for camera misalignment compensation, the eye-wearing device may be oriented in any other direction suitable for the application of the eye-wearing device, such as up, down, sideways, or any other orientation. Furthermore, for the purposes of this document, any directional terms such as front, back, inside, outside, towards, left, right, sideways, longitudinal, up, down, high, low, top, bottom, side, horizontal, vertical, and diagonal are used by way of example only and are not limited to the orientation or orientation of any depth capture camera or components of a depth capture camera constructed as otherwise described herein.

[0027] Other objects, advantages, and novel features of the examples will be set forth in part in the detailed description below, and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings, or may be learned by the generation or operation of the examples. The objects and advantages of this subject matter may be realized and achieved by means of the methods, means, and combinations particularly pointed out in the appended claims.

[0028] Now refer in detail to the accompanying drawings and the examples discussed below.

[0029] As shown in Figures 1A and 1B, the eye-wearing device 100 includes a right visible light camera 114B. The eye-wearing device 100 may include multiple visible light cameras, such as a right visible light camera 114A (Figures 1A and 1B) and a left visible light camera 114B (Figures 1C and 1D), which form a passive type of depth acquisition camera, such as a stereo camera, wherein the right visible light camera 114B is located on the right electronic housing 110B and the left visible light camera 114A is located on the left electronic housing 110A.

[0030] Left visible light camera 114A and right visible light camera 114B are sensitive to wavelengths within the visible light range. Each of the visible light cameras 114A and 114B has a different forward field of view, which overlap to allow the generation of a three-dimensional depth image; for example, the right visible light camera 114B has a depicted right field of view 111B. Generally, a "field of view" is a portion of a scene that is visible in terms of location and orientation in space by the camera. When the visible light cameras capture images, objects or object features outside the fields of view 111A, 111B are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or amplitude of electromagnetic radiation picked up by the image sensors of the visible light cameras 114A and 114B in an image of a given scene. The field of view can be expressed as the angular size of the view frustum, i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0031] In the examples, visible light cameras 114A and 114B have a field of view between 15° and 110° (e.g., 24°) and a resolution of 480×480 pixels or greater. The “coverage angle” describes the angular range within which the lens of the visible light cameras 114A and 114B or the infrared camera 220 (see Figure 2A) can effectively image. Typically, the imaging circle produced by the camera lens is large enough to completely cover the film or sensor, possibly including some degree of vignetting (i.e., reduced image brightness or saturation at the periphery compared to the image center). If the camera lens’s coverage angle does not extend throughout the sensor, the imaging circle will be visible, typically with strong vignetting towards the edges, and the effective field of view will be limited to the coverage angle.

[0032] Examples of such visible light cameras 114A and 114B include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640 × 480 pixels, totaling 0.3 megapixels), 720p, or 1080p. As used herein, the term “overlap” in relation to the field of view means that the pixel matrix in the generated raw or infrared image of the scene overlaps by 30% or more. As used herein, the term “substantially overlap” in relation to the field of view means that the pixel matrix in the generated raw or infrared image of the scene overlaps by 50% or more. Suitable visible light cameras 114 include complementary metal-oxide-semiconductor (CMOS) sensor cameras with rolling shutter readings. In one example, camera 114 includes a V blanking time setting to minimize the time difference T between feature points acquired by two separate cameras. In another example, camera 114 (such as a camera purchased from Sony Corporation, Minato Ward, Japan) includes an exposure delay setting that counts in the sensor rows, based on the readings of one of the rear cameras, so that feature points falling on different rows are exposed substantially simultaneously. Other suitable cameras will be understood by those skilled in the art from the description herein.

[0033] Image sensor data from visible light cameras 114A and 114B, along with geographic location data, are captured, digitized by an image processor, and stored in memory. The captured left and right raw images, captured by the respective visible light cameras 114A and 114B, are in a two-dimensional spatial domain and include a pixel matrix in a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes color attributes (e.g., red pixel light value, green pixel light value, and blue pixel light value); and location attributes (e.g., X-coordinate and Y-coordinate).

[0034] To provide a stereoscopic view, visible light cameras 114A and 114B can be coupled to an image processor (feature 912, FIG. 9) for digital processing, while adding timestamps to the exposed or captured scene images. Image processor 912 includes circuitry for receiving signals from visible light cameras 114A and 114B and processing those signals from the visible light cameras 114A into a format suitable for storage in memory. Timestamps can be added by the image processor or another processor that controls the operation of visible light cameras 114A and 114B. Visible light cameras 114A and 114B allow depth capture cameras to simulate human binocular vision. Depth capture cameras provide the ability to reconstruct three-dimensional images based on two captured images from visible light cameras 114A and 114B with the same timestamp. Such three-dimensional images allow for immersive and realistic experiences, for example, for virtual reality or video games. Three-dimensional depth video can be generated by stitching together a series of three-dimensional depth images with associated time coordinates of the presentation time in the depth video.

[0035] For stereoscopic vision, at any given moment, a pair of raw red, green, and blue (RGB) images of a scene—one captured by each of the left visible light camera 114A and the right visible light camera 114B—are captured. When (e.g., by an image processor) the pair of captured raw images from the left visible light camera 114A and the right visible light camera 114B's forward left field of view 111A and forward right field of view 111B, a depth image is generated, and the generated depth image can be perceived by a user on optical components 180A and 180B or (e.g., on a mobile device) other image displays. The generated depth image is in a three-dimensional spatial domain and may include a vertex matrix in a three-dimensional position coordinate system, which includes an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth positioning (e.g., distance).

[0036] Depth video also associates each of the generated depth images in a series with a temporal coordinate on the time (T) axis of the presentation time in the depth video (e.g., each depth image includes both spatial and temporal components). Depth video may also include one or more input parameter components (e.g., audio components such as audio tracks or audio streams, biometric components such as heart rate maps, etc.), which may be captured by input devices such as microphones or heart rate monitors. Each vertex includes color attributes (e.g., red pixel light values, green pixel light values, and blue pixel light values); localization attributes (e.g., X-coordinates, Y-coordinates, and Z-coordinates); texture attributes; reflectance attributes; or combinations thereof. Texture attributes quantify the perceptual texture of the depth image, such as the spatial arrangement of color or intensity in the vertex regions of the depth image.

[0037] Typically, depth perception is derived from the parallax of a given 3D point in the left and right raw images captured by visible light cameras 114A and 114B. Parallax is the difference in image position (d = x) when the same 3D point is projected from the viewpoints of the visible light cameras 114A and 114B. 左 -x 右 For an image with a parallel optical axis, focal length f, baseline B, and corresponding image point (x) 左 ,y 左 ) and (x 右 ,y 右 The positions (Z-axis coordinates) of 3D points can be derived using visible light cameras 114A and 114B, which determine depth based on parallax. Typically, the depth of a 3D point is inversely proportional to parallax. Various other techniques can also be used.

[0038] In the example, the camera pipeline exposure timestamp error determination system includes an eye-worn device 100. The eye-worn device 100 includes a frame 105, a left temple 125A extending from the left side 170A of the frame 105, and a right temple 125B extending from the right side 170B of the frame 105. The eye-worn device 100 also includes a depth capture camera. The depth capture camera includes: (i) at least two visible light cameras with overlapping fields of view; or (ii) at least one visible light camera 114A and 114B and a depth sensor (feature 213 in FIG. 2A). In one example, the depth capture camera includes a left visible light camera 114A with a left field of view 111A attached to the frame 105 or the left temple 125A to capture a left image of the scene. The eye-wearing device 100 also includes a right visible light camera 114B with a right field of view 111B connected to the frame 105 or the right temple 125B to capture (e.g., simultaneously with the left visible light camera 114A) a right image of the scene that partially overlaps with the left image.

[0039] The camera pipeline exposure timestamp error determination system also includes computing devices, such as a host computer coupled via a network to the eye-wearing device 100 (e.g., mobile device 990 in Figures 9 and 10). The system also includes an image display (optical components 180A and 180B of the eye-wearing device; image display 1080 of the mobile device 990 in Figure 10) for presenting (e.g., displaying) a video including the image. The system also includes an image display driver (element 942 of the eye-wearing device 100 in Figure 9; element 1090 of the mobile device 990 in Figure 10) coupled to the image display (optical components 180A and 180B of the eye-wearing device; image display 1080 of the mobile device 990 in Figure 10) to control the image display to present the initial video.

[0040] In some examples, user input is received to indicate the image the user expects to capture. For example, the camera pipeline exposure timestamp error determination system also includes a user input device for receiving user input. Examples of user input devices include a touch sensor (element 991 of the eye-wear device 100 in FIG. 9), a touchscreen display (element 1091 of the mobile device 1090 in FIG. 10), a vision inspection system (e.g., including machine vision, V, for processing images collected by one or more visible light cameras 114A), and a computer mouse for a personal computer or laptop. The camera pipeline exposure timestamp error determination system also includes a processor (element 932 of the eye-wear device 100 in FIG. 9; element 1030 of the mobile device 990 in FIG. 10) and a depth capture camera coupled to the eye-wear device 100. The camera pipeline exposure timestamp error determination system also includes processor-accessible memory (element 934 of the eye-wearing device 100 in FIG. 9; elements 1040A-B of the mobile device 990 in FIG. 10) and programs (element 945 of the eye-wearing device 100 in FIG. 9; element 945 of the mobile device 990 in FIG. 10) in the eye-wearing device 100 itself, the mobile device (element 990 in FIG. 9), or another part of the camera pipeline exposure timestamp error determination system (e.g., the server system 998 in FIG. 9).

[0041] In one example, the processor (element 932 of FIG. 9) executes a camera pipeline exposure timestamp error procedure (element 945 of FIG. 9) to configure the eye-wearing device 100 to determine the pipeline exposure timestamp error when processing an image. In another example, the processor (element 945 of FIG. 10) executes a camera pipeline exposure timestamp error procedure (element 945 of FIG. 10) to configure the mobile device (element 990 of FIG. 10) of the camera pipeline exposure timestamp error determination system to determine the pipeline exposure timestamp error when processing an image.

[0042] Figure 1B is a top cross-sectional view of the right electronic housing 110B of the eyewear device 100 of Figure 1A, depicting the right visible light camera 114B of the depth capture camera and the circuit board. Figure 1C is a left-side view of an exemplary hardware configuration of the eyewear device 100 of Figure 1A, showing the left visible light camera 114A of the depth capture camera. Figure 1D is a top cross-sectional view of the left electronic housing 110A of the eyewear device of Figure 1C, depicting the left visible light camera 114A of the depth capture camera and the circuit board. Except for the connection and coupling located on the left side 170A, the structure and arrangement of the left visible light camera 114A are substantially similar to those of the right visible light camera 114B. As shown in the example of Figure 1B, the eyewear device 100 includes the right visible light camera 114B and a circuit board, which may be a flexible printed circuit board (PCB) 140B. A right hinge 126B connects the right electronic housing 110B to the right temple 125B of the eyewear device 100. In some examples, components such as the right visible light camera 114B, flexible PCB 140B, or other electrical connectors or contacts may be located on the right temple 125B or the right hinge 126B.

[0043] The right electronic housing 110B includes an electronic housing body 211 and an electronic housing cover, wherein the electronic housing cover is omitted in the cross-section of FIG1B. Inside the right electronic housing 110B are various interconnected circuit boards, such as PCBs or flexible PCBs, which include components for the right visible light camera 114B, a microphone, and low-power wireless circuitry (e.g., for use via Bluetooth). TM Controller circuits for short-range wireless network communication and high-speed wireless circuits (e.g., for wireless LAN communication via Wi-Fi).

[0044] The right visible light camera 114B is coupled to or disposed on the flexible PCB 240 and covered by a visible light camera overlay lens, which is aimed through an opening formed in the frame 105. For example, the right edge 107B of the frame 105 is connected to the right electronic housing 110B and includes an opening for the visible light camera overlay lens. The frame 105 includes a forward-facing side configured to face outwards away from the user's eye. The opening for the visible light camera overlay lens is formed on and extends through the forward-facing side. In this example, the right visible light camera 114B has an outward-facing field of view 111B in the line of sight or viewing angle of the user's right eye in the eyewear device 100. The visible light camera overlay lens may also be adhered to the outward-facing surface of the right electronic housing 110B, wherein the opening is formed with an outward-facing coverage angle but facing a different outward direction. Coupling may also be achieved indirectly via an intermediary member.

[0045] A left (first) visible light camera 114A is connected to the left image display of the left optical assembly 180A and captures the left-eye viewing scene as observed by the wearer of the eye-wearing device 100 in the left raw image. A right (second) visible light camera 114B is connected to the right image display of the right optical assembly 180B and captures the right-eye viewing scene as observed by the wearer of the eye-wearing device 100 in the right raw image. The left and right raw images are partially overlapped for use in presenting the three-dimensional observable space of the generated depth image.

[0046] The flexible PCB 140B is disposed inside the right electronic housing 110B and coupled to one or more other components housed in the right electronic housing 110B. Although shown as being formed on a circuit board on the right electronic housing 110B, the right visible light camera 114B may be formed on a circuit board on the left electronic housing 110A, temples 125A, 125B, or frame 105.

[0047] Figure 2A is a side view of another exemplary hardware configuration of the eye-worn device 100 used in a camera pipeline exposure timestamp error determination system. As shown, the depth capture camera includes a left visible light camera 114A and a depth sensor 213 on frame 105 to generate an initial depth image (e.g., in an initial video) of a series of initial depth images. Here, a single visible light camera 114A and depth sensor 213 are used to generate the depth image, rather than using at least two visible light cameras 114A and 114B. The infrared camera 220 of depth sensor 213 has an outward-facing field of view that substantially overlaps with the left visible light camera 114A in relation to the user's eye. As shown, infrared emitter 215 and infrared camera 220 are located together with the left visible light camera 114A on the upper part of the left frame 107A.

[0048] In the example of Figure 2A, the depth sensor 213 of the eye-worn device 100 includes an infrared emitter 215 and an infrared camera 220 for capturing infrared images. Visible light cameras 114A and 114B typically include a blue light filter to block infrared light detection. In this example, the infrared camera 220 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640 × 480 pixels, 0.3 megapixels in total), where the blue light filter has been removed. The infrared emitter 215 and the infrared camera 220 are located together on the frame 105, for example, both are shown attached to the upper portion of the left frame 107A. As described in further detail below, one or more of the frame 105 or the left electronic housing 110A and the right electronic housing 110B include a circuit board that includes the infrared emitter 215 and the infrared camera 220. The infrared emitter 215 and the infrared camera 220 can be connected to the circuit board, for example, by soldering.

[0049] Other arrangements of the infrared emitter 215 and infrared camera 220 may be implemented, including arrangements where the infrared emitter 215 and infrared camera 220 are both on the right edge 107A, or in different locations on the frame 105, for example, the infrared emitter 215 is on the left edge 107B and the infrared camera 220 is on the right edge 107B. However, the at least one visible light camera 114A and depth sensor 213 typically have substantially overlapping fields of view to generate a three-dimensional depth image. In another example, the infrared emitter 215 is on the frame 105 and the infrared camera 220 is on one of the electronic housings 110A and 110B, or vice versa. The infrared emitter 215 may be substantially connected at any location on the frame 105, the left electronic housing 110A, or the right electronic housing 110B to emit an infrared pattern within the user's line of sight. Similarly, the infrared camera 220 can be attached to virtually any location on the frame 105, the left electronic housing 110A, or the right electronic housing 110B to capture at least one reflection variation in the infrared light emission pattern of a three-dimensional scene within the user's line of sight.

[0050] Infrared emitter 215 and infrared camera 220 are arranged to face outwards to capture infrared images of a scene containing objects or object features observed by a user wearing eyewear 100. For example, infrared emitter 215 and infrared camera 220 are positioned directly in front of the eyes, on the upper part of frame 105, or in electronic housings 110A and 110B at either end of frame 105, having a forward field of view to capture images of the scene the user is looking at for measuring object depth or object features.

[0051] In one example, the infrared emitter 215 of the depth sensor 213 emits infrared illumination in the forward field of view of the scene. This infrared illumination can be near-infrared light or other short-wavelength beams of low-energy radiation. Alternatively or additionally, the depth sensor 213 may include an emitter that emits light of wavelengths other than infrared, and the depth sensor 213 further includes a camera sensitive to that wavelength, which receives and captures images having that wavelength. As described above, the eye-wearing device 100 is coupled to a processor and memory, for example, in the eye-wearing device 100 itself or in another part of a camera pipeline exposure timestamp error determination system. The eye-wearing device 100 or the camera pipeline exposure timestamp error determination system can then process the captured infrared images to generate a three-dimensional depth image of the depth video (such as an initial depth image from the initial video).

[0052] Figures 2B and 2C are rear views of an exemplary hardware configuration of an eye-wearing device 100 including two different types of image displays. The eye-wearing device 100 is in the form of glasses configured for wear by a user. The eye-wearing device 100 may take other forms and may be combined with other types of frames, such as headbands, headphones, or helmets.

[0053] In the example of eyeglasses, the eye-wearing device 100 includes a frame 105 comprising a left edge 107A connected to the right edge 107B via a nose bridge 106 adapted to fit the user's nose. The left and right edges 107A-B include corresponding apertures 175A and 175B, which house corresponding optical elements 180A and 180B, such as lenses and display devices. As described herein, the term "lens" refers to a sheet of transparent or translucent glass or plastic having a curved or flat surface that causes light to converge / diverge or to cause little or no convergence or divergence.

[0054] Although shown as having two optical elements 180A and 180B, the eyewear device 100 may include other arrangements (such as a single optical element), or may not include any optical elements 180A and 180B, depending on the application of the eyewear device 100 or the intended user. As further shown, the eyewear device 100 includes a left electronic housing 110A (which includes a left camera 114A) adjacent to the left side 170A of the frame 105 and a right electronic housing 110B (which includes a right camera 114B) adjacent to the right side 170B of the frame 105. The electronic housings 110A and 110B may be integrated into the respective sides 170A and 170B of the frame 105 (as shown) or implemented as separate components attached to the respective sides 170A and 170B of the frame 105. Alternatively, the electronic housings 110A and 110B may be integrated into a temple (not shown) connected to the frame 105.

[0055] In one example, the image display of optical components 180A and 180B includes an integrated image display. As shown in Figure 2B, optical components 180A and 180B include any suitable type of suitable display matrix 170, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other such display. Optical components 180A and 180B also include one or more optical layers 176, which may include lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components and any combination thereof.

[0056] Optical layers 176A-N may include prisms having suitable dimensions and construction and including a first surface for receiving light from a display matrix and a second surface for emitting light toward a user's eye. The prisms of optical layers 176A-N extend over all or part of apertures 175A, 175B formed in the left and right edges 107A-B to allow the user to see the second surface of the prism when viewing through the corresponding left and right edges 107A-B. The first surface of the prisms of optical layers 176A-N faces upward from the frame 105, and the display matrix covers the prisms such that photons and light emitted by the display matrix illuminate the first surface. The size and shape of the prisms are designed such that light is refracted within the prisms and directed to the user's eye by the second surface of the prisms of optical layers 176A-N. In this respect, the second surface of the prisms of optical layers 176A-N may be convex to direct light toward the center of the eye. The prism can be selectively designed in size and shape to magnify the image projected by the display matrix 170, and light passes through the prism such that the image viewed from the second surface is larger than the image emitted from the display matrix 170 in one or more dimensions.

[0057] In another example, the image display device of optical components 180A and 180B includes a projected image display as shown in FIG2C. Optical components 180A and 180B include a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source (such as the laser projector 150) is positioned within or above one of the temples 125A and 125B of the eyewear device 100. Optical components 180A and 180B include one or more optical strips 155A-N, which are spaced apart across the width of the lens of optical components 180A and 180B or across the depth of the lens between the front and rear surfaces of the lens.

[0058] When photons projected by laser projector 150 travel through the lenses of optical components 180A and 180B, they encounter optical strips 155A-N. Upon encountering an optical strip, the photon is either redirected to the user's eye or propagated to the next optical strip. A combination of modulation of laser projector 150 and modulation of the optical strips can control specific photons or beams. In this example, the processor controls optical strips 155A-N by emitting mechanical, acoustic, or electromagnetic signals. Although shown as having two optical components 180A and 180B, the eye-wearing device 100 may include other arrangements, such as a single or three optical components, or optical components 180A and 180B may have different arrangements, depending on the application of the eye-wearing device 100 or the intended user.

[0059] As further shown in Figures 2B and 2C, the electronic housings 110A and 110B can be integrated into the corresponding sides 170A and 170B of the frame 105 (as shown) or implemented as separate components attached to the corresponding sides 170A and 170B of the frame 105. Alternatively, the electronic housings 110A and 110B can be integrated into the temples 125A and 125B attached to the frame 105.

[0060] In one example, the image display includes a first (left) image display and a second (right) image display. The eye-wearing device 100 includes a first aperture 175A and a second aperture 175B, which respectively accommodate a first optical component 180A and a second optical component 180B. The first optical component 180A includes a first image display (e.g., display matrix 170A of FIG. 2B; optical strips 155A-N' and projector 150A of FIG. 2C). The second optical component 180B includes a second image display (e.g., display matrix 170B of FIG. 2B; optical strips 155A-N' and projector 150B of FIG. 2C).

[0061] Figure 3 shows a rear cross-sectional perspective view of the eye-wearing device of Figure 2A, depicting an infrared camera 220, a front frame 330, a rear frame 335, and a circuit board. The upper portion of the left edge 107A of the frame 105 of the eye-wearing device 100 includes the front frame 330 and the rear frame 335. The front frame 330 includes a forward-facing side configured to face outwards away from the user's eye. The rear frame 335 includes a rearward-facing side configured to face inwards towards the user's eye. An opening for the infrared camera 220 is formed in the front frame 330.

[0062] As shown in the cross-section 4-4 circled in the upper middle portion of the left edge 107A of frame 105, a circuit board, namely a flexible printed circuit board (PCB) 340, is sandwiched between the front portion 330 and the rear portion 335 of the frame. The left electronic housing 110A is also shown in further detail attached to the left temple 325A via the left hinge 126A. In some examples, components of the depth sensor 213 include an infrared camera 220, the flexible PCB 340, or other electrical connectors or contacts located on the left temple 325A or the left hinge 126A.

[0063] In one example, the left electronic housing 110A includes an electronic housing body 311, an electronic housing cover 312, an inward-facing surface 391, and an outward-facing surface 392 (marked but not visible). Set within the left electronic housing 110A are various interconnected circuit boards, such as PCBs or flexible PCBs, which include controller circuitry for charging the battery, inward-facing light-emitting diodes (LEDs), and outward-facing (forward-facing) LEDs. Although shown as formed on a circuit board on the left edge 107A, a depth sensor 213, including an infrared emitter 215 and an infrared camera 220, could be formed on a circuit board on the right edge 107B to, for example, combine with a right visible light camera 114B to capture infrared images for generating three-dimensional depth images or depth videos.

[0064] Figure 4 is a cross-sectional view through the infrared camera 220 and the frame, corresponding to the circled cross-section 4-4 of the eye-wear device in Figure 3. Various layers of the eye-wear device 100 are visible in the cross-section of Figure 4. As shown, a flexible PCB 340 is disposed on the rear portion 335 of the frame and connected to the front portion 330 of the frame. The infrared camera 220 is disposed on the flexible PCB 340 and covered by an infrared camera cover lens 445. For example, the infrared camera 220 is soft-soldered to the back side of the flexible PCB 340. Soft-soldering attaches the infrared camera 220 to electrical contact pads formed on the back side of the flexible PCB 340 by subjecting the flexible PCB 340 to controlled heating, which melts solder paste to connect the two components. In one example, soft-soldering is used to surface-mount the infrared camera 220 onto the flexible PCB 340 and electrically connect the two components. However, through-holes can be used to connect wires from the infrared camera 220 to the flexible PCB 340 via, for example, interconnects.

[0065] The front portion 330 of the frame includes an infrared camera opening 450 for an infrared camera cover lens 445. The infrared camera opening 450 is formed on a forward-facing side of the front portion 330, which is configured to face outwards away from the user's eye and towards the scene being observed by the user. In this example, the flexible PCB 340 can be attached to the rear portion 335 of the frame via a flexible PCB adhesive 460. The infrared camera cover lens 445 can be attached to the front portion 330 of the frame via an infrared camera cover lens adhesive 455. The connection can also be achieved indirectly via an intermediary component.

[0066] Figure 5 shows a rear view of the eye-wearing device of Figure 2A. The eye-wearing device 100 includes an infrared emitter 215, an infrared camera 220, a front frame 330, a rear frame 335, and a circuit board 340. As shown in Figure 3, the upper portion of the left edge of the frame of the eye-wearing device 100 includes the front frame 330 and the rear frame 335. An opening for the infrared emitter 215 is formed in the front frame 330.

[0067] As shown in the cross-section 6-6 circled in the upper middle portion of the left edge of the frame, the circuit board, i.e., the flexible PCB 340, is sandwiched between the front portion 330 and the rear portion 335 of the frame. The left electronic housing 110A is also shown in further detail attached to the left temple 325A via the left hinge 126A. In some examples, components of the depth sensor 213 include an infrared emitter 215, the flexible PCB 340, or other electrical connectors or contacts located on the left temple 325A or the left hinge 126A.

[0068] Figure 6 is a cross-sectional view through the infrared emitter 215 and the frame, corresponding to the circled cross-section 6-6 of the eye-wearing device in Figure 5. The cross-section of Figure 6 shows multiple layers of the eye-wearing device 100. As shown, the frame 105 includes a front frame portion 330 and a rear frame portion 335. A flexible PCB 340 is disposed on the rear frame portion 335 and connected to the front frame portion 330. The infrared emitter 215 is disposed on the flexible PCB 340 and covered by an infrared emitter-covered lens 645. For example, the infrared emitter 215 is soft-welded to the back side of the flexible PCB 340. Soft-welding attaches the infrared emitter 215 to contact pads formed on the back side of the flexible PCB 340 by subjecting the flexible PCB 340 to controlled heating, which melts solder paste to connect the two components. In one example, soft-welding is used to surface-mount the infrared emitter 215 onto the flexible PCB 340 and electrically connect the two components. However, the through-hole can be interconnected to connect the leads from the infrared emitter 215 to the flexible PCB 340.

[0069] The front portion 330 of the frame includes an infrared emitter opening 650 for an infrared emitter cover lens 645. The infrared emitter opening 650 is formed on a forward-facing side of the front portion 330, which is configured to face outwards away from the user's eye and towards the scene being observed by the user. In this example, the flexible PCB 340 can be attached to the rear portion 335 of the frame via a flexible PCB adhesive 460. The infrared emitter cover lens 645 can be attached to the front portion 330 of the frame via an infrared emitter cover lens adhesive 655. Coupling can also be achieved indirectly via an intermediary component.

[0070] Figure 7 is a block diagram depicting an exemplary camera system 704, which includes components for processing images including time-coded barcodes to determine exposure timestamp errors of the camera system 704. The camera system 704 includes an image sensor 114, an ISP 706, an OS 708, and an imaging application 710, which constitute a pipeline 712 for capturing and processing images.

[0071] Light source 702 generates a light pattern presented to image sensor 114 under the control of processor 714 (e.g., processor 932 or an external processor of the test generation system TGS). Light source 702 may be an LED emitting white light or other light that can be detected by image sensor 114. Processor 714 is configured to time-modulate the light source (e.g., via pulse width modulation PWM) at the beginning of light source 702 presenting the light pattern for exposure by image sensor 114.

[0072] A light diffuser (not shown) made of a conventional translucent material can be positioned between the light source 702 and the image sensor 114 to diffuse light from the light source 702, thereby producing uniform light coverage over the entire imaging surface of the image sensor 114. The light diffuser and its proximity to the light source 702 also shield other image information.

[0073] The light source 702 is driven by conventional electronics and has a relatively fast response time (i.e., the speed at which the light source turns on and off when the power is switched on and off by the electronics). By setting the exposure time of the image sensor 114 to be very short (e.g., 10 microseconds per line), multiple lines of information can be transmitted in a single image.

[0074] Image sensor 114 includes a sensor array on a semiconductor chip (CMOS). When light shines on the sensor array, the chip exposes an image by converting the resulting signal into image pixels to obtain a raw image. In one example, the chip may perform additional processing, such as signal conditioning. In one example, image sensor 114 adds a timestamp (exposure timestamp) corresponding to the time the chip initiates the exposure to the raw image.

[0075] The ISP 706 transforms images and image formats. The ISP 706 can be a standalone component or integrated into a system-on-a-chip (SoC). The ISP 706 performs various transformations on images and image formats, including but not limited to one or more of Bayer transform, demosaicing, noise reduction, image sharpening, focusing, exposure adjustment, and white balance. Those skilled in the art will understand suitable ISP 706 features from the description herein. In one example, as an addition to or replacement for image sensor 114, the ISP 706 adds a timestamp corresponding to the time the image was exposed (exposure timestamp).

[0076] OS 708 provides an interface between internal and external hardware. In one example, OS 708 provides an interface to the Hardware Abstraction Layer (HAL) on the SoC, which includes ISP 706 and image sensor 114, and is the first software to receive arriving images. OS 708 also provides notification to imaging application 710 that the image is available. In one example, as an addendum to or replacement for image sensor 114 or ISP 706, OS 708 adds a timestamp (exposure timestamp) corresponding to the time the image was exposed.

[0077] Imaging application 710 receives and uses the image. Imaging application 710 is a software component configured to process images for purposes such as display, modification, and storage. In one example, the imaging application adds a timestamp (application timestamp) corresponding to the time when the image becomes available.

[0078] Figure 8 is an image 802 including a time-coded barcode 804, used in conjunction with the camera system 704 of Figure 7 to determine exposure timestamp errors in the pipeline 712. The barcode 804 shown includes a 14-bit binary value comprising a relatively long light pulse 806a indicating a start bit, followed by fourteen shorter pulses 808 encoding the timestamp. The relatively short pulses include a shorter pulse 808a representing 1 and a longer pulse 808b representing 0. Each encoded timestamp begins with a start bit 806 corresponding to that timestamp. As shown, the barcode 804 can be appended after the bit 812 of a previous timestamp and before another barcode beginning with another start bit 806b.

[0079] Figure 9 is a high-level functional block diagram of an exemplary camera pipeline exposure timestamp error determination system 900, which includes wearable devices (e.g., eye-wearing device 100), mobile devices 990, and server systems 998 connected via various networks. The eye-wearing device 100 includes an input parameter processor and a depth capture camera, such as at least one of visible light cameras 114A and 114B; and a depth sensor 213, shown as an infrared emitter 215 and an infrared camera 220. The depth capture camera may optionally include at least two visible light cameras 114A and 114B (one associated with the left side 170A and one associated with the right side 170B). The depth capture camera generates an initial depth image of the initial video, which is a rendered 3D model as a texture-mapped image of a red-green-blue (RGB) imaging scene. Transformation functions within the wearable device correct the initial image, for example, to facilitate feature matching and format the image for viewing.

[0080] Mobile device 990 may be a smartphone, tablet, laptop, any access point, or other such device capable of connecting to eye-wearing device 100 using both low-power wireless connection 925 and high-speed wireless connection 937. Mobile device 990 connects to server system 998 and network 995. Network 995 may include any combination of wired and wireless connections.

[0081] The eye-wear device 100 further includes two image displays (one associated with the left side 170A and the other with the right side 170B) of optical components 180A and 180B. The eye-wear device 100 also includes an image display driver 942, an image processor 912, low-power circuitry 920, and high-speed circuitry 930. The image displays of optical components 180A and 180B are used to present images and video, and may include a series of depth images, such as an initial depth image from an initial video. The image display driver 942 is coupled to the image displays of optical components 180A and 180B to control the image displays of optical components 180A and 180B to present video including images (such as the initial depth image of the initial video). The eye-wear device 100 further includes a user input device 991 (e.g., a touch sensor) to receive input and selections from a user.

[0082] The components for the eye-wearing device 100 shown in Figure 9 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the rim or temple. Alternatively or additionally, the depicted components may be located in the electronic housing, frame, hinge, or nose bridge of the eye-wearing device 100. The left visible light camera 114A and the right visible light camera 114B may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, lenses, or any other corresponding visible or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0083] The eye-worn device 100 includes a memory 934, which includes an input parameter program and a camera pipeline exposure timestamp error determination program 945, to perform all or a subset of the functions described herein for camera deviation compensation.

[0084] As shown in the figure, the eye-wearing device 100 has orientation sensors, including, for example, an inertial measurement unit (IMU) 972 as shown. Typically, the inertial measurement unit 972 is an electronic device that uses a combination of accelerometers and gyroscopes to measure and report the body's specific force, angular rate, and sometimes also uses a magnetometer to measure and report the magnetic field around the body. In this example, the inertial measurement unit 972 determines the head orientation of the wearer of the eye-wearing device 100, which is related to the camera orientation of the depth capture camera of the eye-wearing device 100 when capturing an associated depth image. The inertial measurement unit 972 operates by detecting linear acceleration using one or more accelerometers and detecting rotational rates using one or more gyroscopes. A typical configuration of the inertial measurement unit includes one accelerometer, one gyroscope, and one magnetometer for each of the following three axes: a horizontal axis (X) for left-right movement, a vertical axis (Y) for top-bottom movement, and a depth or distance axis (Z) for up-down movement. The gyroscope detects the gravity vector. The magnetometer defines rotation in a magnetic field (e.g., facing south, facing north, etc.), much like a compass generates a heading reference. Three accelerometers detect acceleration along the horizontal (X), vertical (Y), and depth (Z) axes defined above, which can be defined relative to the ground, the eye-wearing device 100, the depth-capture camera, or the user wearing the eye-wearing device 100.

[0085] The memory 934 includes head orientation measurements, which correspond to principal axis measurements on the horizontal (X-axis), vertical (Y-axis), and depth or distance (Z-axis) axes tracked (e.g., measured) by the inertial measurement unit 972. The head orientation measurements are used to determine the alignment of the depth acquisition camera, which can be used to identify the bottom plane of the initial depth image. In some IMU applications, the principal axes are referred to as the pitch axis, roll axis, and yaw axis.

[0086] Figures 11A to 11D show flowcharts outlining the functions that can be implemented in the camera pipeline exposure timestamp error determination procedure 945.

[0087] As shown in Figure 9, the high-speed circuit 930 includes a high-speed processor 932, a memory 934, and a high-speed wireless circuit 936. In this example, an image display driver 942 is coupled to the high-speed circuit 930 and operated by the high-speed processor 932 to drive the left and right image displays of the optical components 180A and 180B. The high-speed processor 932 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the eye-wear device 100. The high-speed processor 932 includes the processing resources required to manage high-speed data transmission over a high-speed wireless connection 937 to a wireless local area network (WLAN) using the high-speed wireless circuit 936. In some examples, the high-speed processor 932 executes an operating system, such as the LINUX operating system or other such operating system for the eye-wear device 100, and the operating system is stored in the memory 934 for execution. Among other responsibilities, the high-speed processor 932, which executes the software architecture of the eye-wear device 100, also manages the data transmission utilizing the high-speed wireless circuit 936. In some examples, the high-speed wireless circuit 936 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit 936 may implement other high-speed communication standards.

[0088] The low-power wireless circuit 924 and high-speed wireless circuit 936 of the eye-wearing device 100 may include a short-range transceiver (Bluetooth). TM The device 990 includes a transceiver for wireless wide area networks, local area networks, or wide area networks (e.g., cellular or Wi-Fi). The mobile device 990, including transceivers communicating via low-power wireless connection 925 and high-speed wireless connection 937, can be implemented using the architectural details of the eye-wearing device 100, just like other components of the network 995.

[0089] Memory 934 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 114A and 114B, the infrared camera 220, and the image processor 912, as well as images and videos generated for display by the image display driver 942 on the image displays of the optical components 180A and 180B. While memory 934 is shown as integrated with high-speed circuitry 930, in other examples, memory 934 may be a separate, independent component of the eye-wearing device 100. In some such examples, electrical wiring may provide a connection from a chip, including either a high-speed processor 932 or a low-power processor 922 of the image processor 912, to memory 934. In other examples, the high-speed processor 932 may manage addressing of memory 934, such that the low-power processor 922 will activate the high-speed processor 932 whenever a read or write operation involving memory 934 is required.

[0090] As shown in FIG9, the processor 932 of the eye-wearing device 100 can be coupled to a depth capture camera (visible light cameras 114A and 114B; or visible light camera 114A, infrared emitter 215 and infrared camera 220), an image display driver 942, a user input device 991, and a memory 934. As shown in FIG10, the processor 1030 of the mobile device 990 can be coupled to a depth capture camera 1070, an image display driver 1090, a user input device 1091, and a memory 1040A. As a result of the processor 932 of the eye-wearing device 100 executing the camera pipeline exposure timestamp error determination procedure 945 in the memory 934, the eye-wearing device 100 can perform all or a subset of any of the functions described below. Because the processor 1030 of the mobile device 990 executes the camera pipeline exposure timestamp error determination procedure 945 in the memory 1040A, the mobile device 990 can perform all or a subset of any of the functions described below.

[0091] In one example, the depth capture camera of the eye-wearing device 100 includes at least two visible light cameras, including a left visible light camera 114A with a left field of view 111A and a right visible light camera 114B with a right field of view 111B. The left field of view 111A and the right field of view 111B have overlapping fields of view. The depth capture camera 1070 of the mobile device 990 can be similarly constructed.

[0092] In the example, the depth capture camera of the eye-worn device 100 includes at least one visible light camera 114A and a depth sensor 213 (e.g., an infrared emitter 215 and an infrared camera 220). The at least one visible light camera 114A and the depth sensor 213 have substantially overlapping fields of view. The depth sensor 213 includes an infrared emitter 215 and an infrared camera 220. The infrared emitter 215 is attached to the frame 105 or temples 125A and 125B to emit an infrared light pattern. The infrared camera 220 is attached to the frame 105 or temples 125A and 125B to capture reflection variations in the emitted infrared light pattern. The depth capture camera 1070 of the mobile device 990 can be similarly constructed.

[0093] In one example, user input devices 991, 1091 include a touch sensor comprising an input surface and a sensor array coupled to the input surface to receive contact from at least one finger input by a user. User input devices 991, 1091 further include sensing circuitry integrated into or connected to the touch sensor and connected to processors 932, 1030. The sensing circuitry is configured to measure voltage to track the at least one finger contact on the input surface. The function of receiving input parameter identifiers from a user via user input devices 991, 1091 includes receiving the at least one finger contact input by the user on the input surface of the touch sensor.

[0094] Touch-based user input device 991 can be integrated into eye-wearing device 100. As described above, eye-wearing device 100 includes electronic housings 110A and 110B integrated into or connected to the sides 170A and 170B of frame 105 of eye-wearing device 100. Frame 105, temples 125A and 125B or electronic housings 110A and 110B include circuit boards that include touch sensors. The circuit boards include flexible printed circuit boards. Touch sensors are disposed on flexible printed circuit boards. The sensor array is a capacitive array or a resistive array. The capacitive or resistive array includes a grid forming a two-dimensional Cartesian coordinate system to track X-axis and Y-axis position coordinates.

[0095] Server system 998 may be one or more computing devices as part of a service or network computing system, including, for example, a processor, memory, and a network communication interface for communicating with mobile device 990 and eye-wearing device 100 via network 995. Eye-wearing device 100 is connected to a host. For example, eye-wearing device 100 is paired with mobile device 990 via high-speed wireless connection 937, or connected to server system 998 via network 995.

[0096] The output components of the eye-wearing device 100 include visual components, such as left and right image displays of optical components 180A and 180B, as described in Figures 2B and 2C (e.g., displays such as liquid crystal displays (LCDs), plasma display panels (PDPs), light-emitting diode (LED) displays, projectors, or waveguides). The left and right image displays of optical components 180A and 180B can present an initial video including an initial depth image sequence. The image displays of optical components 180A and 180B are driven by an image display driver 942. The image display driver 942 is coupled to the image displays to control the image displays to present the initial video. The output components of the eye-wearing device 100 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the eye-wearing device 100, mobile device 990, and server system 998 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photographic optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide touch location and touch force or touch gestures), audio input components (e.g., microphones), biometric components (e.g., heart rate monitors), and the like.

[0097] The eye-wear device 100 may optionally include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the eye-wear device 100. For example, peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein.

[0098] For example, biometric components include those that detect facial expressions (e.g., gestures, facial expressions, voice expressions, body posture, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brainwaves), and identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Positioning components include location sensor components (e.g., Global Positioning System (GPS) receiver components) that generate location coordinates, and Wi-Fi or Bluetooth that generates positioning system coordinates. TM Transceivers, altitude sensor components (e.g., altimeters or barometers that detect air pressure and derive altitude from air pressure), orientation sensor components (e.g., magnetometers), etc. Coordinates of such positioning systems can also be received from mobile devices 990 via wireless connections 925 and 937 via low-power wireless circuit 924 or high-speed wireless circuit 936.

[0099] Figure 10 is a high-level functional block diagram of an example mobile device 990. The mobile device 990 includes a user input device 1091 and an input parameter processor 1092 to receive user selections. The mobile device 990 includes a flash memory 1040A, which includes a camera pipeline exposure timestamp error determination procedure 945 for performing all or subsets of the functions described herein. The mobile device 1090 may include a depth capture camera 1070, which includes at least two visible light cameras (first and second visible light cameras with overlapping fields of view) or at least one visible light camera with substantially overlapping fields of view and a depth sensor, as in the eye-worn device 100.

[0100] The memory 1040A further includes multiple initial depth images generated via the depth capture camera of the eye-wearing device 100 or via the depth capture camera 1070 of the mobile device 990 itself. The memory 1040A also includes initial video, which comprises a series of initial depth images and associated time coordinates. Figures 11A through 11D show flowcharts outlining the functions that can be implemented in the camera pipeline exposure timestamp error determination procedure 945.

[0101] As shown in the figure, the mobile device 990 includes an image display 1080, an image display driver 1090 for controlling the image display, and a user input device 1091, similar to the eye-wearing device 100. In the example of Figure 10, the image display 1080 and the user input device 1091 are integrated together into a touchscreen display.

[0102] Examples of usable touchscreen mobile devices include (but are not limited to) smartphones, personal digital assistants (PDAs), tablets, laptops, or other portable devices. However, the structure and operation of touchscreen devices are provided by way of example; and the subject matter described herein is not intended to be limited thereto. For the purposes of this discussion, Figure 10 provides a block diagram illustration of an exemplary mobile device 990 having a touchscreen display for displaying content as a user interface (or as part of a user interface) and receiving user input.

[0103] As shown in Figure 10, the mobile device 990 includes at least one digital transceiver (XCVR) 1010 for digital wireless communication via a wide-area wireless mobile communication network, shown as a WWAN XCVR. The mobile device 990 also includes additional digital or analog transceivers, such as those for communication via NFC, VLC, DECT, ZigBee, Bluetooth, etc. TM The short-range XCVR 1020 can be used for short-range network communication via Wi-Fi. For example, the short-range XCVR 1020 can take the form of any available bidirectional wireless local area network (WLAN) transceiver that is compatible with one or more standard communication protocols implemented in a wireless local area network, such as Wi-Fi compliant with IEEE 802.11 and WiMAX.

[0104] To generate location coordinates for locating the mobile device 990, the mobile device 990 may include a Global Positioning System (GPS) receiver. Alternatively or additionally, the mobile device 990 may utilize either or both of a short-range XCVR 1020 and a WWAN XCVR 1010 to generate location coordinates for positioning, for example, based on a cellular network, Wi-Fi, or Bluetooth. TM The positioning systems can generate very accurate location coordinates, especially when used in combination. These location coordinates can be transmitted to the eye-wearing device via one or more network connections through the XCVR 1010, 1020.

[0105] Transceivers 1010 and 1020 (network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 1010 include (but are not limited to) transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 1010 and 1020 provide bidirectional wireless communication of information including digitized audio signals, still images and video signals, web page information for display and web-related input, and various types of mobile messaging communications to / from mobile device 990.

[0106] As previously described, several of these types of communications via transceivers 1010, 1020 and the network involve protocols and processes supporting communication with eyewear device 100 or server system 998. For example, such communication may transmit packet data to and from eyewear device 100 via wireless connections 925 and 937 through short-range XCVR 1020, as shown in FIG9. For example, such communication may also utilize IP packet data transmission to transmit data via WWAN XCVR 1010 through a network (e.g., the Internet) 995 shown in FIG9. Both WWAN XCVR 1010 and short-range XCVR 1020 are connected to an associated antenna (not shown) via radio frequency (RF) transmit and receive amplifiers (not shown).

[0107] The mobile device 990 further includes a microprocessor, shown as CPU 1030, sometimes referred to herein as the main controller. A processor is a circuit whose elements are constructed and arranged to perform one or more processing functions (typically various data processing functions). While discrete logic components can be used, these examples utilize components that form a programmable CPU. A microprocessor, for example, includes one or more integrated circuit (IC) chips that incorporate electronic components that perform the functions of the CPU. For example, processor 1030 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using the ARM architecture, as is commonly used today in mobile devices and other portable electronic devices. Other processor circuitry may be used to form CPU 1030 or processor hardware in smartphones, laptops, and tablets.

[0108] By configuring the mobile device 990 to perform various operations, for example, according to instructions or programs executable by the processor 1030, the microprocessor 1030 serves as a programmable host controller for the mobile device 990. Such operations may include, for example, various general operations of the mobile device, as well as operations related to the camera pipeline exposure timestamp error determination procedure 945 and communications with the eye-wearing device 100 and the server system 998. Although the processor can be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit configured by executing programs.

[0109] Mobile device 990 includes a memory or storage device system for storing data and programs. In this example, the memory system may include flash memory 1040A and random access memory (RAM) 1040B. RAM 1040B serves as a short-term storage device for instructions and data processed by processor 1030, for example, as working data processing memory. Flash memory 1040A typically provides long-term storage.

[0110] Therefore, in the example of mobile device 990, flash memory 1040A is used to store programs or instructions executed by processor 1030. Depending on the type of device, mobile device 990 stores and runs a mobile operating system, through which specific applications (including camera pipeline exposure timestamp error determination program 945) are executed. Applications such as camera pipeline exposure timestamp error determination program 945 can be native applications or hybrid applications running on mobile device 990. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0111] Figures 11A, 11B, 11C, and 11D are flowcharts 1100, 1110, 1118, and 1130, respectively, illustrating the operation of other components of the eye-wearing device 100 and the camera pipeline exposure timestamp error determination system 900 (e.g., one or more of processors 912, 932 execute instructions stored in memory 934, such as camera pipeline exposure timestamp error determination procedure 945). These steps are described with reference to the hardware described herein, but are not limited to these specific implementations. Although shown as occurring serially, the blocks of Figures 11A, 11B, 11C, and 11D may be reordered or parallelized depending on the specific implementation. Furthermore, those skilled in the art will understand from the description herein that one or more steps / blocks may be omitted, and one or more additional / alternative steps may be incorporated.

[0112] At box 1102, the processor 714 of the Test Generation System (TGS) determines the current time and generates a modulation sequence for encoding the current time for transmission and detection. The TGS processor 714 may be a processor of the Device Under Test (DUT), a separate processor coupled to the DUT's processor, or a processor synchronized with the DUT's processor. In the example, processor 714 runs an application that continuously generates modulation sequences from a clock source (e.g., a system clock) for transmission via the VLC method described herein. These modulation sequences (which result in a barcode image captured by the rolling shutter sensor 114; a barcode timestamp) are independent of the system exposure timestamp appended to the image by the camera system 704 (DUT). For ease of description, the time domains of the TGS clock source and the DUT clock source (which may be the same if the TGS and DUT are the same device or share a clock) are used to refer to the barcode timestamp and the system exposure timestamp, respectively.

[0113] At block 1104, processor 714 modulates light source 702 using the current time modulation sequence. In this example, processor 714 converts the current time modulation sequence into a binary value and actuates a switch (not shown) coupled to light source 702 to selectively turn light source 702 on and off according to the binary representation. The switch may be a field-effect transistor (FET) circuit connected between light source 702 and the output of processor 714. In one example, processor 714 is part of the DUT (e.g., processor 932 or processor 1030). In another example, processor 714 is incorporated into another device. Processor 714 continuously captures time from a source clock (DUT clock source or TGS clock source) in a repetitive cycle, converts the captured time into an appropriate form (e.g., binary representation), and transmits this binary representation using light source 702.

[0114] In one example, processor 714 uses pulse width modulation (PWM) to encode the current time modulation sequence. This allows it to determine which row of the rolling shutter image sensor the resulting image begins on, which can be used to determine the time difference between when the rolling shutter camera system begins image capture and when a timestamp based on the current time modulation sequence is generated. In one example, processor 714 and light source 702 modulate a 14-bit binary value (which may have more or fewer values ​​depending on the desired level of temporal granularity) by using a relatively long light pulse 806a as the start bit, followed by short pulses (which include a shorter pulse 808a and a longer pulse 808b representing 1 and 0, respectively). For example, the long light pulse 806 may be 100 microseconds, the shorter pulse 808a may be 20 microseconds, and the longer pulse 808b may be 40 microseconds.

[0115] At frame 1112, camera system 704 uses rolling shutter image sensor 114 (e.g., visible light camera 114A) to capture an image. The image captured by camera system 704 includes a barcode encoded in response to light source 702 with the current time.

[0116] In the image capture example, the rolling shutter image sensor 114 converts light from a light source into image pixels to obtain a raw image. The ISP 706 transforms the raw image into a processed image. The OS 708 receives the processed image and notifies the imaging application 710 that the processed image is available. One or more of the image sensor 114, ISP 706, or OS 708 can add a timestamp (system exposure timestamp) corresponding to the exposure of the image on the image sensor 114, which is based on the clock source of the DUT. The imaging application 710 can add an application timestamp (which is based on the clock source of the DUT) to image metadata indicating the time when the image is available. Additionally, the imaging application 710 can store an image with metadata including the system exposure timestamp and the application timestamp in memory 934.

[0117] At box 1114, processor 714 obtains the system exposure timestamp value corresponding to the system exposure timestamp added by rolling shutter camera system 704. In the example, processor 714 retrieves the image and associated metadata from memory 934, and parses the metadata to retrieve the system exposure timestamp.

[0118] If the processor 714 of the DUT is the processor of the TGS, the system exposure timestamp will be in the same time domain as the barcode timestamp. If the processor of the DUT is not the processor of the TGS, the clock used by the processor of the DUT can be synchronized with the clock used by the processor of the TGS before determining the current time modulation sequence (see box 1102). According to these examples, the system exposure timestamp value is equal to the system exposure timestamp added by the rolling shutter camera system 704.

[0119] At box 1116, processor 714 obtains a barcode timestamp value corresponding to a barcode timestamp determined by decoding the barcode in the image. In the example, processor 714 determines the barcode timestamp by recognizing the barcode 804 within the captured image 802 and decodes the light strip in the image into a binary timestamp by reversing the process used to encode the timestamp (see boxes 1102 and 1104).

[0120] If the processor 714 of the DUT is a processor of the TGS, the barcode timestamp will be in the same time domain as the system exposure timestamp. If the processor of the DUT is not a processor of the TGS, the clock used by the processor of the DUT can be synchronized with the clock used by the processor of the TGS before determining the current time modulation sequence (see box 1102). According to these examples, the barcode timestamp value is equal to the barcode timestamp determined by decoding the barcode in the image.

[0121] At box 1118, processor 714 determines the pipelined exposure timestamp error of the rolling shutter camera system 704 by comparing the obtained barcode timestamp value (see box 1116) with the system exposure timestamp value (see box 1114) added to image 802 by the rolling shutter camera system 704. The system exposure timestamp added by the rolling shutter camera system 704 indicates the time when image 802, as determined by the rolling shutter camera 114, was captured by the rolling shutter camera 114. In the case where the starting bit 806a of image 802 corresponds to the first row of image 802 (i.e., no bit 812 precedes it), processor 714 subtracts the exposure timestamp from the barcode timestamp to determine the pipelined exposure timestamp error.

[0122] If the starting bit 806a of image 802 does not correspond to the first row of image 802 (i.e., bit 812 precedes it), at least one of the barcode timestamp or exposure timestamp is adjusted to improve accuracy. In the example, the shutter offset / readout time for each row of the sensor is stored in memory 934 for timestamp adjustment. According to this example, at box 1118a, processor 714 determines the location of barcode 804 within image 808. Processor 714 then determines this location by processing the rows of the image and identifying the number of rows in image 804 preceding the first row including the first bit 806a of barcode 804.

[0123] At box 1118b, processor 714 calculates the adjustment time period based on the determined positioning (box 1118a) and the offset / readout of image sensor 114. Processor 714 calculates the adjustment time period (box 1118a) by retrieving the offset / readout rate of each line from memory 934 and multiplying the retrieved rate by the number of lines preceding the identified barcode 804.

[0124] At box 1118c, processor 714 uses an adjustment time period to adjust at least one of the barcode timestamp or the exposure timestamp. In one example, processor 714 adds the adjustment time to the exposure timestamp. In another example, processor 714 subtracts the adjustment time from the barcode timestamp. In yet another example, the processor adds a portion of the adjustment time to the exposure timestamp and subtracts the remainder from the barcode timestamp.

[0125] At box 1118d, processor 714 compares the adjusted barcode timestamp with the exposure timestamp (box 1118c). In the example, after timestamp adjustment (box 1118c), processor 714 subtracts the exposure timestamp from the barcode timestamp to determine the pipeline exposure timestamp error.

[0126] At box 1120, processor 714 determines the current timestamps of the DUT and TGS. In one example, the current timestamp of the DUT is generated by imaging application 710 and represents the time when the image is available for use, while the TGS generates a corresponding timestamp at the same time.

[0127] When the DUT and TGS have different clocks (i.e., in different time domains), as described in reference boxes 1122 and 1124 below, the current timestamps of the DUT and TGS (box 1120) can be used to determine the pipeline exposure timestamp error associated with image exposure. The current timestamps (1120) from the perspectives of both the DUT and TGS can be synchronized using conventional clock synchronization techniques.

[0128] At box 1122, processor 714 determines a first measurement for the DUT and a second measurement for the TGS. The processor determines the first measurement for the DUT by comparing the determined exposure timestamp with the current timestamp determined by the DUT (box 1120). According to this example, the system exposure timestamp value (box 1114) is equal to the difference between the current timestamp determined by the DUT and the system exposure timestamp added by camera system 704 (e.g., DUT(2) - DUT(1)). The processor determines the second measurement by comparing the obtained barcode timestamp with the current timestamp determined by the TGS (box 1120). According to this example, the barcode timestamp value (box 1116) is equal to the difference between the current timestamp determined by the TGS and the barcode timestamp (e.g., TGS(2) - TGS(1)).

[0129] At box 1124, processor 714 determines the pipeline exposure timestamp error. In this example, processor 714 determines the pipeline exposure timestamp error by comparing a first measurement with a second measurement. Processor 714 can determine the error by subtracting the first measurement from the second measurement, where the difference represents the error in the exposure timestamp.

[0130] At box 1126, processor 714 detects digit regularization in image 802. In this example, processor 714 detects digit regularization by comparing features in image 802 with expected features. For example, processor 714 expects the rows of barcode 810 to be horizontal in image 802. Processor 714 uses conventional image processing algorithms to compare the expected result (e.g., horizontal rows) with the actual result (e.g., non-horizontal rows). Additionally, using conventional image processing algorithms, the processor can determine how much the actual result deviates from the expected result. In one example, if the deviation between the expected and actual result exceeds a threshold, the imaging system can be identified as defective. In another example, processor 714 can apply a correction factor based on the degree of deviation when processing image 802, causing image 802 to be corrected to present / depict the expected result.

[0131] At box 1132, processor 714 uses another rolling shutter image sensor (e.g., visible light camera 114B) to capture another image. Processor 714 may capture the image as described in reference box 1112 above (FIG. 11B). In one example, the other image captured by the other image sensor 114B includes the same time-coded barcode captured in the image by image sensor 114A. In another example, the other image captured by the other image sensor 114B includes another time-coded barcode.

[0132] At box 1134, processor 714 determines another system exposure timestamp. Processor 714 may determine another system exposure timestamp of another captured image as described above with reference to box 1114.

[0133] In an example where another image captured by another image sensor 114B includes the same time-coded barcode captured in the image by image sensor 114A, the barcode timestamp obtained at box 1116 can be used in the following steps. In an example where another image captured by another image sensor 114B includes another time-coded barcode, another barcode timestamp, determined and optionally adjusted for the other image as described in reference boxes 1116 and 1118 above, can be used in the following steps to replace the barcode in the image captured by image sensor 114A with the barcode in the other image captured by image sensor 114B.

[0134] At box 1136, processor 714 determines another pipeline exposure timestamp error. Processor 714 may determine another pipeline exposure timestamp error as described above with reference to box 1118. In an example where another image captured by another image sensor 114B includes the same time-coded barcode captured in the image by image sensor 114A, the barcode timestamp obtained at box 1116 is used to determine another pipeline exposure timestamp error as described in reference to box 1118, thereby replacing the system exposure timestamp (box 1114) with another system exposure timestamp (box 1134). In an example where another image captured by another image sensor 114B includes another time-coded barcode, processor 714 determines another pipeline exposure timestamp error as described in reference to box 1118 by replacing the system exposure timestamp (box 1114) with another system exposure timestamp (box 1134) and replacing the obtained barcode timestamp (box 1116) with another barcode timestamp determined for the other image and optionally adjusted.

[0135] At box 1138, processor 714 synchronizes images captured by a pair of rolling shutter image sensors (e.g., image sensors 114A and 114B). In an example, processor 714 first determines the difference between the pipeline exposure timestamp error of one image sensor 114A and the pipeline exposure timestamp error of the other image sensor 114B. In one example, processor 714 then adjusts one or more components of the camera pipeline with the smallest exposure timestamp error to add the determined difference to increase its exposure timestamp error to match the exposure timestamp error of the slower camera pipeline. The exposure timestamp error can be added by processor 714 through at least one of hardware adjustments to ISP 706 or software adjustments to OS 708 or imaging application 710. In another example, processor 714 adjusts the metadata timestamps of images obtained using one image sensor, another image sensor, or a combination thereof, such that the two images are in a common temporal domain, which is advantageous for augmented reality applications.

[0136] As previously described, the camera pipeline exposure timestamp error determination function described herein with respect to eye-wearing device 100, mobile device 990, and server system 998 can be embodied in one or more applications. According to some examples, a “function,” “application,” “instruction,” or “program” is a program that performs a function defined within a program. Various programming languages ​​can be used to create one or more applications structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, a third-party application (e.g., an entity other than a platform-specific vendor using Android) may be involved. TMor iOS TM Applications developed using a Software Development Kit (SDK) can run on mobile operating systems such as iOS. TM ANDROID TM , Mobile software running on a phone or another mobile operating system. In this example, a third-party application may invoke API calls provided by the operating system to facilitate the functionality described herein.

[0137] Therefore, machine-readable media can take many forms of tangible storage media. Non-volatile storage media include, for example, optical discs or magnetic disks, any storage device such as any computer, such as client devices, media gateways, code converters, etc., that can be used to implement the figures shown. Volatile storage media include dynamic memory, such as the main memory of computer platforms. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that form buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves for transmitting data or instructions, cables or links for transmitting such carrier waves, or any other medium from which a computer can read program code or data. Many of these forms of computer-readable media can be used to carry one or more sequences of one or more instructions to a processor for execution.

[0138] The scope of protection is defined solely by the appended claims. When interpreted in accordance with this specification and subsequent application history, this scope is intended and should be interpreted as a broad range consistent with the ordinary meaning of the language used in the claims, and encompasses all structural and functional equivalents. Nevertheless, none of the claims is intended to include subject matter that does not meet the requirements of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in this manner. Therefore, any unintentional inclusion of such subject matter is waived.

[0139] In addition to what has just been stated above, whether or not it is stated in the claims, the stated or described content is not intended or should not be construed as causing any part, step, feature, object, benefit, advantage or equivalent to be offered to the public.

[0140] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meaning consistent with those in the corresponding fields of investigation and research. Relational terms such as “first” and “second” are used only to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between these entities or actions. The terms “comprising,” “including,” “containing,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes or comprises a list of elements or steps includes not only those elements or steps, but may also include other elements or steps not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element prefixed with “a” or “an” does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0141] Unless otherwise stated, any and all measurements, values, ratings, positions, quantities, dimensions, and other specifications set forth in this specification, including those in the appended claims, are approximate, not precise. Such quantities are intended to have a reasonable range consistent with the functions they relate to and the conventions in the fields to which they pertain. For example, unless otherwise expressly stated, parameter values, etc., can vary from said quantities by up to ±10%.

[0142] Furthermore, in the foregoing specific embodiments, various features have been combined in various examples for the purpose of simplifying this disclosure. The disclosed method should not be construed as reflecting an intention to require more features than expressly recited in each claim. Rather, as reflected in the following claims, the subject matter of the claims lies in fewer features than any single disclosed example. Therefore, the following claims are hereby incorporated into the specific embodiments, wherein each claim exists independently as a separately claimed subject matter.

[0143] While examples considered to be best practices and other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in various forms and examples, and is applicable to many applications, of which only some have been described herein. The appended claims are intended to claim protection for any and all modifications and variations falling within the true scope of the inventive concept.

Claims

1. A system for determining exposure timestamp error for a rolling shutter camera, the system comprising: A rolling shutter camera pipeline for a device under test (DUT) includes a DUT clock and a rolling shutter image sensor configured to capture an image generated from light emitted from a light source of a test generation system (TGS); the rolling shutter image sensor includes a sensor array, and the light from the light source is modulated by pulse width modulation (PWM), the light source timestamp used for modulation being generated by the TGS clock; and a processor configured to: The rolling shutter image sensor is used to capture the image, which includes a barcode formed by light emitted from a light source shining onto a sensor array, and the barcode is encoded with a light source timestamp. Obtain the system exposure timestamp value corresponding to the system exposure timestamp, which is determined by the rolling shutter camera pipeline for the captured image based on the DUT clock; The light source timestamp value is obtained from the light source timestamp encoded in the barcode of the image captured using a rolling shutter image sensor; as well as The exposure timestamp error of the rolling shutter camera is determined by comparing the light source timestamp value with the system exposure timestamp value. In order to determine the exposure timestamp error, the processor is configured to: Determine the location of the barcode within the image; The adjustment time period is calculated based on the determined positioning and the shutter offset or readout time of the shutter image sensor. The adjustment time period is calculated by multiplying the shutter offset / readout time of each row of the image sensor by the number of rows preceding the first row of the barcode. The adjustment time period is used to adjust at least one of the light source timestamp value or the system exposure timestamp value; and The timestamp value of the light source, adjusted by the adjustment time period, is compared with the timestamp value of the system exposure to determine the exposure timestamp error.

2. The system of claim 1, wherein the processor is further configured to: record a DUT value corresponding to the current time of the DUT from the DUT clock; and record a TGS value corresponding to the current time of the TGS from the TGS clock; wherein the light source timestamp value is equal to a first difference between the light source timestamp encoded in the barcode of the captured image and the current time of the TGS, the system exposure timestamp value is equal to a second difference between the system exposure timestamp determined by the rolling shutter camera assembly line and the current time of the DUT, and the exposure timestamp error is the difference between the light source timestamp value and the system exposure timestamp value.

3. The system according to claim 1, wherein the light source is a light-emitting diode.

4. The system of claim 1, further comprising a diffuser positioned between the light source and the rolling shutter image sensor.

5. The system of claim 1, wherein the light from the light source is pulse-width modulated by the following steps: transmitting a starting pulse having a first duration, followed by transmitting a series of first-type pulses and second-type pulses, wherein the first-type pulses have a second duration different from the first duration, and the second-type pulses have a third duration different from the second duration.

6. The system according to claim 1, further comprising: Another rolling shutter camera production line, the other rolling shutter camera production line including another rolling shutter image sensor, the other rolling shutter image sensor including another sensor array; The processor is further configured as follows: The other rolling shutter image sensor is used to capture another image, which contains a barcode formed by light emitted from a light source shining onto another sensor array, and the barcode is encoded with a light source timestamp; Obtain the other system exposure timestamp value corresponding to the other system exposure timestamp, which is determined by the other rolling shutter camera pipeline for the captured image based on the DUT clock; Another exposure timestamp error of the rolling shutter camera is determined by comparing the light source timestamp value with the exposure timestamp value of the other system; and The images captured by the rolling shutter image sensor are synchronized by correcting the offset between the exposure timestamp error and the other exposure timestamp error.

7. The system of claim 1, wherein the processor is further configured to: Determine the application timestamp value of the rolling shutter camera; The first delay measurement is determined by comparing the determined system exposure timestamp value with the application timestamp value; and The second delay measurement is determined by comparing the obtained light source timestamp value with the determined application timestamp value; The exposure timestamp error is determined by comparing the first delay measurement with the second delay measurement.

8. The system of claim 1, wherein the processor is further configured to detect digital distortion of the image by comparing actual features of the image with expected features of the image.

9. A method for determining the exposure timestamp error of a rolling shutter camera in a device under test (DUT), the method comprising: The rolling shutter image sensor of the rolling shutter camera production line is used to capture light from a Test Generation System (TGS) light source. This light forms an image containing a barcode on the rolling shutter image sensor. The barcode encodes a light source timestamp generated by the TGS clock of the TGS, wherein the light from the light source is modulated with the light source timestamp using pulse width modulation (PWM). Obtain the system exposure timestamp value corresponding to the system exposure timestamp, which is determined by the rolling shutter camera pipeline for the captured image based on the DUT clock; Obtain the light source timestamp value from the light source timestamp encoded in the barcode of the image captured using a rolling shutter image sensor; and The exposure timestamp error of the rolling shutter camera is determined by comparing the light source timestamp value with the system exposure timestamp value. The determination of the exposure timestamp error includes: Determine the location of the barcode within the image; The adjustment time period is calculated based on the determined positioning and the shutter offset or readout time of the shutter image sensor. The adjustment time period is calculated by multiplying the shutter offset / readout time of each row of the image sensor by the number of rows preceding the first row of the barcode; and The adjustment time period is used to adjust at least one of the light source timestamp value or the system exposure timestamp value; and The timestamp value of the light source, adjusted using the adjustment time period, is compared with the timestamp value of the system exposure to determine the exposure timestamp error.

10. The method of claim 9, further comprising: Record the DUT value corresponding to the current time of the DUT from the DUT clock; And record the TGS value corresponding to the current TGS time from the TGS clock; The light source timestamp value is equal to a first difference between the light source timestamp encoded in the barcode of the captured image and the current time of the TGS, the system exposure timestamp value is equal to a second difference between the system exposure timestamp determined by the rolling shutter camera assembly line and the current time of the DUT, and the exposure timestamp error is the difference between the light source timestamp value and the system exposure timestamp value.

11. The method of claim 9, wherein the pulse width modulation comprises: Transmit a start pulse with a first duration; And transmit a series of first-type pulses and second-type pulses, wherein the first-type pulse has a second duration different from the first duration, and the second-type pulse has a third duration different from the second duration.

12. The method of claim 9, further comprising: Another image is captured using another sensor array from another rolling shutter image sensor from another rolling shutter camera production line. This other image contains a barcode formed by light emitted from a light source shining onto the other sensor array, and the barcode is encoded with a light source timestamp. Obtain the other system exposure timestamp value corresponding to the other system exposure timestamp, which is determined by the other rolling shutter camera pipeline for the captured image based on the DUT clock; Another exposure timestamp error of the other rolling shutter camera is determined by comparing the light source timestamp value with the exposure timestamp value of the other system. as well as The images captured by the rolling shutter image sensor are synchronized by correcting the offset between the exposure timestamp error and the other exposure timestamp error.

13. The method of claim 9, further comprising: Determine the application timestamp value of the rolling shutter camera; The first delay measurement is determined by comparing the determined system exposure timestamp value with the determined application timestamp value; as well as The second delay measurement is determined by comparing the obtained light source timestamp value with the determined application timestamp value; The exposure timestamp error is determined by comparing the first delay measurement with the second delay measurement.

14. The method of claim 9, further comprising: Digital distortion of an image is detected by comparing the actual features of the image with the expected features of the image.

15. A non-transitory computer-readable medium comprising instructions for determining an exposure timestamp error of a rolling shutter camera in a device under test (DUT), the instructions causing an electronic system, when executed by a processor, to: The rolling shutter image sensor of the rolling shutter camera pipeline of the device under test (DUT) is used to capture light from a Test Generation System (TGS) light source. This light forms an image containing a barcode on the rolling shutter image sensor, the barcode encoding a light source timestamp generated by the TGS clock. The light from the light source is modulated using pulse width modulation (PWM) with the timestamp of the light source. Obtain the system exposure timestamp value corresponding to the system exposure timestamp, which is determined by the rolling shutter camera pipeline for the captured image based on the DUT clock; The light source timestamp value is obtained from the light source timestamp encoded in the barcode of the image captured using a rolling shutter image sensor; as well as The exposure timestamp error of the rolling shutter camera is determined by comparing the light source timestamp value with the system exposure timestamp value, including: Determine the location of the barcode within the image; The adjustment time period is calculated based on the determined positioning and the shutter offset or readout time of the shutter image sensor. The adjustment time period is the shutter offset / readout time of each row of the image sensor multiplied by the number of rows before the first row of the barcode. as well as The adjustment time period is used to adjust at least one of the light source timestamp value or the system exposure timestamp value; as well as The timestamp value of the light source, adjusted using the adjustment time period, is compared with the timestamp value of the system exposure to determine the exposure timestamp error.

Images