Self-adjusting sensor synchronization

Abstract

This document describes systems and techniques for self-adjusting sensor synchronization involving one sensor and self-adjusting synchronization for multi-sensor fusion. In aspects, one or more processors transmit synchronization signals to two or more image sensors effective to cause each of the two or more image sensors to generate a start of a frame. In response to a start of a frame, each of the two or more image sensors determines a time delay between the synchronization signal and the start of the respective frame. Based on the time delay of each of the two or more image sensors, the two or more image sensors individually adjust respective timing of the start of successive frames. The one or more processors may obtain timing information related to the time delay and align the captured frames based on the timing delay of each of the two or more image sensors. The one or more processors may then fuse the aligned frames to enhance image quality.

Description

Background Technology

[0001] Many personal electronic devices are equipped with advanced display technologies, reliable communication systems, and multiple digital camera components. Users of these devices are pleased that these multiple digital camera components enable improved image quality, increased optical zoom levels, wide-angle photography, and other such benefits. The use of multiple digital camera components often requires multi-sensor fusion, including, for example, the fusion of sensor data from a main camera and an ultra-wide-angle camera.

[0002] However, any two digital camera components within a personal electronic device may have different response characteristics. For example, a first digital camera component may have a first response delay when it receives a first request to capture a scene, while a second digital camera component may have a second response delay, different from the first response delay, when it receives a second request to capture the scene simultaneously with the first request. Such differences in response delay, such as the difference between the first and second response delays, among the digital camera components within a personal electronic device can lead to scene capture at different time intervals, posing technical challenges for frame merging and / or resulting in reduced fusion quality. Summary of the Invention

[0003] This document describes systems and techniques relating to self-adjusting synchronization for multi-sensor fusion. In various aspects, a method of an electronic device for self-adjusting synchronization of multi-sensor fusion includes: transmitting a synchronization signal to each of two or more image sensors; generating a start of a frame at each of the two or more image sensors based on the synchronization signal; determining a time delay between the synchronization signal and the start of the frame at each of the two or more image sensors in response to generating the start of the frame; and adjusting the vertical blanking time of at least one of the two or more image sensors based on the time delay, the adjustment being sufficient to advance or delay the start of a future frame.

[0004] In an additional aspect, a computing system is disclosed, comprising two or more image sensors, one or more processors, and computer-readable instructions that, when executed by the processors, cause the computing system to perform the methods described above.

[0005] In another aspect, a non-transitory computer-readable medium is disclosed, comprising computer-readable instructions that, when executed by a processor, cause a computing system to perform the methods described above.

[0006] This invention is provided to introduce a simplified concept of a self-adjusting synchronization scheme for multi-sensor fusion, which is further described in the following detailed description and accompanying drawings. This invention is not intended to identify key features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0007] This document describes in detail one or more aspects of self-adjusting synchronization for multi-sensor fusion with reference to the following figures. The same numbers are used throughout the figures to refer to similar features and components:

[0008] Figure 1 An example environment is shown, based on one or more implementations, in which an electronic device utilizes an image capture manager to capture and fuse multiple frames.

[0009] Figure 2 Example implementations of electronic devices for self-adjusting synchronization for multi-sensor fusion are shown, according to one or more implementation methods.

[0010] Figure 3 An exploded view of the components of an example camera system, including an example image sensor, according to one or more implementations, is shown.

[0011] Figure 4 An example implementation of a camera system that is operationally coupled to one or more processors, depending on one or more implementation methods, is shown.

[0012] Figure 5 A more detailed illustration is shown of the source based on one or more implementations. Figure 1 and Figure 2 An example implementation of an image capture manager.

[0013] Figure 6 An example implementation of a self-adjusting synchronization scheme for multi-sensor fusion, based on one or more implementation methods, is shown.

[0014] Figure 7 Timing diagrams for example periodic synchronization signals (sync), example negative relaxation, and example positive relaxation are shown, based on one or more implementations.

[0015] Figure 8 An example method for self-adjusting synchronization for multi-sensor fusion is shown, based on some implementations.

[0016] Figure 9 Another example method for self-adjusting synchronization for multi-sensor fusion is shown, based on some implementations.

[0017] Figure 10This illustrates some implementation methods that can be used as a reference to previous examples. Figures 1 to 9 The various components of an example computing system for self-adjusting synchronization of multi-sensor fusion can be implemented using any type of client, server, and / or electronic device. Detailed Implementation Overview

[0018] Original equipment manufacturers (OEMs) of various electronic devices, such as smartphones and laptops, typically embed two or more cameras within these devices. These cameras enable users of the electronic devices to capture still images (single frames) or video (multiple frames per second (FPS)) of a scene. In some applications, two or more cameras on a single electronic device can be used in conjunction to capture frames. For this purpose, multiple cameras can be synchronized to simultaneously capture different perspectives and / or create composite views from multiple angles.

[0019] Some techniques for synchronizing multiple cameras include hardware-based synchronization, software-based synchronization, or a combination thereof. For example, hardware-based synchronization can utilize dedicated hardware to generate a common timing reference for all cameras. This can be achieved using a master clock or timecode generator that sends synchronization signals (e.g., vertical synchronization signals) to each camera. When a camera receives a synchronization signal, it can begin its integration and streaming output process. In additional examples, some cameras have built-in features or software plugins for achieving synchronization. Such cameras can use protocols such as Network Time Protocol (NTP) or Precision Time Protocol (PTP) to synchronize their internal clocks. Software applications can then be used to align captured frames.

[0020] However, such techniques may have their drawbacks. For example, one or more cameras and the processor transmitting the synchronization signal may operate in independent clock domains. Furthermore, the processor may not know how each camera responds to the synchronization signal (e.g., response delay). Therefore, the processor does not know how the synchronization signal relates to the start of a frame in each camera. Moreover, due to factors such as response delay, clock drift, signal propagation delay, processing delay, etc., the timing of the start of frames for any two cameras may differ from each other and may differ from the synchronization signal.

[0021] For example, upon receiving a synchronization signal, the first camera might respond within microseconds, while the second camera might respond within milliseconds. These response time differences can be particularly pronounced when any two cameras come from different providers. Due to these different response times, when the processor receives captured frames from the cameras, it must estimate which frames should be aligned and merged. For instance, the first camera might respond to the synchronization signal at a time interval N+1, while the second camera responds at a time interval N+2. Without additional information, the processor might align the N+2 frames from the second camera with the N+1 frames from the first camera. Such inconsistencies make the sensor fusion process more difficult and the merged results more unpredictable.

[0022] To this end, this document describes a system and technique for self-adjusting synchronization for multi-sensor fusion. In each respect, one or more processors transmit synchronization signals to two or more image sensors to effectively enable each of the two or more image sensors to generate the start of a frame. In response to the start of the frame, each of the two or more image sensors determines a time delay between the synchronization signal and the corresponding start of the frame. Based on this time delay of each of the two or more image sensors, the two or more image sensors individually adjust the corresponding timing of the start of successive frames. The one or more processors can obtain timing information associated with this time delay and align the captured frames based on this timing delay of each of the two or more image sensors. The one or more processors can then fuse the aligned frames to enhance image quality.

[0023] This document describes additional examples and implementations. This document now turns to the example operating environment, followed by descriptions of the example environment, apparatus, implementation, methods, and system. Operating environment

[0024] Figure 1 An example environment 100, according to one or more implementations, is shown, in which an electronic device utilizes an image capture manager to capture and fuse multiple image frames. Example environment 100 includes an electronic device 102, shown as a smartphone only by way of example. Electronic device 102 includes, or is operatively coupled to, one or more processors 104 and a camera system 106 having two or more image sensors 108. Electronic device 102 also includes an image capture manager 110 having an image synchronization manager 112. As further described below, the image capture manager 110 with the image synchronization manager 112 is configured to implement one or more aspects of self-adjusting synchronization for multi-sensor fusion.

[0025] In various aspects, the image capture manager 110 instructs each of two or more image sensors 108 to capture image frames of scene 116, resulting in two or more captured image frames. For example, this instruction may be in response to user 114 opening an image capture application or providing a command to electronic device 102 to capture an image. After capturing two or more image frames of scene 116, the image synchronization manager 112 fuses the two or more captured image frames based on timing information from metadata transmitted from the two or more image sensors 108. Thus, the resulting fused frame may include more information (e.g., pixel data) and fuses image frames of scene 116 that are temporally close.

[0026] For example, user 114 may open an image capture application on electronic device 102. In response, image capture manager 110 instructs one or more processors 104 to activate two or more image sensors 108. Image capture manager 110 further instructs one or more processors 104 to periodically transmit a synchronization signal to each of the two or more image sensors 108. For one or more periodic transmissions, the two or more image sensors 108 determine the time delay between a given synchronization signal and the start of the corresponding frame (e.g., a marker indicating the start of a new frame in the image data stream). At the end of a frame (e.g., a marker indicating the completion of an image frame in the image data stream), each of the two or more image sensors 108 transmits timing information from metadata (e.g., data related to the time delay) to one or more processors 104. If the start of a previous frame is not time-aligned with a given synchronization signal (within a threshold), the two or more image sensors 108 independently (or under the instruction of one or more processors 104) adjust the corresponding timing of the start of successive frames based on the timing information of each of the two or more image sensors 108. In an implementation, the two or more image sensors 108 can adjust the corresponding timing of the start of successive frames by increasing or decreasing the vertical blanking period (e.g., the time interval from the end of one frame to the start of the successive frame, during which no valid image data is captured).

[0027] Using this technique, the start of a future frame for each of the two or more image sensors 108 can be temporally aligned with a future synchronization signal (e.g., after one or more periodic transmissions). Therefore, when user 114 provides a command to electronic device 102 to capture an image of scene 116, the start of the corresponding frame for each of the two or more image sensors 108 can be temporally aligned (e.g., approximately aligned). Regardless of whether the start of the first frame from the first image sensor and the start of the second frame from the second image sensor are aligned, one or more processors 104 can use metadata indicating the timing delay of each of the two or more image sensors 108 to fuse temporally close image frames. This establishes a feedback loop between one or more processors 104 and the two or more image sensors 108, thereby achieving frame start alignment and image quality enhancement.

[0028] The resulting fused frame can then be presented as a digital image 118-1 on the display of the electronic device 102. For example, two of the two or more image sensors 108 can be positioned on the back of the electronic device 102, opposite, for example, the display. These two image sensors can also be positioned behind lenses set at different zoom levels. The lens of the first image sensor of the two or more image sensors 108 can be configured to capture a first image frame of scene 116 at low zoom. The lens of the second image sensor of the two or more image sensors 108 can be configured to capture a second image frame of scene 116 at high zoom. Therefore, when the first and second image frames captured simultaneously or nearly simultaneously are fused together, the resulting fused frame may not contain motion artifacts (e.g., motion artifacts that might otherwise occur due to the timing offset between the first and second image frames). Furthermore, the resulting fused frame may include enhanced image data such that when the user 114 zooms in on the digital image 118-2 at the display of the electronic device 102, as... Figure 1 As shown, the digital image 118-2 may include reduced pixelation, higher levels of detail, and / or better focus. These techniques can be implemented with or without temporally aligned frames between the first and second image sensors (e.g., due to insufficient time for adjustment).

[0029] consider Figure 2 The figure illustrates an example implementation 200 of an electronic device 102 configured to implement self-adjusting synchronization for multi-sensor fusion, according to one or more implementation methods. Figure 2The electronic device 102 is shown as various example devices, including a smartphone 102-1, a tablet computer 102-2, a laptop computer 102-3, a computing watch 102-4, computing glasses 102-5, and an artificial reality / virtual reality (AR / VR) headset 102-6. Although not shown, the electronic device 102 can also be implemented as any of the following: a mobile station (e.g., a fixed or mobile STA), a mobile communication device, a client device, a home automation and control system, an entertainment system, a game console, a personal media device, a health monitoring device, a drone, a camera, an internet-connected home appliance supporting wireless internet access and browsing, an IoT device, a security system, etc. It should be noted that the electronic device 102 can be wearable, non-wearable but mobile, or relatively stationary (e.g., a desktop computer, an appliance). It should also be noted that the electronic device 102 can be used with or embedded within electronic devices or peripherals, such as in a car or as an accessory to a laptop computer. The electronic device 102 may include components designed for clarity or visual simplicity. Figure 2 Components or interfaces omitted in the text.

[0030] In some implementations, electronic device 102 includes a printed circuit board assembly (PCBA) on which components and interconnects of electronic device 102 are embodied. Alternatively or additionally, components of electronic device 102 may be embodied on other substrates such as flexible circuit materials or other insulating materials, and optionally may be operatively coupled to the PCBA. Electronic device 102 may also include a housing defining at least one cavity. In some implementations, the housing may be supported and / or defined by a frame. The housing includes an outer surface and an opposing inner surface. The outer surface may include at least one portion that contacts a physical medium associated with a user (e.g., hair, skin, tissue, clothing) or a physical medium associated with an accessory (e.g., a base, padding). For example, a computing watch 102-4 may include an outer surface that contacts a user's wrist. In another example, a smartphone 102-1 may include an outer surface that contacts a charging stand. In various aspects, the housing may be any of a variety of plastics, metals, acrylics, or glass. In some implementations, the outer surface of the housing includes one or more openings such as ports (e.g., barometric ports).

[0031] As shown, electronic device 102 includes one or more processors 104 and computer-readable medium 204. Processor 104 may include any suitable single-core or multi-core processor (e.g., application processor (AP), digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU), computer vision and machine learning (CVML) processor). Processor 104 may be configured to execute instructions or commands stored within computer-readable medium 204. In an example implementation, electronic device 102 includes a multi-core system-on-a-chip (SoC) configured to execute instructions or commands stored within computer-readable medium 204 in parallel. Applications and / or operating systems (not shown) implemented as computer-readable instructions on computer-readable medium 204 may be executed by computer processor 104 to provide some or all of the functionality described herein. For example, computer-readable medium 204 may include an image capture manager 110 and an image synchronization manager 112. In an additional implementation, image capture manager 110 and / or image synchronization manager 112 may be stored at a remote computing system and accessed by electronic device 102 via a wired or wireless connection.

[0032] Computer-readable medium 204 may be stored in one or more non-transitory storage devices, such as random access memory (RAM, dynamic RAM (DRAM), non-volatile RAM (NVRAM), or static RAM (SRAM)), read-only memory (ROM) or flash memory, hard drives, solid-state drives (SSDs), or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus. The term "coupled" may refer to two or more elements in direct contact (physical, electrical, magnetic, optical, etc.), or to two or more elements that are not in direct contact but still cooperate and / or interact with each other.

[0033] At least one of the processors 104 includes an image signal processor 202 (e.g., an image processing engine, an image processing unit (IPU), or an image processor). In example implementations, the image signal processor 202 may be operatively coupled to or integrated within a multi-core SoC (e.g., an application processor). The image signal processor 202 may be a dedicated processing unit responsible for converting raw image data into digital images, and for processing and enhancing image data captured by a camera system (e.g., camera system 106). In some examples, the image signal processor 202 may perform demosaicing (e.g., to reconstruct panchromatic information), noise reduction, white balance, tone mapping and dynamic range compression, sharpening and detail enhancement, image stabilization, and / or compression and encoding.

[0034] Electronic device 102 may further include, or be operatively coupled to, a communication system 206. Communication system 206 enables communication of device data, such as received data, transmitted data, or other information as described herein, and may provide connectivity to one or more networks and other devices connected thereto. Example communication systems include NFC transceivers, WPAN radios compliant with various IEEE 802.15 (Bluetooth®) standards, WLAN radios compliant with any of the various IEEE 802.11 (WiFi®) standards, WWAN (3GPP compliant) radios for cellular phones, Wireless Metropolitan Area Network (WMAN) radios compliant with various IEEE 802.16 (WiMAX®) standards, infrared (IR) transceivers compliant with the Infrared Data Association (IrDA) protocol, and wired local area network (LAN) Ethernet transceivers. Depending on the communication protocol or standard followed by which electronic device 102 communicates, device data transmitted through communication system 206 may be packetized or framed. Communication system 206 may include a wired interface, such as an Ethernet or fiber optic interface, for communication over a local network, private network, intranet, or the Internet. Alternatively or additionally, communication system 206 may include a wireless interface that facilitates communication over a wireless network, such as a wireless LAN, cellular network, or WPAN.

[0035] As further shown, the electronic device 102 includes a plurality of sensors 208. The plurality of sensors 208 can acquire information about the local environment of the electronic device 102. For example, the plurality of sensors 208 may include any of a variety of sensors, such as an audio sensor (e.g., a microphone), a touch input sensor (e.g., a touchscreen), a proximity sensor (e.g., a capacitive sensor, a radar sensor), a depth sensor (e.g., a distance estimator), or an ambient light sensor (e.g., a photodetector). In one example, the electronic device 102 uses the proximity sensor among the plurality of sensors 208 to adjust the zoom level of a first image sensor to low zoom and the zoom level of a second image sensor to high zoom.

[0036] The electronic device 102 further includes a camera system 106 having two or more image sensors 108, lenses 210, and apertures 212. The two or more image sensors 108, implemented as, for example, complementary metal-oxide-semiconductor (CMOS), charge-coupled device (CCD) sensors, and / or any other such image capture technology, can be configured to capture light, such as light reflected from one or more objects in a scene, and generate digital signals (e.g., data related to the captured light in the scene).

[0037] consider Figure 3This figure illustrates an exploded view 300 of components of an example camera system 106, including an example image sensor 302, according to one or more implementations. The components of the camera system 106 may be partially or completely housed within the housing of the electronic device 102. In one implementation, the housing of the electronic device 102 includes an opening in which one or more components of the corresponding camera system 106 may be located or behind the opening. In another implementation, the image sensor 302 includes a photosensitive material, microlenses, and / or microelectronic components.

[0038] In one example, a CMOS image sensor may include one or more of the following: a microlens array, a color filter array (CFA), a photodiode layer, pixel transistors, one or more adhesive layers, and / or an application-specific integrated circuit (ASIC). More specifically, the microlens array may include small lenses that collect and focus light onto photosensitive areas of the CMOS image sensor. The CFA may be a patterned layer comprising a mosaic of tiny color filters placed on one or more photosensitive points (sometimes called pixels) to capture color information. The CFA may allow light of different wavelengths to pass through to the photosensitive points. For example, the CFA pattern may include Bayer filters, which may consist of red, green, and blue filters. A corresponding photosensitive point among the one or more photosensitive points may be a single photosensitive area that captures light and converts it into an electrical signal. In some implementations, each photosensitive point may correspond to a single pixel in a digitally reproduced image of a scene. The photosensitive points may be arranged (e.g., in a grid pattern) to form a pixel array. Each photosensitive point may include a photodiode, thereby defining a photodiode layer. Each photodiode in the photodiodes can be made of a semiconductor material such as silicon. Due to incident light from the scene, the photodiodes undergo the photoelectric effect (e.g., electron emission in response to electromagnetic radiation). A pixel transistor can then receive electrical signals from each photodiode. The pixel transistor can provide gain and / or buffering of the charge from the photodiodes. The ASIC can be implemented as a logic layer and can include a readout circuitry system, a timing and control circuitry system, and an output interface. The readout circuitry system can convert the accumulated charge from each photodiode into a digital signal. For example, the readout circuitry system can include one or more amplifiers, analog-to-digital converters (ADCs), and signal processing circuitry. The timing and control circuitry system controls the operation and timing of the image sensor 302, including exposure time, pixel readout, and synchronization with other hardware components of the image sensor 302 and / or electronic device 102. The output interface enables data to be transferred from the image sensor 302 to other hardware components such as one or more processors and memories.

[0039] Lens 210 (e.g., lens 304) may include, for example, a glass lens or a plastic lens (e.g., an element) that focuses light from the scene onto at least one of two or more image sensors 108 (e.g., image sensor 302). By adjusting the distance between lenses 210 and / or the distance between one or more lenses 210 and their respective image sensors 302, camera system 106 can change the magnification (e.g., zoom in, zoom out) and adjust the focus level of the scene. Although Figure 3 The example camera system 106 is shown as having seven lenses 304, but the number of lenses 304 can vary for each of two or more image sensors 108.

[0040] The camera system 106 may further include an aperture 306 for each of the two or more image sensors 108. In one implementation, the respective aperture 306 is configured to adjust the amount of incident light at the respective image sensor 302 by adjusting the size of the opening through which light from the scene travels. The aperture 306 may also affect the depth of field (DOF) in the digitally reproduced image.

[0041] Back Figure 2 The camera system 106 further includes a camera drive system 214 and optional one or more illuminators 216 (e.g., camera flashes). The camera drive system 214 (e.g., an output interface of a corresponding image sensor) enables communication between the camera system 106 and other components of the electronics 102, such as the processor 104 and the computer-readable medium 204. The camera drive system 214 can be activated by any suitable trigger, such as user input received via actuation controls or pressed buttons (e.g., graphical user interface (GUI) buttons), or signals received from one or more sensors 208. The camera drive system 214 can also control settings for the camera system 106 in response to commands from the processor 104, including settings for gain, white balance, focus, zoom, image stabilization, and exposure duration.

[0042] In this implementation, one or more hardware components of the camera system 106, including the image sensor 108, lens 210, and aperture 212, can be manually adjusted by the user (e.g., via commands provided at a user interface, via physical adjustment) and / or automatically adjusted by one or more algorithms stored in computer-readable medium 204 and executed by one or more processors 104. For example, the corresponding aperture in one or more apertures 212 can be automatically controlled by an automatic exposure algorithm. The automatic exposure algorithm can calculate and adjust exposure parameters (e.g., exposure time, gain, f-number) based on the exposure value.

[0043] In additional examples, one or more lenses 210 and / or one or more apertures 212 can be automatically controlled by an autofocus algorithm. In one example, camera system 106 can be operatively coupled to a distance estimator that determines the distance from an object to camera system 106, allowing the autofocus algorithm to automatically adjust the focus of camera system 106. In another example, the focus of camera system 106 can be adjusted using passive autofocus techniques. Passive autofocus techniques can use image data captured at the respective image sensor 302 to determine whether an image is in focus and adjust one or more lenses 210 to compensate. For example, passive autofocus techniques can involve an autofocus algorithm that detects contrast within the image data and adjusts the focus until the contrast is maximized. In yet another example, the focus of camera system 106 can be adjusted using phase detection autofocus techniques.

[0044] Figure 4 An example implementation 400 of a camera system 106 operatively coupled to one or more processors 104 is illustrated. As shown, and by way of example only, and not limitation, the camera system 106 includes two or more image sensors 108 comprising three image sensors (e.g., a first image sensor 108-1, a second image sensor 108-2, and a third image sensor 108-3). Furthermore, and by way of example only, and not limitation, each of the three image sensors is operatively coupled to one or more processors 104. In some implementations, at least one of the one or more processors 104 is integrated within a multi-core SoC, and at least one of the three image sensors is operatively coupled to at least one processor 104 via a connection to the multi-core SoC.

[0045] In the example implementation, the lens of the first image sensor 108-1 is integrated on the forward portion of the electronic device 102, the lens of the second image sensor 108-2 is integrated on the rearward portion of the electronic device 102, and the lens of the third image sensor 108-3 is integrated on the rearward portion of the electronic device 102. Thus, the electronic device 102 includes a forward-facing camera and two rearward-facing cameras to capture a first scene in the area in front of the electronic device 102 and a second scene in the area behind the electronic device 102.

[0046] As shown in the figure, each of the three image sensors can be operatively coupled to one or more processors 104 via an output Mobile Industrial Processor Interface (MIPI) interface 402 (e.g., a first output MIPI interface 402-1, a second output MIPI interface 402-2, and a third output MIPI interface 402-3). For example, a first image sensor 108-1 can be operatively coupled to one or more processors 104 via a first output MIPI interface 402-1 (e.g., a single-channel MIPI), a second image sensor 108-2 can be operatively coupled to one or more processors 104 via a second output MIPI interface 402-2 (e.g., a multi-channel MIPI), and a third image sensor 108-3 can be operatively coupled to one or more processors 104 via a third output MIPI interface 402-3. One or more output MIPI interfaces in the output MIPI interfaces 402 can be camera serial interfaces (CSI) such as MIPI CSI-2. Alternatively or additionally, one or more output MIPI interfaces in output MIPI interface 402 may be an inter-integrated circuit such as MIPI I2C or MIPI I3C. In an implementation, one or more image sensors in image sensor 108 may transmit data to one or more processors 104 via output MIPI interface 402. For example, the first image sensor 108-1 may transmit metadata with timing information related to time delays (e.g., the duration between a synchronization signal and the start of a frame) via the first output MIPI interface 402-1.

[0047] Furthermore, each of the two or more image sensors 108 can be operatively coupled to one or more processors 104 via an input MIPI interface 404 (e.g., a first input MIPI interface 404-1, a second input MIPI interface 404-2, and a third input MIPI interface 404-3). For example, a first image sensor 108-1 can be operatively coupled to one or more processors 104 via a first input MIPI interface 404-1 (e.g., single-channel MIPI), a second image sensor 108-2 can be operatively coupled to one or more processors 104 via a second input MIPI interface 404-2 (e.g., multi-channel MIPI), and a third image sensor 108-3 can be operatively coupled to one or more processors 104 via a third input MIPI interface 404-3. The one or more input MIPI interfaces 404 can be CSIs such as MIPI CSI-2. Alternatively or additionally, one or more input MIPI interfaces 404 can be integrated circuit buses such as MIPII2C or MIPI I3C. In one implementation, one or more image sensors in image sensor 108 can receive data from one or more processors 104 via input MIPI interface 404. For example, one or more processors 104 transmit instructions via input MIPI interface 404 to program register settings (e.g., tolerance registers) at one or more of the three image sensors. Programming the register settings (e.g., parameterization) can include programming tolerances (e.g., tolerance thresholds) for timing variations such as time skew at one or more of the three image sensors. In one implementation, a given register includes a threshold that allows the associated image sensors to have a predefined time difference between the start of a frame and a synchronization signal.

[0048] It is further shown that each of the three image sensors can be operatively coupled to one or more processors 104 via synchronization interfaces 406 (e.g., first synchronization interface 406-1, second synchronization interface 406-2, and third synchronization interface 406-3). Synchronization interfaces 406 enable one or more processors 104 to transmit synchronization signals to the three image sensors. For example, a first image sensor 108-1 can be operatively coupled to one or more processors 104 via a first synchronization interface 406-1, a second image sensor 108-2 can be operatively coupled to one or more processors 104 via a second synchronization interface 406-2, and a third image sensor 108-3 can be operatively coupled to one or more processors 104 via a third synchronization interface 406-3. Synchronization interfaces 406 may be specific to the camera system or follow standardized protocols. For example, synchronization interfaces 406 may include general purpose input / output (GPIO) pins, MIPI CSI interfaces, dedicated synchronization lines, and / or embedded synchronization protocols. In at least some implementations, the input MIPI interface 404 can replace the synchronization interface 406 to transmit synchronization signals.

[0049] In this implementation, one or more processors 104 include (or are operatively coupled to) timing devices (e.g., a master clock, a clock generator) that generate synchronization signals. These synchronization signals can be transmitted to each of the three image sensors via synchronization interface 406. However, since the response times of the three image sensors may differ upon receiving the corresponding synchronization signals, each of the three image sensors can be operatively coupled to one or more processors 104 via output MIPI interface 402. This allows a feedback loop to be established between the three image sensors and one or more processors 104, and using the timing information transmitted via output MIPI interface 402, one or more processors 104 can more reliably fuse image frames from two or more image sensors 108.

[0050] Return to Figure 2 The electronic device 102 may also include a display 218. The display 218 may include any suitable visual display device, such as a liquid crystal display (LCD), a thin-film transistor (TFT) LCD, an in-situ switched-to-position (IPS) LCD, a capacitive touchscreen display, an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode (AMOLED) display, a super AMOLED display, and so on. The display 218 may be referred to as a screen, allowing content to be displayed on the screen. For example, in response to the execution of instructions in the computer-readable medium 204, one or more processors 104 may instruct the display 218 to present a still image (a single frame) or video (multiple FPS) of a scene captured by the camera system 106.

[0051] Figure 5 A more detailed description of the sources based on one or more implementations. Figure 1 and Figure 2 Example implementation of image capture manager 110, method 500. Although Figure 5 Various entities and components as part of the image capture manager 110 are shown, but any of these entities and components can be separated from the image capture manager 110, allowing the image capture manager 110 to access and / or communicate with them to achieve and / or facilitate self-adjusting synchronization for multi-sensor fusion. Furthermore, various implementations of the image capture manager 110 and / or its components and entities may include a SoC, one or more integrated circuits (ICs), a processor with embedded processor instructions or configured to access processor instructions stored in memory, hardware with embedded firmware, one or more algorithms, machine learning techniques, or any combination thereof. For example, execution of the image capture manager 110 may occur partially or entirely on the image signal processor 202, one or more CPUs, an application processor, a multi-core SoC, etc.

[0052] like Figure 5 As shown, the image capture manager 110 includes an image synchronization manager 112 and a "3A" (autofocus, auto white balance, auto exposure) engine 502. The image synchronization manager 112 may include a synchronization signal generator 504, a metadata module 506, and a fusion module 508. Although not shown, the image capture manager 110 may also include additional entities and components such as a facial recognition module for facial authentication.

[0053] In various aspects, the image capture manager 110 activates two or more image sensors (e.g., two or more image sensors 108). Activation of the two or more image sensors may be in response to a user (e.g., user 114) opening an image capture application, powering on the electronic device 102, or some other triggering event (e.g., motion detected by sensor 208). Simultaneously or subsequently, the image capture manager 110 transmits (e.g., periodically) synchronization signals to the two or more image sensors via the image synchronization manager 112 using a synchronization signal generator 504. The synchronization signals can be transmitted to each of the two or more image sensors via the synchronization interface 406. The synchronization signal generator 504 can be implemented as, for example, a timing device.

[0054] Upon receiving a synchronization signal, each of the two or more image sensors 108 determines the duration between the synchronization signal and the start of the corresponding frame. For example, a counter in the corresponding image sensor counts the duration between the synchronization signal and the start of the nearest neighbor frame. In this implementation, the duration can be positive or negative. For example, a positive duration (“positive slack”) indicates that the synchronization signal was received at the corresponding image sensor after the start of a temporally close (e.g., nearest neighbor) frame (e.g., during stream on). A negative duration (“negative slack”) indicates that the synchronization signal was received at the corresponding image sensor before the start of a temporally close (e.g., nearest neighbor) frame (e.g., during the vertical blanking time, after the end of stream on).

[0055] If the corresponding image sensor determines that positive relaxation exists, the finite state machine (FSM) within the corresponding image sensor is configured to delay the start of a future frame, for example, such that the duration between the future synchronization signal and the start of the future frame is within a tolerance threshold. Delaying the start of a future frame can be achieved by increasing the vertical blanking time. If the corresponding image sensor determines that negative relaxation exists, the FSM within the corresponding image sensor is configured to advance the start of a future frame, for example, such that the duration between the future synchronization signal and the start of the future frame is within a tolerance threshold. Advancing the start of a future frame can be achieved by decreasing the vertical blanking time. In at least some implementations, depending on camera system parameters, the start of a future frame can be delayed or advanced (e.g., by increasing or decreasing the vertical blanking time) after receiving one or more synchronization signals. For example, the FSM can gradually advance the start of each successive future frame in predetermined increments (e.g., a few microseconds, a few milliseconds) until, after one or more synchronization signals, the duration between the future synchronization signal and the start of the future frame is within a tolerance threshold. In this way, two or more image sensors 108 can self-adjust to align with the synchronization signals.

[0056] At the beginning or end of a frame, timing information from each image sensor relating to the duration between a given synchronization signal and the start of the corresponding frame can be transmitted to the image capture manager 110 as, for example, embedded metadata via the output MIPI interface 402. The image capture manager 110 can then utilize the metadata module 506 to extract timing information for each of the two or more image sensors 108. Thus, when the image capture manager 110 receives two or more image frames from the two or more image sensors 108, it aligns the image frames based on the timing information in the metadata. Therefore, the image capture manager 110 can use the fusion module 508 to fuse (e.g., merge) time-proximity captured frames based on the timing information. By doing so, misaligned captured frames can be reduced, and the image quality of the fused frames can be improved.

[0057] In some implementations, two or more image sensors 108 can operate at a common set of frequencies. For example, the first image sensor may operate at a frame rate of 60 Hz, while the second image sensor may operate at a frame rate of 30 Hz. In a further implementation, a synchronization signal generator 504 is configured to periodically generate a synchronization signal at the same frequency (e.g., 30 Hz) as the fastest image sensor frame rate. In an additional implementation, the two or more image sensors 108 can be configured not to interrupt captured frames.

[0058] These and other capabilities and configurations are described in more detail below. Figure 5 The way entities act and interact. These entities can also be further divided, combined, and so on. Figure 1 Environment 100 Figure 2 Example implementation 200 and Figures 6 to 7 The detailed illustrations show some of the many possible environments and devices in which the described technology can be employed. Example implementation

[0059] Figure 6 An example implementation 600 of a self-adjusting synchronization scheme for multi-sensor fusion, according to one or more implementation methods, is shown. In some implementations, example implementation 600 is implemented entirely on a single electronic device 102. In other implementations, example implementation 600 is implemented among multiple electronic devices. For example, portions of example implementation 600 are implemented on a first surveillance camera, a second surveillance camera, a third surveillance camera, and an optional central device. In yet another example, portions of example implementation 600 are implemented on a portable electronic device and a server system.

[0060] As shown in the figure, example implementation 600 includes a first image sensor 108-1, a second image sensor 108-2, and a third image sensor 108-3. One or more of these image sensors 108 can be implemented on one or more electronic devices. Example implementation 600 also includes a synchronization signal generator 504, a metadata module 506, and a fusion module 508 of an image capture manager 110 (not shown). Each of these components and entities of the image capture manager 110 can be implemented on one or more electronic devices.

[0061] Furthermore, the synchronization signal generator 504 triggers a frame request and transmits a synchronization signal to each image sensor in the image sensor 108. In one implementation, the synchronization signal generator 504 may trigger a frame request and transmit the synchronization signal simultaneously. In an alternative implementation, the synchronization signal generator 504 may trigger frame requests and transmit the synchronization signal sequentially.

[0062] Upon receiving a frame request and / or synchronization signal, the corresponding image sensor generates the start of a frame and begins reading out pixel data. For example, the corresponding image sensor first generates the start of a frame and begins reading out pixel data line by line according to the raster pattern (e.g., stream on). After reading out the last line, the corresponding image sensor generates the end of a frame (e.g., indicating that the current frame reading is complete).

[0063] Each corresponding image sensor can determine the duration between the received synchronization signal and the start of the corresponding frame. For example, a counter in the corresponding image sensor counts the duration between the received synchronization signal and the start of a time-proximity frame. In this implementation, the duration of time (e.g., time delay) can be positive or negative.

[0064] consider Figure 7 The figure illustrates an example timing diagram 700 of an example synchronization signal (sync 702), an example negative relaxation (negative relaxation 704), and an example positive relaxation (positive relaxation 706) according to one or more implementations. As shown, the synchronization signal is transmitted periodically, for example, from a timing device integrated within an application processor. Further, it is shown that negative relaxation can occur when the synchronization signal is received at the corresponding image sensor before the start of the nearest neighbor frame (SoF) in a time proximity (e.g., during the vertical blanking time, after the end of stream on). Positive relaxation can occur when the synchronization signal is received at the corresponding image sensor after the start of the nearest neighbor frame in a time proximity (e.g., during stream on).

[0065] If the corresponding image sensor determines that positive relaxation exists, the FSM within the corresponding image sensor is configured to delay the start of a future frame, for example, such that the duration between the future synchronization signal and the start of the future frame is within a tolerance threshold (or closer to it). Delaying the start of a future frame can be achieved by increasing the vertical blanking time. If the corresponding image sensor determines that negative relaxation exists, the FSM within the corresponding image sensor is configured to advance the start of a future frame, for example, such that the duration between the future synchronization signal and the start of the future frame is within a tolerance threshold (or closer to it). Advancing the start of a future frame can be achieved by decreasing the vertical blanking time. In at least some implementations, depending on camera system parameters, the start of a future frame can be delayed or advanced (e.g., gradually) after receiving one or more synchronization signals (e.g., by increasing or decreasing the vertical blanking time). For example, the FSM can gradually advance the start of each successive future frame by a few microseconds or milliseconds until, after one or more synchronization signals, the duration between the future synchronization signal and the start of the future frame is within the tolerance threshold. In this way, the image sensor can self-adjust to align with the synchronization signal.

[0066] Back Figure 6 If the start of a frame is within the tolerance threshold, the image sensor can avoid increasing or decreasing the vertical blanking time (the "No" branch of the new configuration) and generate the start of subsequent frames (e.g., at a predetermined frequency after receiving another synchronization signal). Otherwise, if the start of a frame is not within the tolerance threshold, the image sensor can increase or decrease the vertical blanking time (the "Yes" branch of the new configuration) and generate the start of subsequent frames (e.g., at a predetermined frequency after receiving another synchronization signal). Further shown, each of the three image sensors transmits timing information embedded in the metadata to the metadata module 506. In some implementations, this can be achieved by the respective image sensor first transmitting the metadata data to a wireless communication component, which then wirelessly transmits the metadata to a server system having the metadata module 506.

[0067] Metadata module 506 can extract timing information for each of the two or more image sensors 108, including the time delay between the received synchronization signal and the start of the corresponding frame. Thus, when image capture manager 110 (e.g., operating on image signal processor 202) receives two or more image frames from the two or more image sensors 108, image capture manager 110 can align the two or more image frames based on the timing information in the metadata. Therefore, image capture manager 110 can use fusion module 508 to fuse (e.g., merge) captured frames that are close in time based on the timing information. By doing so, if one or more of the two or more image sensors 108 (e.g., initially) respond at different speeds (e.g., response delays) when receiving frame requests and / or synchronization signals, image capture manager 110 can align the captured image frames to improve the image quality of the fused frame. Example Method

[0068] Example methods 800 and 900 are for illustrative purposes only. Figures 1 to 7 And described according to one or more aspects of self-adjusting synchronization for multi-sensor fusion. These methods are shown as a set of blocks specifying operations to be performed, but are not necessarily limited to the order or combination of operations shown for performance by the respective blocks. For example, any number of described method blocks may be skipped or combined in any order for implementing one or more methods. The technology is not limited to being performed by one or more entities operating on a device.

[0069] Generally, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., a fixed logic circuit system), manual processing, or any combination thereof. Some operations of the example methods can be described in a general context of executable instructions stored on a computer-readable medium 204 located locally and / or remotely on a computer processing system, and implementations can include software applications, programs, functions, etc. Alternatively or additionally, any of the functionalities described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, field-programmable gate arrays (FPGAs), ASICs, application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0070] Figure 8An example method 800 for self-adjusting synchronization for multi-sensor fusion, according to some implementations, is shown. As shown, image sensors (e.g., corresponding image sensors in two or more image sensors 108) can be configured to receive a synchronization signal 802 (“receive sync 802”). Furthermore, the image sensors can be configured to generate a start 806 of a frame, a stream start 808, and a frame end 810. The image sensors can also be configured to count 804 the amount of time between receiving the synchronization signal 802 and the start 806 of frame generation.

[0071] The image sensor can also be configured to determine whether relaxation 812 exists. If relaxation does not exist (the "No" branch of relaxation 812), the image sensor can repeat operations 802 to 812 without adjustment. If relaxation 812 exists (the "Yes" branch of relaxation 812), the image sensor can be configured to determine whether the relaxation is positive or negative 814. If the relaxation is positive (the "Positive" branch), the image sensor can increase the vertical blanking time 816. If the relaxation is negative (the "Negative" branch), the image sensor can decrease the vertical blanking time 818. The image sensor can then repeat one or more of operations 802 to 818.

[0072] Figure 9 Another example method 900 for self-adjusting synchronization for multi-sensor fusion, according to some implementations, is shown. As shown, at 902, the image sensor receives a frame request and a synchronization signal. In one implementation, the synchronization signal is a first synchronization signal among a plurality of synchronization signals. In a further implementation, a plurality of synchronization signals are received at a consistent frequency. At 904, the image sensor generates the start of a frame. In one implementation, the image sensor generates the start of a frame in response to receiving a frame request. In a further implementation, the image sensor generates the start of one or more additional frames in response to receiving a frame request until an end frame request is received. At 906, the image sensor determines the time delay between the synchronization signal and the start of the frame. In one implementation, the image sensor determines the time delay via an integrated counter. At 908, the image sensor adjusts the vertical blanking time sufficiently to advance or delay the start of future frames to effectively reduce the future time delay between future synchronization signals and the start of future frames. Example computing system

[0073] Figure 10 This shows what can be used as a reference to the previous ones. Figures 1 to 9 The various components of the example computing system 1000 for self-adjusting synchronization for multi-sensor fusion can be implemented using any type of client, server, and / or electronic device.

[0074] The computing system 1000 includes a communication device 1002 that enables wired and / or wireless transmission of device data 1004 (e.g., radar data, authentication data, reference data, received data, data being received, data scheduled for broadcast, or data packets). Device data 1004 or other device content may include device configuration settings, media content stored on the device, and / or information associated with the user of the device (e.g., the identity of personnel within the radar field or customized aerial gesture data). Media content stored on the computing system 1000 may include any type of radar, biometric, audio, video, and / or image data. The computing system 1000 includes one or more data inputs 1006 through which any type of data, media content, and / or input can be received, such as human speech, interaction with the radar field, touch input, user-selectable input or interaction (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0075] The computing system 1000 also includes a communication interface 1008, which can be implemented as a serial and / or parallel interface, a wireless interface, any type of network interface, a modem or any other type of communication interface. The communication interface 1008 provides a connection and / or communication link between the computing system 1000 and a communication network through which other electronic, computing, and communication devices transmit data with the computing system 1000.

[0076] The computing system 1000 includes one or more processors 1010 (e.g., any of a microprocessor, controller, or other controller) capable of processing various computer-executable instructions to control the operation of the computing system 1000 and implement techniques for self-adjusting synchronization for multi-sensor fusion, or techniques for self-adjusting synchronization for multi-sensor fusion that can be implemented therein. Alternatively or additionally, the computing system 1000 may be implemented using any or a combination of hardware, firmware, or a fixed logic circuitry system combined with processing and control circuitry generally identified as 1012. Although not shown, the computing system 1000 may include a system bus or data transfer system for various components within a coupling device. The system bus may include any or a combination of different bus architectures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures.

[0077] The computing system 1000 also includes a computer-readable medium 1014, such as one or more memory devices that implement persistent and / or non-transitory data storage (i.e., in contrast to simple signal transmission). Examples of such computer-readable media include RAM, non-volatile memory (e.g., any one or more of ROM, flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device can be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and / or rewritable optical disc (CD), any type of digital versatile disc (DVD), etc. The computing system 1000 may also include a mass storage medium device (storage medium) 1016.

[0078] The computer-readable medium 1014, which can be stored in a data storage facility, may include device data 1004, as well as various device applications 1018 and any other types of information and / or data related to operational aspects of the computing system 1000. For example, the operating system 1020 may be maintained as a computer application having the computer-readable medium 1014 and executed on the processor 1010. The device application 1018 may include an image capture manager 110, which may be any form of control application, software application, signal processing and control module, device-specific native code, abstraction module, machine learning model, etc. The device application 1018 may also include system components, engines, modules, or managers to enable self-adjusting synchronization for multi-sensor fusion. The computing system 1000 may also include or have access to one or more machine learning systems.

[0079] In one example, the example computing system 1000 is a server system having an image capture manager (e.g., image capture manager 110) and an image synchronization manager (e.g., image synchronization manager 112). The image capture manager of the server system is configured to acquire one or more image frames from, for example, a client device, and perform one or more actions for self-adjusting synchronization for multi-sensor fusion. Additional examples

[0080] Additional examples are provided in the following sections.

[0081] Example 1: A method for self-adjusting sensor synchronization, the method comprising: receiving a frame request and a synchronization signal at an image sensor; generating a start of a frame at the image sensor based on the frame request; determining a time delay between the synchronization signal and the start of the frame at the image sensor in response to generating the start of the frame; and adjusting a vertical blanking time of the image sensor based on the time delay, the adjustment being sufficient to advance or delay the start of a future frame of the image sensor, thereby effectively reducing the future time delay between the future synchronization signal and the start of the future frame. Optionally, the frame request and synchronization signal may be transmitted to the image sensor by one or more processors. For example, an image capture manager executing on the one or more processors may transmit the frame request and synchronization signal to the image sensor.

[0082] Example 2: According to the method of Example 1, wherein adjusting the vertical blanking time of the image sensor based on device parameters of the image sensor to advance or delay the start of the future frame of the image sensor by a first increment, the method further includes: receiving a second synchronization signal at the image sensor, the second synchronization signal corresponding to the future synchronization signal; generating the start of a second frame at the image sensor, the start of the second frame corresponding to the start of the future frame; determining a second time delay between the second synchronization signal and the start of the second frame at the image sensor in response to generating the start of the second frame, the second time delay corresponding to the future time delay; and adjusting the vertical blanking time of the image sensor based on the determined second time delay, the adjustment being sufficient to advance or delay the start of the second future frame of the image sensor, thereby effectively reducing the second future time delay between the second future synchronization signal and the start of the second future frame.

[0083] Example 3: According to the method of Example 2, the vertical blanking time of the image sensor is adjusted based on the device parameters of the image sensor to advance or delay the start of the second future frame of the image sensor by a second increment, and wherein the second increment is less than or equal to the first increment.

[0084] Example 4: The method according to any of the foregoing examples further includes: in response to adjusting the vertical blanking time of the image sensor, determining that another time delay between another synchronization signal and the start of another frame is within a tolerance threshold; and avoiding adjusting the vertical blanking time of the image sensor.

[0085] Example 5: The method according to any of the preceding examples, wherein: the time delay between the synchronization signal and the start of the frame indicates positive relaxation; and adjusting the vertical blanking time based on the positive relaxation indication includes delaying the start of the future frame by increasing the vertical blanking time.

[0086] Example 6: The method according to any one of Examples 1 to 4, wherein: the time delay between the synchronization signal and the start of the frame indicates negative relaxation; and adjusting the vertical blanking time based on the indication of negative relaxation includes advancing the start of the future frame by reducing the vertical blanking time.

[0087] Example 7: The method according to any of the foregoing examples further includes: programming a register of the image sensor before transmitting the synchronization signal, the programming being used to effectively initialize a tolerance threshold for an acceptable length of the time delay. For example, before the one or more processors transmit the synchronization signal to the image sensor, the one or more processors may transmit instructions to the image sensor to program the register of the image sensor.

[0088] Example 8: The method according to any of the preceding examples, wherein the image sensor is one of a plurality of image sensors, each of the plurality of image sensors: receives the frame request and the synchronization signal; and generates the start of a corresponding frame based on the frame request.

[0089] Example 9: According to the method of Example 8, wherein: the time delay between the synchronization signal and the start of the frame is determined by at least one of the plurality of image sensors in response to generating the start of the corresponding frame; and the vertical blanking time is adjusted by the at least one of the plurality of image sensors based on the time delay.

[0090] Example 10: The method according to Example 9 further includes: in response to adjusting the vertical blanking time by at least one of the plurality of image sensors, receiving a plurality of synchronization signals at each of the plurality of image sensors, the plurality of synchronization signals being received sequentially and at a consistent frequency; generating a plurality of frame start and corresponding plurality of frame end at each of the plurality of image sensors for one or more of the plurality of synchronization signals, the start of a corresponding frame in the plurality of frame start and the end of a corresponding corresponding frame in the corresponding plurality of frame end indicating a corresponding generated image frame in a plurality of generated image frames; generating timing metadata at each of the plurality of image sensors, the timing metadata being correlated with a time delay between the frame start and the synchronization signal of one or more of the plurality of generated image frames; aligning two or more of the plurality of generated image frames based on the timing metadata correlated with the time delay; fusing the two or more generated image frames sufficient to produce a fused frame based on the alignment of the two or more generated image frames; and presenting the fused frame on a display of an electronic device. Optionally, the alignment, blending, and rendering can be performed by the one or more processors. Optionally, the one or more image sensors can transmit the timing metadata to the one or more processors.

[0091] Example 11: According to the method of Example 10, the plurality of synchronization signals, received sequentially and at the same frequency, are received at a frequency corresponding to a first frame rate of a first image sensor among the plurality of image sensors, wherein the first frame rate of the first image sensor is faster than a second frame rate of a second image sensor among the plurality of image sensors. For example, the one or more processors may know or be able to access the frame rate of the image sensors. For example, the one or more processors may access memory storing the frame rate, or may receive data indicating the frame rate of the image sensors. The one or more processors may transmit the synchronization signals at a frequency corresponding to the fastest frame rate of the image sensors.

[0092] Example 12: The method according to any of the preceding examples, wherein the plurality of image sensors includes three image sensors, each of the three image sensors being set with different zoom and / or exposure levels.

[0093] Example 13: According to the method of Example 12, wherein: determining the time delay between the synchronization signal and the start of the frame includes determining a first time delay and a second time delay by a first image sensor and a second image sensor of the three image sensors, the first time delay being different from the second time delay; and adjusting the vertical blanking time by at least one of the plurality of image sensors includes adjusting the vertical blanking time of the first image sensor and the second image sensor respectively based on the first time delay and the second time delay, the adjustment being sufficient to: advance the start of the future frame of the first image sensor and the second image sensor; delay the start of the future frame of the first image sensor and the second image sensor; advance the start of the future frame of the first image sensor and delay the start of the future frame of the second image sensor; or delay the start of the future frame of the first image sensor and advance the start of the future frame of the second image sensor.

[0094] Example 14: A device including an image sensor configured to perform the method according to any one of Examples 1 to 6.

[0095] Example 15: The device according to Example 14 further includes one or more additional image sensors and at least one processor, said at least one processor being configured to perform the method according to any one of Examples 7 to 13.

[0096] Example 16: A method for a self-adjusting synchronization electronic device for multi-sensor fusion, the method comprising: transmitting a synchronization signal to each of two or more image sensors; generating a start of a frame at each of the two or more image sensors based on the synchronization signal; determining a time delay between the synchronization signal and the start of the frame at each of the two or more image sensors in response to generating the start of the frame; and adjusting the vertical blanking time of at least one of the two or more image sensors based on the time delay, the adjustment being sufficient to advance or delay the start of a future frame.

[0097] Example 17: According to the method of Example 16, the time delay between the synchronization signal and the start of the frame indicates slack.

[0098] Example 18: The method according to Example 17, wherein adjusting the vertical blanking time based on the positive relaxation includes delaying the start of a future frame by increasing the vertical blanking time.

[0099] Example 19: According to the method of Example 16, the time delay between the synchronization signal and the start of the frame indicates negative relaxation.

[0100] Example 20: The method according to Example 19, wherein adjusting the vertical blanking time based on the negative relaxation includes advancing the start of a future frame by reducing the vertical blanking time.

[0101] Example 21: The method according to Example 16 further includes: programming a register of at least one of the two or more image sensors before transmitting the synchronization signal, the programming effectively initializing a tolerance threshold of the at least one image sensor.

[0102] Example 22: The method according to Example 21, wherein the vertical blanking time of the at least one image sensor is adjusted to advance or delay the start of a future frame within a corresponding tolerance threshold of the at least one image sensor. in conclusion

[0103] Unless the context otherwise requires, the use of the word “or” herein may be interpreted as a term that is “inclusive or” or that permits the inclusion or application of one or more items connected by the word “or” (e.g., the phrase “A or B” may be interpreted as permitting only “A”, only “B”, or both “A” and “B”). Furthermore, as used herein, the phrase “at least one” in the list of references refers to any combination of these items, including single members. For example, “at least one of a, b, or c” may cover a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c). Further, the items represented in the figures and the terms discussed herein may indicate one or more items or terms, and therefore may be used interchangeably with the singular or plural forms of items and terms in this written description.

[0104] While implementations of self-adjusting synchronization for multi-sensor fusion have been described in language specific to particular features and / or methods, the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Instead, specific features and methods are disclosed as exemplary implementations of self-adjusting synchronization for multi-sensor fusion.

Claims

1. A method for self-adjusting sensor synchronization, the method comprising: Receive frame requests and synchronization signals at the image sensor; Based on the frame request, the start of a frame is generated at the image sensor; At the image sensor, in response to the start of frame generation, a time delay between the synchronization signal and the start of the frame is determined; as well as Based on the time delay, the vertical blanking time of the image sensor is adjusted, and the adjustment is sufficient to advance or delay the start of future frames of the image sensor, thereby effectively reducing the future time delay between the future synchronization signal and the start of the future frame.

2. The method of claim 1, wherein adjusting the vertical blanking time of the image sensor based on device parameters of the image sensor to advance or delay the start of the future frame of the image sensor by a first increment, the method further comprising: A second synchronization signal is received at the image sensor, the second synchronization signal corresponding to the future synchronization signal; The start of the second frame is generated at the image sensor, and the start of the second frame corresponds to the start of the future frame; In response to the start of generating the second frame, a second time delay is determined at the image sensor between the second synchronization signal and the start of the second frame, the second time delay corresponding to the future time delay; as well as Based on the determined second time delay, the vertical blanking time of the image sensor is adjusted, such adjustment being sufficient to advance or delay the start of the second future frame of the image sensor, thereby effectively reducing the second future time delay between the second future synchronization signal and the start of the second future frame.

3. The method of claim 2, wherein adjusting the vertical blanking time of the image sensor based on the device parameters of the image sensor to advance or delay the start of the second future frame of the image sensor by a second increment, and wherein the second increment is less than or equal to the first increment.

4. The method according to any one of the preceding claims, further comprising: In response to adjusting the vertical blanking time of the image sensor, determine that another time delay between the other synchronization signal and the start of another frame is within a tolerance threshold; and Avoid adjusting the vertical blanking time of the image sensor.

5. The method according to any one of the preceding claims, wherein: The time delay between the synchronization signal and the start of the frame indicates positive slack; and Adjusting the vertical blanking time based on the positive relaxation instruction includes delaying the start of the future frame by increasing the vertical blanking time.

6. The method according to any one of claims 1 to 4, wherein: The time delay between the synchronization signal and the start of the frame indicates negative relaxation; and Adjusting the vertical blanking time based on the negative relaxation instruction includes advancing the start of the future frame by reducing the vertical blanking time.

7. The method according to any one of the preceding claims further comprises: Before transmitting the synchronization signal, the registers of the image sensor are programmed to effectively initialize a tolerance threshold for an acceptable length of time delay.

8. The method according to any one of the preceding claims, wherein the image sensor is one of a plurality of image sensors, each of the plurality of image sensors: Receive the frame request and the synchronization signal; and The start of generating the corresponding frame is based on the frame request.

9. The method according to claim 8, wherein: The time delay between the synchronization signal and the start of the frame is determined by at least one of the plurality of image sensors in response to generating the start of the corresponding frame; and The adjustment of the vertical blanking time is made by at least one of the plurality of image sensors based on the time delay.

10. The method of claim 9, further comprising: After the vertical blanking time is adjusted by at least one of the plurality of image sensors, a plurality of synchronization signals are received at each of the plurality of image sensors, the plurality of synchronization signals being received sequentially and at a consistent frequency; For one or more of the plurality of synchronization signals, at each of the plurality of image sensors, a plurality of frame beginnings and corresponding plurality of frame endings are generated, wherein the beginning of the corresponding frame in the plurality of frame beginnings and the end of the corresponding frame in the corresponding plurality of frame endings indicate the corresponding generated image frame in the plurality of generated image frames. Timing metadata is generated at each of the plurality of image sensors, the timing metadata being related to the time delay between the start of a frame and a synchronization signal of one or more of the plurality of generated image frames; Based on the timing metadata associated with the time delay, align two or more of the generated image frames from the plurality of generated image frames; Based on the alignment of the two or more generated image frames, the two or more generated image frames sufficient to produce a fused frame are fused. as well as The fused frame is displayed on the screen of the electronic device.

11. The method of claim 10, wherein the plurality of synchronization signals, received sequentially and at the consistent frequency, are received at a frequency corresponding to a first frame rate of a first image sensor among the plurality of image sensors, the first frame rate of the first image sensor being faster than a second frame rate of a second image sensor among the plurality of image sensors.

12. The method according to any one of claims 9 to 11, wherein the plurality of image sensors comprises three image sensors, each of the three image sensors being configured with different zoom and / or exposure levels.

13. The method according to claim 12, wherein: Determining the time delay between the synchronization signal and the start of the frame includes determining a first time delay and a second time delay by the first and second image sensors of the three image sensors, wherein the first time delay is different from the second time delay; and Adjusting the vertical blanking time by at least one of the plurality of image sensors includes adjusting the vertical blanking times of the first image sensor and the second image sensor based on the first time delay and the second time delay, respectively, the adjustment being sufficient to: The start of the future frame is advanced for both the first and second image sensors; Delay the start of the future frame for the first image sensor and the second image sensor; The start of the future frame of the first image sensor is advanced and the start of the future frame of the second image sensor is delayed; or The start of the future frame of the first image sensor is delayed, and the start of the future frame of the second image sensor is advanced.

14. An apparatus comprising: An image sensor configured to perform the method according to any one of claims 1 to 6.

15. The apparatus of claim 14, further comprising: One or more additional image sensors and at least one processor, said at least one processor being configured to perform the method according to any one of claims 7 to 13.

Images