Sensor readout mode for high-resolution and low-light imaging synchronized with light detection and ranging timing

Abstract

This disclosure describes devices, systems, and methods for obtaining image frames with variable resolutions in synchronization with a clock source. An example device may include an image sensor, a clock input, and a controller. The controller includes at least one processor and memory. The at least one processor is operable to execute program instructions stored in the memory to perform operations. The operations include receiving a clock signal via the clock input. The clock signal is a periodic signal that defines at least one scanning interval. The operations also include causing the image sensor to capture a full-resolution image frame during the scanning interval. The operations also include causing the image sensor to capture at least one reduced-resolution image frame during the scanning interval.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 16 / 688,443, filed November 19, 2019, the contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to a sensor readout mode for high resolution and low light imaging synchronized with light detection and ranging timing. Background Art

[0004] The sensor system can include a variety of different types of sensors, such as image capture systems (e.g., cameras), radars, and / or light detection and ranging (LIDAR or Lidar) systems. Such sensor systems can be used, for example, in conjunction with autonomous or semi-autonomous robots and / or vehicles (e.g., self-driving cars / trucks). One challenge with these types of sensor systems is synchronizing image capture (using a camera) and lidar scanning. For example, conventional systems are not designed to capture high-resolution images and low-light images in a synchronized manner using the same camera during a single lidar scanning interval. Summary of the Invention

[0005] The present disclosure generally relates to sensor systems and methods that provide time-coordinated sensor information from at least two different types of sensors.

[0006] In a first aspect, a device is provided. The device includes an image sensor, a clock input, and a controller having at least one processor and memory. The at least one processor is operable to execute program instructions stored in the memory to perform operations. The operations include receiving a clock signal via the clock input. The clock signal is a periodic signal defining at least one scanning interval. The operations also include causing the image sensor to capture a full-resolution image frame during the scanning interval. The operations also include causing the image sensor to capture at least one reduced-resolution image frame during the scanning interval.

[0007] In a second aspect, a system is provided. The system includes an image sensor, a light detection and ranging (lidar) device, and a controller having at least one processor and a memory. The at least one processor is operable to execute program instructions stored in the memory to perform operations. The operations include causing the lidar device to scan a field of view based on a scan timing sequence. The scan timing sequence includes a plurality of scan intervals. The operations also include causing the image sensor to capture a full-resolution image frame during a given scan interval. The operations also include causing the image sensor to capture at least one reduced-resolution image frame during a given scan interval.

[0008] In a third aspect, a method is provided. The method includes causing a lidar device to scan a field of view based on a scan timing sequence. The scan timing sequence includes a plurality of scan intervals. The method also includes causing an image sensor to capture a full-resolution image frame during a given scan interval. The full-resolution image frame includes a correlated double-sampled image. The method also includes causing the image sensor to capture at least one reduced-resolution image frame during the given scan interval.

[0009] Other aspects, embodiments, and implementations will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference to the accompanying drawings as appropriate. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Shown is a schematic block representation of an apparatus according to example embodiments.

[0011] Figure 2 shows a schematic diagram according to an example embodiment Figure 1 part of the device.

[0012] Figure 3A An operation scenario according to an example embodiment is shown.

[0013] Figure 3B An operation scenario according to an example embodiment is shown.

[0014] Figure 3C An operation scenario according to an example embodiment is shown.

[0015] Figure 4 Shown is a schematic block representation of a system according to an example embodiment.

[0016] Figure 5 A method according to an example embodiment is shown.

[0017] Figure 6 An operation scenario according to an example embodiment is shown.

[0018] Figure 7 An operation scenario according to an example embodiment is shown.

[0019] Figure 8 An operation scenario according to an example embodiment is shown. DETAILED DESCRIPTION

[0020] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0021] Therefore, the example embodiments described herein are not meant to be limiting. As generally described herein, and as shown in the accompanying drawings, various aspects of the present disclosure may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0022] In addition, unless the context indicates otherwise, the features shown in the various figures may be used in combination with each other. Therefore, the figures should generally be viewed as forming aspects of one or more overall embodiments, with it being understood that not all of the features shown are necessary for every embodiment.

[0023] I. Overview

[0024] In a conventional sensor system including a lidar and a single camera, the high-resolution image camera image and the low-light camera image are captured with asynchronous timing relative to the lidar scan timing. For example, in a conventional "staggered resolution readout mode" or "serial CDS imaging mode," the lidar scan period can be approximately 100 milliseconds (ms). That is, the lidar can be configured to scan a predetermined area or sector of three-dimensional space during a given lidar scan period. In such a conventional scenario, a 12-megapixel image sensor / camera can be configured to capture a single 12-megapixel "high-resolution" correlated double sampling (CDS) image frame, followed by a 3-megapixel "low-resolution, low-light" CDS image frame. As an example, the 12-megapixel CDS image frame can have a total exposure and readout time of approximately 60-70 milliseconds. The subsequent 3-megapixel CDS image frame can have a total exposure and readout time of approximately 30 milliseconds. However, in some cases, the subsequent low-luminance CDS image frame may be delayed due to readout time, integration time, latency, etc., so that the low-luminance CDS image frame is completed more than 30 milliseconds (e.g., 34 milliseconds) after the initial lidar scan cycle is completed.

[0025] The devices, systems, and methods described herein provide various ways to temporally coordinate the various functions of high-resolution image capture, reduced-resolution image capture, and lidar scanning. For example, some embodiments can provide high-resolution camera images that are temporally and spatially correlated with lidar-based point cloud map data, with the two being acquired at synchronized frame rates. Furthermore, embodiments provide the ability to capture one or more reduced-resolution images using the same camera, all during a single lidar scan interval.

[0026] Other aspects, embodiments, and implementations will become apparent to those of ordinary skill in the art by reading the following detailed description and, where appropriate, referring to the accompanying drawings.

[0027] II. Sample Equipment

[0028] Figure 1 A schematic block representation of a device 100 according to an example embodiment is shown. The device 100 includes an image sensor 110 and a clock input 120. In an example embodiment, the image sensor 110 may include a plurality of light sensing elements 112. The plurality of light sensing elements 112 are spatially grouped into a plurality of light sensing regions 114 (e.g., a plurality of low-resolution pixels). The image sensor 110 is configured to capture a full-resolution image frame 130 and another image frame (e.g., a reduced-resolution image frame 140) corresponding to at least one light sensing region 114 during a single scanning interval 124.

[0029] In some embodiments, image sensor 110 may include a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, and / or an active pixel sensor. It will be understood that other types of image sensors are possible and contemplated within the context of the present disclosure.

[0030] In some embodiments, image sensor 110 may include more than 12 million light sensing elements 112 (eg, 12 million pixels, 15 million pixels, or more).

[0031] The device 100 also includes a controller 150 having at least one processor 152 and a memory 154. In some embodiments, the controller 150 may include at least one of a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). Additionally or alternatively, the at least one processor 152 may include a general-purpose processor or a special-purpose processor (e.g., a digital signal processor, etc.). The processor 152 may be configured to execute computer-readable program instructions stored in the memory 154. In some embodiments, the processor 152 may execute the program instructions to provide at least some of the functions and operations described herein.

[0032] Memory 154 may include or take the form of one or more computer-readable storage media that are readable or accessible by one or more processors 152. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, which may be integrated in whole or in part within at least one of the one or more processors 152. In some embodiments, memory 154 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other embodiments, memory 154 may be implemented using two or more physical devices.

[0033] In some embodiments, operation may include receiving a clock signal 122 via a clock input 120. In such a scenario, the clock signal 122 is a periodic signal that defines at least one scan interval 124. The clock signal 122 may be an analog or digital signal that oscillates between at least a high state (e.g., +5 volts) and a low state (e.g., -5 volts). In some embodiments, the clock signal 122 may be used as a trigger for synchronizing digital circuits. In an example embodiment, the clock signal 122 may be generated by a clock signal generator. In some scenarios, the clock signal generator may be part of the device 100 and / or coupled to the device 100. Alternatively, the clock signal generator need not be physically located near the device 100.

[0034] Operations also include causing the image sensor 110 to capture a full resolution image frame 130 during the scanning interval 124. In some embodiments, the full resolution image frame 130 may comprise a correlated double sampled (CDS) image.

[0035] As used herein, correlated double sampling can include methods of measuring voltage or current to eliminate unwanted signals (or absence of signals) associated with defective pixels (e.g., "dead" or "stuck" pixels). For example, when measuring the output of an image sensor, the output of the sensor can be measured twice. In such a process, the sensor output can be measured once under "known" conditions and once under "unknown" conditions. The value measured under the known conditions is then subtracted from the unknown conditions to generate a value that has a known relationship to the physical quantity being measured (in this case, photons received from the environment).

[0036] In some embodiments, correlated double sampling can be used as a noise reduction technique. For example, at the end of each integration period, the reference voltage of a given image sensor pixel (i.e., the pixel voltage after it is reset) can be subtracted from the pixel's signal voltage (i.e., the pixel voltage at the end of the integration period). This subtraction can offset and / or mitigate thermal noise (e.g., kTC noise) associated with the capacitance of the corresponding light sensing element of the image sensor.

[0037] The operations further include causing the image sensor 110 to capture at least one reduced-resolution image frame 140 during the scanning interval 124. In some embodiments, the reduced-resolution image frame 140 may be a non-correlated double-sampled image. That is, the reduced-resolution image frame 140 may be read out from the image sensor without performing image subtraction or normalization.

[0038] In some embodiments, the operations may further include receiving information indicating a desired region of interest 160. In such a scenario, causing the image sensor 110 to capture at least one reduced resolution image frame 140 may include causing the image sensor 110 to capture another image frame (e.g., with reference to FIG. Figure 2 During a given scanning interval 124 , another image frame may correspond to a desired region of interest 160 .

[0039] Figure 2 shows a schematic diagram according to an example embodiment Figure 1 A portion 200 of the device 100. Figure 2As shown in , capturing at least one reduced-resolution image frame 140 may include capturing a first reduced-resolution image frame 140a using the image sensor 110. In such a scenario, the first reduced-resolution image frame 140a may include a first non-correlated double-sampled image 142a. In an example embodiment, the first non-correlated double-sampled image 142a may be obtained from the same exposure as the full-resolution image frame 130. In such a scenario, the first reduced-resolution image frame 140a may be provided faster than the CDS image because no further exposure is required.

[0040] like Figure 2 As shown in , capturing at least one reduced resolution image frame 140 may additionally include capturing a second reduced resolution image frame 140b. In such a scenario, the second reduced resolution image frame 140b may include a dark frame 144. In some embodiments, the image sensor 110 may be used to provide the second reduced resolution image frame 140b based on a short "dark" exposure period followed by a readout period.

[0041] In some embodiments, dark frame 144 may be captured using the same exposure time, ISO sensitivity, and ambient temperature as first reduced-resolution image frame 140a. However, it will be appreciated that dark frame 144 may be obtained using other exposure parameters. In some scenarios, an opaque shutter mechanism may be utilized to prevent light from reaching image sensor 110 when capturing dark frame 144.

[0042] In some embodiments, dark frame 144 is used for sensor noise reduction. In this scenario, dark frame 144 can be read out with the charge transfer gate TX closed. Thus, the image can be read out as a dark frame, but may include the same or similar noise as the CDS image. It will be understood that other ways of capturing dark frame 144 to obtain sensor noise information are also possible and contemplated.

[0043] Additionally, the operations performed by the controller 150 may include performing dark image subtraction on the first reduced-resolution image frame 140a based on the second reduced-resolution image frame 140b. In this scenario, dark image subtraction can correct for fixed pattern noise, such as noise associated with dark current and / or "amp glow." Visible fixed pattern noise may be caused by "hot pixels" (e.g., pixels with higher than normal dark current), stuck pixels, and / or flickering pixels.

[0044] In some embodiments, full resolution image frame 130 is formed from information received from each of the plurality of light sensing elements 112. For example, where image sensor 110 has 12 million light sensing elements 112, the corresponding full resolution image 130 may include a resolution of 12 megapixels.

[0045] Furthermore, reduced resolution image frame 140 is formed from information received from a subset of the plurality of light sensing elements 112. That is, where image sensor 110 has 12 million light sensing elements 112, the corresponding reduced resolution image frame 140 may include a resolution of 3 megapixels.

[0046] In some embodiments, the scanning interval 124 can be less than 40 milliseconds. For example, the scanning interval 124 can correspond to a clock signal 122 (e.g., a lidar clock signal) having a period of approximately 30 milliseconds. However, it will be understood that longer or shorter scanning intervals are also possible and contemplated.

[0047] In various examples, the clock input 120 can be based on a scan timing sequence 430 of a light detection and ranging (lidar) device (e.g., lidar device 410), as shown in FIG. Figure 4 As shown and described.

[0048] In such a scenario, full-resolution image frames 130 may be captured while the lidar device is scanning a field of view (e.g., field of view 420). Additionally or alternatively, at least one reduced-resolution image frame 140 may be captured while the lidar device is not scanning the field of view.

[0049] Figure 3A An operational scenario 300 according to an example embodiment is shown. The operational scenario 300 may illustrate a "serial image capture" scenario. As an example, to provide a first full-resolution CDS image frame 306a, during a scanning interval 124a (e.g., between t0 and t1), the image sensor 110 may be exposed and accumulate charge during an exposure 302a, and the accumulated charge may be read out during a readout 304a. Furthermore, to provide a first reduced-resolution CDS image frame 308a, the image sensor 110 may be exposed and accumulate charge again during an exposure 302b, and the corresponding accumulated charge may be read out during a readout 304b.

[0050] The serial image capture process shown in operating scenario 300 can continue during scan intervals 124a, 124b, and 124c. For example, a second full-resolution image frame 306b can be captured via exposure 302c and readout 304c, while a second reduced-resolution image frame 308b can be captured via exposure 302d and readout 304d. However, the serial capture process may result in image frames that are not synchronized with scan intervals 124a-c or clock signal 122. Consequently, information about objects and other features in the image frames captured in operating scenario 300 may be more difficult to combine and / or compare with other types of captured information about the environment obtained based on scan intervals 124a-c. For example, in the case of a lidar device, the image sensor data captured during the serial capture process may be spatially offset relative to the lidar data and / or more difficult to use for sensor fusion and / or other perception determinations.

[0051] Figure 3B An operational scenario 320 is shown according to an example embodiment. The operational scenario 320 may include, during a scanning interval 124a, capturing a first full-resolution image frame 326a during exposure 322a and readout 324a. Subsequently, a first reduced-resolution non-CDS image frame 328a may be captured during readout 324b. The first reduced-resolution non-CDS image frame 328a may utilize the same exposure 322a as the first full-resolution image frame 326a. However, due to the non-CDS nature of the first reduced-resolution image frame 328a, the resulting image may be noisier or of lower quality than an equivalent CDS reduced-resolution image frame.

[0052] During subsequent scanning intervals (e.g., scanning intervals 124b and 124c), a second full-resolution image frame 326b can be captured during exposure 322b and readout 324c, while a second reduced-resolution image frame 328b can be captured during readout 324d. Although only partially shown, a third full-resolution image frame 326c can be captured during exposure 322c and the corresponding readout time.

[0053] By operating according to the operating scenario 320, the device 100 can provide full-resolution and reduced-resolution image frames synchronized with the scanning intervals 124a-c. Synchronization with the scanning intervals 124a-c can make sensor fusion and / or perception tasks more efficient and less computationally intensive.

[0054] Figure 3CAn operational scenario 330 is shown according to an example embodiment. The operational scenario 330 includes capturing a first full-resolution image frame 336a corresponding to an exposure 332a and a readout 334a. Thereafter, the operational scenario 330 includes capturing a first reduced-resolution image frame 338a during the readout 334b. Next, the operational scenario 330 includes capturing a dark frame 340a associated with the exposure 332b and the readout 334c. In the operational scenario 330, the first reduced-resolution image frame 338a may be non-CDS, while the dark frame 340a may be an image obtained with a closed charge transfer gate (e.g., a "closed" electronic shutter) or with a closed physical shutter.

[0055] In subsequent scanning intervals 124b and 124c, the capture sequence may repeat capturing a second full-resolution image frame 336b, a second reduced-resolution image frame 338b, and a second dark frame 340b.

[0056] By utilizing such an operating mode, device 100 may be configured to provide a full-resolution image frame and a dark-current-corrected reduced-resolution image frame during each scanning interval.

[0057] III. Example System

[0058] Figure 4 4 shows a schematic block representation of a system 400 according to an example embodiment. The system 400 may include Figure 1 Similar or identical elements are shown and described with respect to the apparatus 100. For example, the system 400 includes the image sensor 110.

[0059] The system 400 also includes a light detection and ranging (lidar) device 410. The lidar device 410 can be configured to provide information (e.g., point cloud data) about one or more objects (e.g., location, shape, etc.) in a given environment. In an example embodiment, the lidar system can provide point cloud information, object information, mapping information, or other information to a vehicle. The vehicle can be a semi-autonomous or fully autonomous vehicle. For example, the vehicle can be a self-driving car, an autonomous drone, an autonomous truck, or an autonomous robot. Other types of vehicles and lidar systems are also contemplated herein.

[0060] Furthermore, the system 400 includes a controller 150, which includes at least one processor 152 and a memory 154. The at least one processor 152 is operable to execute program instructions stored in the memory 154 in order to perform operations.

[0061] In some embodiments, operations may include causing the lidar device 410 to scan the field of view 420 based on a scan timing sequence 430. In such a scenario, the scan timing sequence 430 may include a plurality of scan intervals 124. Operations may also include causing the image sensor 110 to capture a full-resolution image frame 130 during a given scan interval 124. Operations may additionally include causing the image sensor 110 to capture at least one reduced-resolution image frame 140 during a given scan interval 124. In some embodiments, the full-resolution image frame 130 may be captured while the lidar device 410 is scanning the field of view 420. Additionally or alternatively, the at least one reduced-resolution image frame 140 may be captured while the lidar device 410 is not scanning the field of view 420.

[0062] like Figure 4 As shown in FIG, in some embodiments, the controller 150 can transmit a lidar control signal 158 to the lidar device 410. The lidar control signal 158 can be used to maintain and / or change the operation of the lidar device 410.

[0063] In some embodiments, the Figure 3A 、 Figure 3B and Figure 3C The system 400 is controlled by the operating scenarios 300 , 320 and / or 330 .

[0064] IV. Example Methods

[0065] Figure 5 5. A method 500 is shown according to an example embodiment. It will be understood that the method 500 may include fewer or more steps or blocks than those explicitly shown or disclosed herein. Furthermore, the various steps or blocks of the method 500 may be performed in any order, and each step or block may be performed one or more times. In some embodiments, some or all of the steps or blocks of the method 500 may be different from those described with respect to FIG. Figure 1 and Figure 4 The elements shown and described are related to the device 100 and / or system 400. Some steps or blocks of the method 500 may be related to Figure 3B-3C 、 Figure 6 and Figure 7 to show and describe.

[0066] Block 502 includes causing a lidar device (eg, lidar device 410 ) to scan a field of view (eg, field of view 420 ) based on a scan timing sequence (eg, scan timing sequence 430 ). The scan timing sequence includes a plurality of scan intervals (eg, scan interval 124 ).

[0067] Block 504 includes causing an image sensor (eg, image sensor 110 ) to capture a full-resolution image frame (eg, full-resolution image frame 130 ) during a given scanning interval. In such a scenario, the full-resolution image frame may include a correlated double-sampled image.

[0068] Block 506 includes causing the image sensor to capture at least one reduced-resolution image frame during a given scanning interval. In such a scenario, capturing the at least one reduced-resolution image frame may include capturing a first reduced-resolution image frame (e.g., first reduced-resolution image frame 338a). In some examples, the first reduced-resolution image frame may include a non-correlated double-sampled image. Additionally, capturing the at least one reduced-resolution image frame may also include capturing a second reduced-resolution image frame (e.g., second reduced-resolution image frame 340a). In some embodiments, the second reduced-resolution image frame may include a dark frame. Method 500 also includes performing dark image subtraction on the first reduced-resolution image frame based on the second reduced-resolution image frame.

[0069] In some embodiments, a full-resolution image frame can be captured while the lidar device is scanning the field of view. As an example, at least one reduced-resolution image frame can be captured while the lidar device is not required to scan the field of view.

[0070] In some embodiments, a given scanning interval may be less than 40 milliseconds (eg, 20-30 milliseconds). However, other scanning intervals are possible and contemplated.

[0071] Figure 6 An operational scenario 600 according to an example embodiment is shown. The operational scenario 600 includes an image sensor 610, a smart sensor 620, and a central controller 630. The operational scenario 600 may include obtaining information (e.g., pixel charge) using the image sensor 610. Thereafter, the operational scenario 600 may include processing at least a portion of the obtained information using the smart sensor 620. The smart sensor 620 may include one or more circuits configured to efficiently transmit and / or filter the information provided by the image sensor 610 before transmitting the information to the central controller 630, based on, for example, a desired field of view. In turn, the central controller 630 may process the information and / or present the processed information to a user.

[0072] Figure 7An operational scenario 700 according to an example embodiment is shown. In some embodiments, operational scenario 700 may include image sensor 610 capturing a high-resolution image. Next, image sensor 610 may capture a low-resolution non-CDS image. Subsequently, image sensor 610 may capture a low-resolution dark frame. As described herein, the dark frame may be captured with charge transfer gate TX closed. Such an image may include similar or identical noise information as the CDS image, providing a noisy reference frame that can be subtracted from the low-resolution non-CDS image.

[0073] In some embodiments, the smart sensor 620 can be configured to perform dark image subtraction between the two low-resolution images in an attempt to reduce noise generated by faulty pixels and / or associated readout circuitry. Thereafter, the smart sensor 620 can be configured to further adjust or finalize the image and send the processed frame to the central controller 630. After the processed frame is transmitted, the method can be repeated again during a subsequent scanning interval.

[0074] Figure 8 Operation scenarios 800 and 810 according to example embodiments are shown. Operation scenario 800 illustrates the same Figure 3A The serial image capture process is similar or identical to that described above. In the serial image capture process, a CDS full-resolution image frame is captured, immediately followed by a CDS reduced-resolution image frame. However, due to the length of time required to capture back-to-back CDS image frames, after the first full-resolution image frame, subsequent image frames may be "out of sync" with respect to the lidar scan interval.

[0075] Operation scenario 810 shows the reference Figure 3C A similar or identical process to that described above may be employed. In such an embodiment, a CDS full-resolution image frame (e.g., 12-megapixel resolution) may be captured by triggering on the rising or falling edge of a scan trigger pulse (e.g., trigger pulse 126). Thereafter, a non-CDS reduced-resolution image frame (e.g., 3-megapixel resolution) may be captured, followed by a reduced-resolution dark frame. In this manner, CDS full-resolution image frames and non-CDS, dark-image-subtracted, reduced-resolution image frames may be provided "in sync" with the lidar scan interval.

[0076] It will be appreciated that while the image frames are captured in a particular order, other orders of image frame capture are contemplated and possible.

[0077] The particular arrangements shown in the accompanying drawings should not be considered restrictive. It should be understood that other embodiments may include more or less of each element than shown in a given drawing. In addition, some of the elements shown may be combined or omitted. In addition, the illustrative embodiments may include elements not shown in the accompanying drawings.

[0078] The steps or blocks representing information processing may correspond to circuits that can be configured to perform the specific logical functions of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing information processing may correspond to a module, a fragment, a physical computer (e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)), or a portion of a program code (including related data). The program code may include one or more instructions executable by a processor to implement the specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer-readable medium, such as a storage device including a disk, a hard drive, or other storage medium.

[0079] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media for short-term storage of data, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media for long-term storage of program code and / or data. Thus, computer-readable media may include secondary or permanent long-term storage, such as read-only memory (ROM), optical or magnetic disks, or compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. For example, computer-readable media may be considered to be computer-readable storage media, or tangible storage devices.

[0080] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for purposes of illustration only and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

1. A system comprising: Image sensor; Light detection and ranging lidar equipment; Clock input; as well as A controller comprising at least one processor and a memory, wherein the at least one processor is operable to execute program instructions stored in the memory to perform operations, the operations comprising: receiving a clock signal via a clock input, wherein the clock signal is a periodic signal defining at least one scanning interval of the lidar device; causing the image sensor to capture full-resolution image frames during scanning intervals of the lidar device; and During a scanning interval of the lidar device, an image sensor is caused to capture at least one reduced-resolution image frame, wherein the at least one reduced-resolution image frame has a lower resolution than the full-resolution image frame.

2. The system of claim 1, wherein: The image sensor includes a plurality of light sensing elements, wherein the plurality of light sensing elements are spatially grouped into a plurality of light sensing regions, wherein the image sensor is configured to capture a full-resolution image frame and another image frame corresponding to at least one light sensing region during a single scanning interval.

3. The system of claim 2, wherein: The full resolution image frame is formed from information received from each of the plurality of light sensing elements.

4. The system of claim 2, wherein: The reduced resolution image frame is formed from information received from a subset of the plurality of light sensing elements.

5. The system of claim 1, wherein: The operations further include: Information indicative of a desired region of interest is received, wherein causing the image sensor to capture at least one reduced-resolution image frame includes causing the image sensor to capture another image frame corresponding to the desired region of interest during a given scanning interval.

6. The system of claim 1, wherein: The full-resolution image frame includes a correlated double-sampled image.

7. The system of claim 1, wherein: The reduced resolution image frames include non-correlated double sampled images.

8. The system of claim 1, wherein: Capturing at least one reduced-resolution image frame includes: capturing a first reduced-resolution image frame, wherein the first reduced-resolution image frame comprises a non-correlated double-sampled image; and A second reduced-resolution image frame is captured, wherein the second reduced-resolution image frame comprises a dark frame.

9. The system of claim 8, wherein: The operations further include: Dark image subtraction is performed on the first reduced resolution image frame based on the second reduced resolution image frame.

10. The system of claim 1, wherein: The scanning interval is less than 40 milliseconds.

11. The system of claim 1, wherein: The clock signal is based on the scan timing sequence of the lidar device.

12. The system of claim 11, wherein: The full-resolution image frames are captured as the lidar device scans the field of view.

13. The system of claim 12, wherein: The at least one reduced resolution image frame is captured when the lidar device is not scanning the field of view.

14. A method comprising: causing the light detection and ranging device to scan the field of view based on a scan timing sequence, wherein the scan timing sequence includes a plurality of scan intervals; causing the image sensor to capture a full-resolution image frame during a given scanning interval, wherein the full-resolution image frame comprises a correlated double-sampled image; and During a given scanning interval, the image sensor is caused to capture at least one reduced resolution image frame, wherein the at least one reduced resolution image frame has a lower resolution than the full resolution image frame.

15. The method of claim 14, wherein: Capturing at least one reduced-resolution image frame includes: capturing a first reduced-resolution image frame, wherein the first reduced-resolution image frame comprises a non-correlated double-sampled image; and A second reduced resolution image frame is captured, wherein the second reduced resolution image frame comprises a dark frame, and wherein the method further comprises performing dark image subtraction on the first reduced resolution image frame based on the second reduced resolution image frame.

16. The method of claim 14, wherein: The full-resolution image frame is captured while the light detection and ranging device is scanning a field of view, wherein the at least one reduced-resolution image frame is captured while the light detection and ranging device is not scanning the field of view.

17. The method of claim 14, wherein: The given scanning interval is less than 40 milliseconds.

Images