Image sensor system and control method and computer program

Abstract

A dToF measurement system is described, comprising: a light emitter configured to emit at least one light region at a predetermined position to illuminate a portion of a real-world object; a controller configured to select a real-world position different from the predetermined position for illumination; and an actuator configured to move, under control of the controller, the light region to be emitted to illuminate the selected real-world position.

Description

Background of the Invention Technical Field

[0001] This disclosure relates to image sensor systems, control methods, and computer programs.

[0002] Description of related fields

[0003] The “Background Art” description provided herein is intended to present the general context of this disclosure. The work of the currently attributed inventors, within the scope described in the Background Art section and in aspects that may not be described as prior art at the time of submission, is neither expressly nor impliedly acknowledged as prior art to this invention.

[0004] In image capture, especially when using handheld communication devices, point-based dToF is used to capture distance information. Point-based dToF is a combination of known point projection and array-based dToF systems. In point-based dToF, a specific number of points on a real-world scene are illuminated by a light emitter. These light points are captured by an image sensor, and the time-of-flight of the illuminated points on the object relative to the image sensor is determined.

[0005] However, as will be understood, since the position of the light spot is fixed, a particular relevant point in a real-world scene may not have a light spot positioned on it.

[0006] The purpose of this disclosure is to resolve this problem. Summary of the Invention

[0007] The following provides a brief overview of this disclosure to provide a basic understanding of some aspects of this disclosure.

[0008] The embodiments of this disclosure are defined by the independent claims. Other aspects of this disclosure are defined by the dependent claims.

[0009] This disclosure is not particularly limited to these advantageous technical effects. Other technical effects will become apparent to those skilled in the art upon reading this disclosure. Attached Figure Description

[0010] A more comprehensive understanding of this disclosure and its many accompanying advantages can be readily obtained by referring to the following detailed description in conjunction with the accompanying drawings, in which: Figure 1 An apparatus according to an embodiment of the present disclosure is shown; Figure 2 An image sensor system according to an embodiment of the present disclosure is shown; Figure 3 A pixel array of an image sensor system according to an embodiment of the present disclosure is shown; Figure 4 A first image region and a second image region according to an embodiment of the present disclosure are shown; Figure 5 A first image region and a second image region according to an embodiment of the present disclosure are shown; Figure 6 A first image region and a second image region according to an embodiment of the present disclosure are shown; Figures 7A to 7D An example actuation of a second image sensor according to an embodiment of the present disclosure is shown; Figure 8 An example configuration of a second image sensor according to an embodiment of the present disclosure is shown; Figure 9 An example filter pattern according to an embodiment of the present disclosure is shown; Figure 10 An example of generating an output image according to an embodiment of the present disclosure is shown; Figures 11A to 11E The configuration of an image sensor system according to an embodiment of the present disclosure is shown; Figure 12 Example methods according to embodiments of this disclosure are shown; Figure 13 An example method according to an embodiment of this disclosure is shown; Figure 14 An example method according to an embodiment of this disclosure is shown; Figure 15 This illustrates how distance information of a real-world box is captured according to other implementation methods; Figure 16 It shows when the light spot is irradiated Figure 15 The upper right of the real-world box is the corresponding view of the image sensor 1600; Figure 17A and Figure 17B This illustrates the use of a spot array to capture distance information of a real-world box according to an embodiment; Figure 18 A flowchart according to other embodiments is shown; and Figures 19A to 19D The configuration of an image sensor system according to other embodiments of this disclosure is shown. Detailed Implementation

[0011] Reference Figure 1This illustration shows a device 1000 (an example of an information processing apparatus) according to an embodiment of the present disclosure. Typically, the device 1000 according to an embodiment of the present disclosure is a computer device, such as a personal computer, entertainment system, or terminal connected to a server. In practice, the device may also be a server in this embodiment. A microprocessor or other processing circuitry 1002 is used to control the device 1000. In some examples, the device 1000 may be a portable computing device, such as a mobile phone (e.g., a so-called "smartphone"), a laptop computer, or a tablet computer. More generally, a computer device may include any other type of portable computing device.

[0012] The processing circuitry 1002 may be a microprocessor that executes computer instructions or an application-specific integrated circuit (ASIC). The computer instructions are stored on a storage medium 1004, which may be a magnetically readable medium, an optically readable medium, or a solid-state type circuit. The storage medium 1004 may be integrated into the device 1000 or may be separate from the device 1000 and connected to the device using a wired or wireless connection. The computer instructions may be implemented as computer software containing computer-readable code that, when loaded onto the processor circuitry 1002, configures the processor circuitry 1002 to perform methods according to embodiments of the present disclosure.

[0013] Additionally, an optional user input device 1006 is shown connected to the processing circuitry 1002. The user input device 1006 may be a touchscreen or an input device of the type of mouse or stylus. The user input device 1006 may also be a keyboard, a controller, or any combination of these devices. In some examples, the user input device 1006 may be a microphone or other device. The user can then provide input via sound or voice.

[0014] Network connection 1008 may optionally be coupled to processor circuitry 1002. Network connection 1008 may be a connection to a local area network (LAN) or wide area network (WAN), such as the Internet or a virtual private network (VPN). Network connection 1008 may be connected to a server that allows processor circuitry 1002 to communicate with another device to obtain or provide relevant data. Network connection 1002 may be behind a firewall or some other form of network security.

[0015] Additionally, display device 1010 is shown as being connected to processing circuitry 1002. While shown as being integrated into device 1000, display device 1010 may also be separate from device 1000 and may be a monitor or some other device (e.g., a display screen or head-mounted display) that allows a user to visualize the operation of the system. Alternatively, display device 1010 may be a printer, projector, or some other device that allows a user or third party to view relevant information generated by device 1000.

[0016] As explained in the background section, image sensors can be used to acquire image data. However, the type of image sensor used can significantly affect the image quality of the acquired image. For example, some types of image sensors (such as charge accumulation sensors) may struggle to produce images with the desired image quality level under certain image capture conditions (e.g., low-light conditions). Other types of image sensors (such as single-photon detectors) are highly sensitive, but manufacturing such sensors that meet physical size and spatial constraints can be very difficult. Furthermore, the cost of these image sensors can be very high. Specifically, the high cost of these image sensors is due to the aforementioned constraints on image sensors (which in turn constrain the chip size of the image sensor).

[0017] Therefore, an image sensor system, a control method, and a computer program are provided according to this disclosure.

[0018] Image Sensor Systems

[0019] This is publicly available. Figure 2 An example image sensor system according to an embodiment of the present disclosure is shown.

[0020] This is publicly available. Figure 2 The image sensor system 2000 includes a first image sensor 2002, a second image sensor 2004, an actuator 2006, and a controller 2008.

[0021] The first image sensor 2002 includes a pixel array configured to detect light within at least a first wavelength range in a first image region.

[0022] The second image sensor 2004 includes a pixel array configured to detect light of at least a second wavelength range within a second image region, wherein the number of pixels in the pixel array of the second image sensor is less than the number of pixels in the pixel array of the first image sensor, and / or the second image region is smaller than the first image region, and each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon.

[0023] The actuator 2006 of the image sensor system 2000 is configured to actuate the second image sensor 2004.

[0024] Finally, the controller 2008 is configured to control the actuator 2006 to actuate the second image sensor such that during image capture using the image sensor system 2000, the second image region is positioned within a portion of the first image region.

[0025] In this way, the image sensor system 2000 can change the position of the second image region within the first image region during image capture. This makes it possible to use the second image sensor 2004 to obtain image data that can be optimized or enhanced using image data from the first image sensor 2002. This will be described in more detail later.

[0026] In one example, the controller 2008 may be located in the same device as the first image sensor 2002, the second image sensor 2004, and the actuator 2006. Alternatively, the controller 2008 may be located in a different device than the first image sensor 2002, the second image sensor 2004, and the actuator 2006. For example, the controller 2008 may be located in a separate device and provide instructions for controlling the actuator 2006 using any suitable wired or wireless communication.

[0027] In this example, the controller 2008 may be the processing circuitry of the image sensor system 2000. For instance, the controller 2008 may be a microprocessor that executes computer instructions or may be an application-specific integrated circuit (ASIC).

[0028] In this example, a power source (as a power source) can be provided as part of the image sensor system. In this example, the power source can include a battery. In this example, the power source can be located externally to the image sensor system; the image sensor system can then use power from this external power source. For example, when the image sensor system is included in an information processing device (such as a camera), the image sensor system can use the power source of the information processing system.

[0029] The type of actuator used according to embodiments of this disclosure is not particularly limited. In fact, the type of actuator used may vary at least in part depending on the application of embodiments of this disclosure. In examples, the actuator may include at least one of the following: mechanical actuators, piezoelectric actuators, electro-actuators, shape memory alloy actuators, and / or ultrasonic actuators. In examples, different combinations of actuators may be used to provide different ranges of motion. In examples, different combinations of actuators may be used to provide motion along different axes.

[0030] In this example, the first image sensor may include an accumulation image sensor. In this example, the accumulation image sensor may include a charge-coupled device. In this example, the accumulation sensor may include a complementary metal-oxide-semiconductor detector.

[0031] In this example, the second image sensor is configured to detect the arrival of a single photon. In this example, the second image sensor may include a single-photon avalanche diode (SPAD). SPADs are capable of detecting the arrival of single photons, and this can be utilized to improve the sensitivity of the image sensor.

[0032] Further details about the Image Sensor System 2000 will now be provided.

[0033] <pixel array>

[0034] Now, referring to this disclosure Figure 3 . Figure 3 A pixel array of an image sensor system according to an embodiment of the present disclosure is shown.

[0035] Figure 3 An image sensor system is illustrated. The image sensor system can be, for example, those described with reference to this disclosure. Figure 2 The image sensor system 2000 described herein is an image sensor system. Therefore, the image sensor system 2000 includes a first image sensor 2002 and a second image sensor 2004.

[0036] Figure 3 The image sensor system 2000 also includes, as referenced in this disclosure, Figure 2 The actuator 2006 and controller 2008 are described. However, the actuator 2006 and controller 2008 are not described in this disclosure. Figure 3 As shown in the image.

[0037] The first image sensor 2002 includes a pixel array configured to detect light within at least a first wavelength range in a first image region. This disclosure... Figure 3 The image shows the pixel array 2002A of the first image sensor 2002.

[0038] exist Figure 3 In the example, the pixel array of the first image sensor 2002 is a uniform pixel array. However, this disclosure is not particularly limited in this respect. In the example, the pixel array of the first pixel sensor 2002 may be a non-uniform pixel array.

[0039] Each pixel of the first image sensor 2002 is configured to detect light within at least a first wavelength range. For example, the pixels of the first image sensor 2002 may be configured to detect visible light (e.g., light in the wavelength range of 380 nm to 700 nm). Different filters may be provided above different pixels of the first image sensor 2002, allowing the first image sensor 2002 to capture image data of light at different wavelengths within that wavelength range. For example, the first image sensor may be configured to acquire red, green, and blue image data (which can be used to generate a color image). This will be described in more detail later.

[0040] As previously described, the pixels of the first image sensor 2002 may include light accumulation detectors. In this example, the pixels of the first image sensor are complementary metal-oxide-semiconductor (CMOS) sensors. Therefore, in this example, each pixel includes multiple different transistors. CMOS sensors convert light into electrical charge. Specifically, CMOS sensors convert photons into electrons. Each pixel of the CMOS sensor can be read out individually (the charge corresponds to the light intensity of that pixel).

[0041] The number of pixels in the pixel array 2002A of the first image sensor 2002 is not particularly limited to that disclosed herein. Figure 3 The number shown. Specifically, it will be understood that the number of pixels in the pixel array 2002A of the first image sensor 2002 can be much larger than that disclosed herein. Figure 3 The number shown. For example, the pixel array 2002A of the first image sensor 2002 may include millions of pixels. However, this disclosure is not particularly limited to the existence of any particular number of pixels in the pixel array.

[0042] The pixels of the first image sensor 2002 capture images within a first image region. (Refer to this disclosure.) Figure 4 This will be described in more detail. The frame rate of the first image sensor can be a first frame rate (such as 60 frames per second).

[0043] Figure 3 The disclosure also shows a second image sensor 2004. The second image sensor 2004 includes a pixel array configured to detect light within at least a second wavelength range in a first image region (an image region where pixels of the first image sensor are configured to detect light), wherein the number of pixels in the pixel array of the second image sensor is less than the number of pixels in the pixel array of the first image sensor, and / or the second image region is smaller than the first image region. Figure 3 The pixel array 2004A of the second image sensor 2004 is shown in the figure.

[0044] exist Figure 3In the example, the pixel array of the first image sensor 2004 is a uniform pixel array. However, this disclosure is not particularly limited in this respect. In the example, the pixel array of the second pixel sensor 2004 may be a non-uniform pixel array.

[0045] As explained, each pixel of the second image sensor 2004 is configured to detect light in at least a second wavelength range. In examples, the second wavelength range may be the same as the first wavelength range (i.e., visible light, such as light in the wavelength range of 380 nm to 700 nm). However, in some examples, the wavelength range of light that the pixels of the second image sensor 2004 are configured to detect may be different from the first wavelength range. For example, when used as part of a time-of-flight sensor, the wavelength range of the second image sensor 2004 may be a wavelength range such as 650 nm to 1400 nm. In an example, the wavelength range of the second image sensor 2004 as part of a time-of-flight sensor may be 940 nm. However, this disclosure is not particularly limited to these example wavelength ranges of the second image sensor, and any suitable wavelength range may be used as needed. The pixels of the second image sensor may also be provided with different filters, such that the second image sensor can be configured to acquire red, green, and blue image data (which can be used to generate a color image). This will be described in more detail later.

[0046] As explained earlier, the pixels of the second image sensor 2004 may include single-photon detectors (i.e., pixels capable of detecting the arrival of a single photon). In this example, the pixels in the pixel array 2004A of the second image sensor 2002 are single-photon avalanche diode detectors (SPADs). A SPAD includes a photodiode with a reverse bias voltage higher than the breakdown voltage of a photodiode. A single photon incident on a SPAD can trigger a self-sustaining avalanche. Therefore, a SPAD is highly sensitive and capable of detecting a single photon.

[0047] However, SPAD production can be very expensive. Furthermore, manufacturing SPADs within image sensors with their compact physical dimensions (small form factor) can be extremely difficult. Therefore, as explained in the background section, the use of SPADs in image sensors is very limited.

[0048] However, in this disclosure, the second image sensor has a small number of pixels. That is, the number of pixels in the second image sensor can be less than the number of pixels in the first image sensor, and / or the second image area can be smaller than the first image area. Therefore, the second pixel sensor is a low-resolution array and can be manufactured small and compact. In fact, the second pixel sensor can detect light in only a portion of the first image area at a given time. Alternatively, the second pixel sensor can be able to detect light in the entire first image area, but with lower resolution.

[0049] In this disclosure, the second image sensor with a small number of pixels can be a second image sensor with an absolutely small number of pixels. Alternatively, the second image sensor with a small number of pixels can include a second image sensor with a relatively small number of pixels relative to the number of pixels in the first image sensor. For example, the second image sensor can be 1 megapixel (MP), while the first image sensor can include millions of pixels. However, this disclosure is not particularly limited to any particular number of pixels on the second image sensor, as long as the number of pixels in the second image sensor is less than the number of pixels in the first image sensor (so that the second pixel sensor can be manufactured small and compact).

[0050] Actuator 2006 ( Figure 3 (Not shown) is configured to actuate a second image sensor such that the second image sensor can be positioned to detect light from a second image region within a portion of the first image region during image capture using the image sensor. Therefore, the second image sensor can be used to obtain image data for optimizing or enhancing image data from the first image sensor.

[0051] In the example, the frame rate of the second image sensor 2004 can be significantly higher than that of the first image sensor. For instance, when the pixels of the second image sensor 2004 are SPAD detectors, the SPAD frame rate can exceed 1000 frames per second. This is because SPAD detectors can have very fast readout. Specifically, SPAD-based quantum image sensors (QIS) can generate low-bit images and can operate with very fast readout. In the example, these SPAD detectors can operate at a rate of 1000 frames per second or higher.

[0052] Therefore, in this example, the frame rate of the image data from the second image sensor can be higher than the frame rate of the image data from the first image sensor.

[0053] Therefore, in this example, the actuator can move the second sensor 2004 to acquire fast single-photon image data, which can be used to supplement the image data from the first image sensor. That is, even if the pixel array 2004A of the second pixel sensor has a low resolution, the actuation of the second pixel sensor (combined with the high frame rate of the pixels of the second pixel sensor (compared to the pixels of the first image sensor 2002)) can be used to provide enhanced image data (such as super-resolution imaging).

[0054] This is publicly available. Figure 4 A first image region and a second image region according to an embodiment of the present disclosure are shown.

[0055] This example illustrates images captured by a first sensor 2002 and an image captured by a second sensor 2004. Specifically, the pixel array 2002A of the first image sensor 2002 detects light in the first image region, and the second image sensor 2004 detects light in the second image region. In this example, the second image region is smaller than the first image region. In this example, the image region is the field of view of the image sensor (the extent of the observable world visible to the image sensor at any given time).

[0056] Therefore, the first image sensor 2002 captures image data with a first image region I1, and the second image sensor 2004 captures image data with a second image region I2. In this example, the second image region I2 is a smaller region than the first image region I1; this is because the second image sensor 2004A has a smaller pixel array 2004A (and a lower spatial resolution) than the first image sensor 2002A.

[0057] In this disclosure Figure 4 In the example, the second image region I2 is the region centered on the first image region I1. Therefore, image data from the first sensor is acquired across the entire image region I1; and data from the first sensor 2002 and the second sensor 2004 is acquired within the image region I2.

[0058] However, the position of the second image region I2 within the image region I1 is not limited to... Figure 4 The central region is shown. That is, the image sensor system 2000 includes an actuator 2006 configured to actuate a second image sensor. Therefore, the actuator 2006 (under the control of the controller 2008) is capable of changing the position of the second image sensor 2004. This means that the controller can control the actuator to actuate the second image sensor, and thus position the second image region I2 within a portion of the first image region I1 during image capture within the image sensor system 2000.

[0059] exist Figure 4 In this example, the controller 2008 is configured to control the actuator 2006 to move the second image sensor 2004, and thus can change the position of the second image region I2 within the first image region I1. In this example, the image region I2 can be moved (by actuation of the second image sensor 2004) in both vertical and horizontal directions (to a vertical direction parallel to the plane of the second image sensor 2004).

[0060] However, although this disclosure Figure 4 In the example, the movement of the second image sensor (and therefore the second image region I2) is limited to movement in the horizontal and vertical image directions, but this disclosure is not particularly limited in this respect. In the example, movement can also be made perpendicular to the reference. Figure 4 The description provides movement in the third dimension, encompassing both the horizontal (x) and vertical (y) directions. In practice, in the example, the actuator can be configured to actuate the second image sensor in the z-direction, which is perpendicular to both the horizontal and vertical directions (and also perpendicular to the plane of the image sensor). This provides improved control when moving the image sensor to position the second image region I2 at a location relative to the first image region I1.

[0061] Furthermore, in the example, the actuator can tilt or rotate the second image sensor 2004. Therefore, in the example, the actuator 2006 can provide linear and / or rotational actuation.

[0062] In this way, the controller can move the second image region I2 to a position other than the center position (in this example, the center position could be the default or initial position). This will change the portion of the first image data I1 that is directly related to its capture of the second image data (from the second image sensor 2004) (i.e., the overlapping portion of the first and second image regions). Therefore, the second image region I2 can be changed during image capture by the image sensor system 2000; thus, data from the second image sensor can be used to optimize or enhance the image data from the first image sensor.

[0063] As previously mentioned, in this example, the second image sensor 2004 can have a higher frame rate than the first image sensor. For example, when the second image sensor 2004 is a SPAD, its frame rate (the rate at which data can be read) can be greater than 1000 frames per second. This can be much higher than the frame rate of the first image sensor 2002.

[0064] In this example, the controller can be configured to control the actuator to change the position of the second image sensor multiple times during a single image capture frame from the first image sensor 2002 (and thus change the portion of the first image region I1 that the second image region I2 is subsequently positioned (or aligned) with). For example, if the frame rate of the second image sensor 2004 is 10 times higher than the frame rate of the first image sensor 2002, the position of the second image sensor can be changed approximately 10 times during a single image capture frame from the first image sensor. In this example, this would enable the acquisition of image data from the second image sensor 2004 from 10 different regions across the first image region I1 for a single image data frame from the first image sensor 2002.

[0065] In the example, the controller can be configured to control the actuator to change the position of the second image sensor multiple times corresponding to a single image capture frame from the first image sensor 2002 (and thus change the portion of the first image region I1 that the second image region I2 is subsequently positioned (or aligned) with). That is, the controller can control the actuator such that the second image sensor acquires multiple image capture frames from multiple different positions corresponding to a single image frame captured from the first image sensor 2002. Multiple image capture frames from the second image sensor corresponding to the image capture frame from the first sensor can be captured within a time period during which the first frame of the image capture is performed (due to the higher frame rate of the second image sensor). In the example, multiple image capture frames from the second image sensor can be acquired before or after the time period during which the image capture from the first image sensor is performed. Therefore, this disclosure is not specifically limited to the case where frames from the second image sensor are acquired during the period of performing the image capture from the first image sensor. More generally, image capture frames from the second image sensor corresponding to the image capture from the first image sensor can be captured at a different time than the capture from the first image sensor.

[0066] Therefore, in the example, multiple frames from the second image sensor can correspond to a single frame from the first image sensor.

[0067] Therefore, in the example, the controller is configured to control the actuator to actuate the second image sensor, such that the second image region is located multiple times in association with an image frame of the first image sensor.

[0068] Therefore, in this example, the actuator can move the second sensor 2004 to acquire fast single-photon image data, which can be used to supplement the image data from the first image sensor. That is, even if the pixel array 2004A of the second pixel sensor has a low resolution, the actuation of the second pixel sensor (combined with the high frame rate of the pixels of the second pixel sensor (compared to the pixels of the first image sensor 2002)) can be used to provide enhanced image data (such as super-resolution imaging).

[0069] This disclosure is not specifically limited to, as referenced herein, Figure 4 An example of a first image region I1 and a second image region I2 is described. For example, the relative sizes of the first and second image regions can be very different from this example. In fact, in the example, the first image region I1 can be much larger than the second image region I2.

[0070] Furthermore, in the example, the first image region I1 and the second image region I2 can be the same size (where the second image sensor has fewer pixels than the first image sensor). In this example, the frames of image data from the second image sensor span the same image region as the first image sensor, providing lower resolution image data. However, in the example, the frame rate of the second image sensor is higher than that of the first image sensor. Therefore, the controller is configured to control the actuator to actuate the second image sensor such that the second image region is positioned within a portion of the first image region during image capture using the image sensor system. This means that oversampling can be performed using the second image sensor, thereby improving the effective spatial resolution of the second image sensor (described in more detail with reference to FIG. 7 of this disclosure). Therefore, the second image sensor can be used to obtain high-quality image data, which can be used to optimize or enhance image data from the first image sensor.

[0071] Furthermore, the movement of the second image region I2 is not limited to the type of movement described with reference to this example.

[0072] Now, referring to this disclosure Figure 5 and Figure 6 A further example illustrating the movement of the second image region I2 during image capture (by actuation of the second image sensor 2004). Figures 7A to 7D (A further example of the movement of the second image sensor is also provided (described in more detail later).

[0073] Now, referring to this disclosure Figure 5 .and Figure 4 Similarly, this disclosure Figure 5 A first image region and a second image region according to an embodiment of the present disclosure are shown. Again, as referenced in the present disclosure... Figure 4 As described, the first image I1 is the area covered by the first image sensor 2002, and the second image area I2 is the area covered by the second image sensor 2004.

[0074] In this example, the initial position of the second image region I2 is the upper left corner of the first image region I1. For example, this could be the position of the second image region I2 at the start of image capture. Then, during image capture using the image capture sensor 2000 (i.e., a single image capture frame of the first image sensor 2002), the second image region I2 (by actuation of the actuator 2006 under the control of the controller 2008) moves. In this example, the movement of the second image region I2 follows path P, such that the second image region I2 is positioned to cover all areas of the first image region I1 for a frame of image data from the first image sensor 2002. This means that data from the second image sensor 2004 is available for all areas of the first image region I1. For example, the first image sensor may acquire a single high-resolution frame, while the second image sensor (actuated by the actuator under the control of the controller) acquires multiple frames at a high frame rate (each frame individually having lower spatial resolution and lower bit depth). Thus, in this example, image data from the second image sensor 2004 (which may include multiple data frames) can be used to supplement the image data from the first image sensor for each frame of the image data acquired by the first image sensor 2002 across the entire first image region I1.

[0075] Additionally, data from the second image sensor can be used to supplement image data acquired from the first image sensor based on multiple data frames (including, for example, burst-shot photographic or video data). Furthermore, data from the second image sensor can be used to supplement image data from a single (partial) area or region of image data acquired from the first image sensor.

[0076] Path P can be with Figure 5 The paths shown are different. That is, in the example, when covering the entire first image area (such as...) Figure 5 In the example, any suitable path of the second image sensor 2004 can be used. For example, the second image region I2 can move across the first image region in a vertical column, the second image region I2 can move across the first image region in a horizontal row, the second image region can move across the first image region in a diagonal, the second image region can move across the first image region in a concentric circle, the second image region can move across the first image region in a sweep path, etc.

[0077] Therefore, this disclosure Figure 5An example is provided where the controller is configured to control the actuator according to a predetermined pattern. Furthermore, in the example (as in this disclosure), Figure 5 In the case of a second image region covering the entire first image region within multiple image data frames of the second image sensor, the predetermined mode can be a mode where the second image region covers the entire first image region within multiple image data frames of the second image sensor.

[0078] Although examples have been described that utilize the movement of the second image region to cover the entire first image region within a single image data frame of the first image sensor 2002, Figure 5 Examples are provided, but this disclosure is not particularly limited in this respect. In the examples, directional movement of a second image region can be provided (via actuation of a second image sensor 2004).

[0079] Now, referring to this disclosure Figure 6 .and Figure 4 and Figure 5 similar, Figure 6 A first image region and a second image region according to an embodiment of the present disclosure are shown. Again, as referenced in the present disclosure... Figure 4 and Figure 5 As described, the first image I1 is the area covered by the first image sensor 2002, and the second image area I2 is the area covered by the second image sensor 2004.

[0080] However, with this disclosure Figure 5 In contrast, in this example, the movement of the second image region I2 is a directional movement.

[0081] Directional movement can be performed to align the second image region I2 with a portion of the first image region I1, so that when compensation is performed with image data from the second image sensor 2004, the data from the first image sensor 2002 can be maximized (the image data from the second image sensor 2004 will be the most relevant portion of the first image region I1).

[0082] As explained, in the example, the first image sensor 2002 can be an image sensor such as an accumulation-type image sensor (such as a CMOS sensor). This type of sensor may suffer in terms of image quality under certain image conditions (e.g., low-light image conditions) or for certain types of images (e.g., images with fast-moving objects). Thus, there may be areas of lower image quality from the image data of the first image sensor 2002 (e.g., areas with fast-moving objects); the data from the second image sensor 2004 will be most relevant to these image areas.

[0083] Therefore, in this example, the controller can be configured to use data from one or more previous image data frames from a first image sensor or a second image sensor to control the actuator (and thus the position of the second image region). For example, the controller can be configured to analyze the first image data to identify portions of the first image region that have image quality below a predetermined threshold, and control the actuator to actuate the second image sensor and position the second image region within the identified portions of the first image region.

[0084] In the example, a test frame of image data can be obtained, which is used to identify and determine the target region of the image region that requires the second image data.

[0085] exist Figure 6 In the example, image region I2 is located at an initial position within image region I1. The controller identifies a target region T within the first image region I1, for which acquiring data from the second image sensor 2004 would be particularly advantageous. Therefore, the controller is configured to control actuator 2006 to actuate the second image sensor, causing the second image sensor to move along... Figure 6 The path P shown is moved so that the second image region I2 is positioned at the target region T, thereby aligning with that part of the first image region I1.

[0086] In this way, the use of the second image sensor 2004 can be optimized to acquire data from the second image sensor 2004 only for the areas where it is most needed. This reduces the amount of image data from the second image sensor 2004 (acquiring only for its most relevant areas), which improves the data efficiency of the image sensor system 2000.

[0087] The predetermined threshold can be varied depending on the application of embodiments of this disclosure. For example, different predetermined thresholds can be set depending on the type of image being captured, the image capture mode, and / or the intended use of the captured image. For instance, if an information processing device (such as an image capture device) equipped with an image sensor 2000 is used in "motion mode," it can be determined that reducing blur in the image is most relevant. Therefore, the predetermined threshold for image quality related to the degree of blur in the image can be lowered to ensure that second image data is acquired in a manner that reduces blur.

[0088] In practice, in the examples, image quality may include the brightness level in the first image data, the degree of blurring in the first data, and / or the level of image detail in the first image data (indicating that higher resolution imaging will be required for areas with high image detail). However, this disclosure is not particularly limited to these examples of image quality, and according to embodiments of this disclosure, the controller may use any suitable indicator of image quality to identify the target area for locating the second image area I2.

[0089] In the example, the movement of the second image region I2 (by actuation of the second image sensor 2004) can result in a large change in position. For example, the second image region I2 may be located at a first position (such as the upper left corner of the first image region I1), and then may be moved to a second position at the lower right corner of the second image region I2. Therefore, the positional change of the second image region I2 can be large (i.e., the movement exceeds the pixel size of the second image sensor).

[0090] However, in other examples, the movement of the second image region I2 (by actuation of the second image sensor 2004) can result in a small change in position; in this paper, a small change in position is a change in position corresponding to a movement magnitude smaller than the pixel size of the second image sensor.

[0091] Now, referring to this disclosure Figure 7A This disclosure Figure 7A An example actuation of a second image sensor according to an embodiment of the present disclosure is shown.

[0092] In this example, multiple pixels P1, P2, and P3 of a first image sensor are shown. The pixels of the first image sensor 2002 can be pixels of an image sensor such as a CMOS sensor. Furthermore, pixel P4 of a second image sensor 2004 is shown. This pixel of the second image sensor can be a SPAD detector, etc. The pixel size of the second image sensor 2004 is larger than the pixel size of the first image sensor 2002.

[0093] According to embodiments of this disclosure, the image sensor system 2000 includes an actuator 2006 configured to actuate a second image sensor 2004. Therefore, the second image sensor 2004 can be as described in this disclosure. Figure 7A Move as shown (in this example, move horizontally).

[0094] The initial position of pixel P4 of the first image sensor P4 is determined by this disclosure. Figure 7AThe shaded area shown is illustrated. At this position, pixel P4 is aligned with pixels P1 and P2 of the first image sensor. Then, after actuation, pixel P4 (by actuation of the second image sensor 2004) moves to a second position. The new position of pixel P4 after actuation is determined by the present disclosure. Figure 7A The dashed line in the diagram illustrates this. At this new position, pixel P4 is aligned with pixels P2 and P3 of the first image sensor 2002. Therefore, the movement of the second image sensor 2004 can be less than the pixel size of the second image sensor (i.e., in this example, the movement is less than the size of pixel P4).

[0095] In this way, the movement of the actuator 2006 on the second image sensor 2004 can be performed in a movement quantization smaller than the size of a single pixel of the second image sensor 2004; the precise movement of the second image sensor 2004 in this way enables the efficient use of data from the second image sensor 2004 when generating the output image of the image sensor system 2000 (even when the size of the pixels of the second image sensor 2004 is larger than the size of the pixels of the first image sensor).

[0096] Now, referring to this disclosure Figure 7B . Figure 7B An example actuation of a second image sensor (such as a SPAD) according to an embodiment of the present disclosure is shown.

[0097] Figure 7B The first sub-figure illustrates the detector array 7000 of the second image sensor. The detector array 7000 includes a pixel array 7002, wherein, in this example, each pixel in the array 7000 is a SPAD pixel.

[0098] An example light distribution 7004 falling on the detector array 7000 is shown. This is light from the object being imaged (e.g., focused onto the detector by a lens).

[0099] The pixels in the detector array are SPAD pixels; they are capable of detecting the arrival of individual photons of light. Therefore, at a specific moment (a specific frame of the image), photons may arrive at some pixels in the detector array but not others. That is, pixels in the detector array outside the light distribution 7004 will not detect any photons. Pixels within the light distribution 7004 can detect photon arrivals for a specific frame. However, in different frames, pixels within the light distribution 7004 may not detect any photons (because photons will not be received continuously within the light distribution).

[0100] therefore, Figure 7BThe second sub-figure shows an example output of the image sensor when operating in 1-bit (single-frame) mode. Here, the arrival of photons has been detected in pixel 7006 (within this image frame) within the light distribution area 7004.

[0101] Therefore, it can be seen that the effective resolution of an image sensor's frame may be quite low.

[0102] Figure 7B The third sub-figure illustrates an example output of the image sensor when operating in multi-bit (multi-frame) mode. Here, the output of the SPAD pixels is accumulated and combined across multiple frames. Therefore, even if a pixel does not receive a photon at a particular moment, it will still record a value in the output if the pixel is received at different moments (different frames) within the combined multiple frames. The more frames in which photons are detected, the higher the value output by the pixel for the combined multi-frame image will be. Thus, it can be seen that the effective resolution of the image sensor's frames can be improved by using a multi-frame image mode. However, the resolution is still quite low.

[0103] However, as previously described, embodiments of this disclosure utilize actuation of a second image sensor to provide improved image quality. For example, actuation of the second image sensor can be used to improve the effective resolution of the output.

[0104] Now, referring to this disclosure Figure 7C . Figure 7C An example output of an image sensor according to an embodiment of the present disclosure is shown. Specifically, in this example, a subpixel-level shift in the x and y directions is performed by an actuator between image frames of a second image sensor.

[0105] Figure 7C The top subplot shows the shifting of the pixel array between image frames.

[0106] The pixel array of the second image sensor is located at a first position 7000 at a first moment (e.g., a first image frame). Then, under the control of the controller 2008, the pixel array is actuated by the actuator 2006 to cause a displacement in the x and y directions. After this displacement, the pixel array is located at a second position 7010 at a second moment (e.g., a second image frame). In this example, the displacement in the x and y directions is smaller than the pixel size of the pixel array of the second image sensor (similar to that described with reference to this disclosure). Figure 7A describe).

[0107] Although Figure 7CThe top subfigure shows only two moments, but it will be understood that shifts in the x-direction and y-direction can be performed for more moments (e.g., for more image frames). This disclosure is not particularly limited in this respect. For example, the number of shifts in the pixel array can vary depending on the frame rate of the second image sensor and the nature of the object being imaged. However, it will be understood that the frame rate of the pixels of the second image sensor can be very high (typically, SPAD pixels can have a frame rate exceeding 1000 frames per second). Therefore, the light distribution 7004 can be considered to remain in a constant position between image frames. Thus, since the light distribution 7004 can be considered to remain in a constant position, the relative position of the light distribution 7004 and the pixel array will change due to the actuation of the pixel array. In this way, due to the actuation of the pixel array at different moments, the light distribution will fall on different parts of the pixel array (and therefore on different pixels of the pixel array) at different moments.

[0108] Now, referring to this disclosure Figure 7C The second subplot shows the output of the pixel array at three different times: 7012A, 7012B, and 7012C. For each of these different times, the position of the light distribution on the pixel array is slightly different; this is because the pixel array has shifted in the x and y directions between image frames.

[0109] The output of array 7012A at the first time step shows the pixels where photons have been detected at the first time step. The output of array 7012B at the second time step shows the pixels where photons have been detected at the second time step. The output of array 7012A at the third time step shows the pixels where photons have been detected at the third time step. Therefore, each of 7012A, 7012B, and 7012C shows a possible 1-bit frame sensor output at different XY sampling positions (where the actuator has shifted the x and y positions of the pixel array between different times).

[0110] By sampling the light distribution using the actuation of the pixel array in this manner, the spatial resolution of the second image sensor can be improved. In other words, a high spatial resolution image can be reconstructed by the controller using different image data (i.e., 7012A, 7012B, and 7012C in this example) that have already been acquired (at different sampling locations).

[0111] This is publicly available. Figure 7CThe third sub-figure illustrates a high spatial resolution image reconstructed from the acquired image data. The reconstructed image has a virtual resolution of N×M pixels compared to the actual n×m pixels in the pixel sensor array (where N>>n and M>>m). This is because image oversampling can be performed by shifting the positions of the pixel array and acquiring image data at each different position in the pixel array (image sampling). Those skilled in the art can use any suitable signal processing to perform image reconstruction from the performed spatial oversampling.

[0112] Therefore, the actuation of the second image sensor can be used to improve the spatial resolution of the output image. Thus, the second image sensor can be used to obtain high-quality image data, which can then be used to optimize or enhance the image data from the first image sensor. Furthermore, the second image sensor can be implemented within an image sensor system without increasing the physical size (form-size) or cost of the image sensor system.

[0113] Although the location shift has been described with reference to a pixel size smaller than that of the second image sensor, reference to this disclosure is provided. Figure 7C The actuation described herein is not particularly limited in this respect. In other examples, the magnitude of the displacement by actuation of the sensor may be larger than the pixel size of the second image sensor.

[0114] Now, referring to this disclosure Figure 7D . Figure 7D An example output of an image sensor according to an embodiment of the present disclosure is shown. Specifically, in this example, a subpixel-level shift in the x and y directions is performed by an actuator between image frames of a second image sensor. However, in this example, the magnitude of the pixel shift exceeds the pixel size of the second image sensor.

[0115] In practice, in this example, the actuator is controlled to perform shifts to random or pseudo-random positions. Furthermore, the SPAD sensor mask for the activated pixels is also generated randomly or pseudo-randomly and changes over time. This means that both the position of the pixel array and the pattern of the activated SPAD pixels change over time.

[0116] therefore, Figure 7D The first sub-figure shows the shifts in the x and y positions of the pixel array for two different moments; again, it can be assumed that the light distribution 7004 remains in the same absolute position. Therefore, the change in the pixel array position alters the portion of the pixel array where the light distribution is incident. It is worth noting that, although Figure 7DThe top subplot illustrates x-shifts and y-shifts of a specific size, but it should be understood that this disclosure is not particularly limited in this respect. Rather, the size and direction of the shifts will change as the actuator is controlled to perform shifts to random or pseudo-random positions.

[0117] Figure 7D The second subplot shows the image sensor's output at two different moments. Figure 7D The left side of the second sub-figure shows the output of the array at the first moment. Here, the array is in its initial position such that light distribution 7004 is incident on the first region of the pixel array. Furthermore, multiple pixels of the SPAD array are activated. Activated pixels are shown as for pixel 7016. Therefore, in the pixel array, only one pixel is activated, exists within the region of light distribution 7004, and a photon is detected at this first moment.

[0118] exist Figure 7D The right side of the second sub-figure shows the array's output at the second time step. Here, the array, in different positions, has been shifted to random or pseudo-random positions by the actuator. Furthermore, the masks of the activated pixels have also been randomly or pseudo-randomly generated and changed at the second time step (compared to the first time step). Therefore, at the second time step, the light distribution is incident on the second region of the pixel array (different from the first region of the pixel array). Additionally, two pixels have been activated, existing within the region of light distribution 7004, and photons have been detected at the second time step.

[0119] Pseudo-random sampling of the light distribution 7004 can be performed by actuating the pixel array to random or pseudo-random positions and by changing the random or pseudo-random positions of the activated pixels. This can be performed over multiple image frames; the number of frames can be greater than the reference. Figure 7D The two frames described. Therefore, random (or pseudo-random) oversampling of an image can be performed by shifting the positions of the pixel array and acquiring image data at each different position in the pixel array (sampling the image). Those skilled in the art can use any suitable signal processing to reconstruct the image from the performed spatial oversampling. This provides an output image with a much higher resolution than the pixel array.

[0120] Therefore, random (or pseudo-random) actuation of the second image sensor can be used to improve the spatial resolution of the output image. The second image sensor can be used to acquire high-quality image data, which can be used to optimize or enhance the image data from the first image sensor. Furthermore, this second image sensor can be implemented within an image sensor system without increasing the physical size (form-size) or cost of the image sensor system.

[0121] <Actuation of the second image sensor>

[0122] The movement of the second image region I2 by actuation of the second image sensor 2004 has been described. However, further details of the actuation of the second image sensor 2004 will now be described.

[0123] Image sensor system 2000 includes an actuator 2006 for actuating a second image sensor 2004. In one example, actuator 2006 may be configured to actuate the second image sensor 2004 as a whole (causing the second image sensor 2004 to change its position as a unit). In another example, actuator 2006 may be configured to actuate the second image sensor 2004 by actuating individual parts or components of the second image sensor. This allows different parts or components of the second image sensor to move as needed, thereby providing improved control over the movement of the second image sensor (and the second image region I2).

[0124] Now, referring to this disclosure Figure 8 . Figure 8 An example configuration of a second image sensor according to an embodiment of this disclosure is shown.

[0125] In this example, image sensor 2004 includes pixel array 2004A (such as a SPAD array) and lens 2104. Lens 2104 is an example of a refractive element that can be used in an image sensor such as image sensor 2004. That is, lens 2104 receives light from the scene (possibly after passing through an image filter) and focuses that light onto pixel array 2004A so that pixel array 2004A can acquire image data of the scene. In addition to refractive elements (such as lenses), reflective or diffractive elements can be provided to focus light onto the image sensor. In this example, a combination of optical elements (including refractive, reflective, and diffractive elements) can be provided.

[0126] In the example, actuator 2006 can be configured to actuate image sensor 2004 as a whole (e.g., the actuator can be coupled to the housing of image sensor 2004). In this way, the actuator can cause image sensor 2004 to undergo rotational and / or linear actuation. The relative positions of the components of image sensor 2004 (i.e., the relative positions between pixel array 2004 and lens 2104) can be maintained.

[0127] In this example, actuator 2006 can be configured to actuate one or more individual parts of image sensor 2004. For example, actuator can be configured to actuate pixel array 2004A. In this example, actuator can be configured to actuate lens 2104. In this way, actuator can cause pixel array 2004A and / or lens 2104 to undergo rotational and / or linear actuation. The relative positions of the components of image sensor 2004 (i.e., the relative positions between pixel array 2004 and lens 2104) can therefore be changed. This provides a higher level of control when the position of the second image region within the first image region is changed.

[0128] In the example, actuator 2006 can be configured to adjust the position of one or more elements of the second image sensor 2004 as part of active alignment of the image sensor. For example, displacement of one or more elements of the image sensor in the z-direction of the second image sensor 2004 can be performed. Thus, actuator 2006 can be controlled by controller 2008 to perform active alignment of the image sensor based on one or more external factors (such as temperature) to maintain the configuration of image sensor 2004 in an optimal configuration.

[0129] Although it has been described with reference to the second image sensor 2004 Figure 8 This is an example, but it should be understood that this disclosure is not particularly limited to the internal configuration of the second image sensor 2004 illustrated by this example. For example, although Figure 8 Only a single lens 2104 is shown, but it should be understood that multiple lenses (or optical elements) may be provided. Furthermore, although only a single actuator 2006 is shown, multiple different actuators may be provided. Different actuators may be configured for different types of movement (e.g., linear or rotational), may be configured for actuating different elements within the second image sensor (e.g., pixel array 2004A and lens 2014), and may be of different types (e.g., piezoelectric actuators and ultrasonic actuators or shape memory alloy actuators).

[0130] <Filter>

[0131] A first image sensor 2002 and a second image sensor 2004 of an image sensing system 2000 have been described. Each of the first image sensor 2002 and the second image sensor 2004 includes a pixel array (pixel array 2002A for the first image sensor and pixel array 2004A for the second image sensor). The pixels of the first image sensor are configured to detect light within at least a first wavelength band, and the pixels of the second image sensor are configured to detect light within at least a second wavelength band. In an example, the first and second wavelength bands may be the same wavelength band, wherein each image sensor is configured to detect light within the visible wavelength range (e.g., light within the wavelength range of 380 to 700 nm).

[0132] However, in the example, to provide a color image, a filter can be configured such that a portion of the image sensor's pixels receives light in a first band of wavelength range (e.g., red light), a second portion of the image sensor's pixels receives light in a second band of wavelength range (e.g., green light), and a third portion of the image sensor's pixels receives light in a third band of wavelength range (e.g., blue light). In this way, an image sensor can be used to provide a color image (e.g., composed of separate red, green, and blue image data from the image sensor).

[0133] In the example, the filter can be a filter such as a color filter array. In the example, the color filter array can be a Bayer filter.

[0134] Now, referring to this disclosure Figure 9 . Figure 9 An example filter pattern according to an embodiment of the present disclosure is shown.

[0135] Figure 9 An example filter array could be a filter array used for a first image sensor 2002. Therefore, Figure 9 The pixel array 2002A of the first image sensor 2002 is shown in three different sub-figures. In this example, the first pixel array is a 6x6 pixel array (however, this disclosure is not particularly limited in this respect). Figure 9 The first sub-figure R shows the configuration of the filter corresponding to the portion of the filter that allows light in the red wavelength range to pass through (so that the pixel at that location can be used to generate image data of red light). Figure 9 The second sub-figure G shows the configuration of the filter corresponding to the filter portion that allows light in the green wavelength range to pass through (so that the pixel in that location can be used to generate image data for green light). Figure 9The third sub-figure B shows the configuration of the filter corresponding to the filter portion that allows light in the blue wavelength range to pass through (so that the pixel in that location can be used to generate image data for blue light).

[0136] Utilize, such as Figure 9 The filter shown indicates that the first image sensor 2002 can be used to acquire image data that can generate a color image.

[0137] Although it has been described with reference to the first image sensor 2002 Figure 9 For example, but it will be understood that the second image sensor 2004 may also be configured with filters such as a color array so that the second image sensor 2004 can acquire image data of different colors, which can then be used to supplement or enhance the data acquired by the first image sensor.

[0138] Furthermore, although color filter arrays (such as Bayer filters) have been described with reference to them... Figure 9 Examples are provided, but it will be understood that this disclosure is not particularly limited in this respect. More generally, when imaging is performed on demand using the image sensor system 2000 according to an embodiment of the application of this disclosure, any suitable filter or filter type material may be used.

[0139] <Generation of output image>

[0140] like Figure 2 The described image sensor system 2000 can be configured to generate an output image using image data from a first image sensor 2000 and a second image sensor 2002.

[0141] In other words, in this example, the controller can use image data from the second image sensor 2004 (which has been actuated to align the second image region with the position within the first image region) to enhance the image data from the first image sensor. For example, when the first image sensor is a sensor such as a CMOS sensor, the image quality of the image data acquired from the first image sensor may be lower under certain conditions (such as at low brightness levels). Therefore, data from the second image sensor (which is capable of detecting the arrival of a single photon) can be used to enhance the image data from the first image sensor, thereby providing an output image from the image system 2000 with improved image quality (such as better image quality under low light conditions).

[0142] The method of combining image data from the first image sensor 2000 with image data from the second image sensor 2004 to generate an output image from the image system 2000 is not particularly limited. Any suitable algorithm or process known in the art for combining or fusing image data can be used as needed. In the example, a spatial domain fusion method can be used. In the example, a transform domain fusion method can be used. In the example, a fusion method such as principal component analysis (an example of a spatial domain fusion method) can be used.

[0143] Furthermore, in the example, a trained model can be used to generate the output image.

[0144] Now, referring to this disclosure Figure 10 This disclosure Figure 10 An example of generating an output image according to an embodiment of the present disclosure is shown.

[0145] In this example, a trained model 9000 is provided. The trained model 9000 has been trained on (simulated or historical) image data to generate output images using image data from multiple image sensors. The training of the trained model will be described in more detail later.

[0146] The trained model receives image data 1 from a first image sensor 2002. Furthermore, the trained model receives image data 2 from a second image sensor 2004. The second image sensor 2004 may have a much higher frame rate than the first image sensor 2002. Therefore, the trained model can receive multiple image data frames from the second image sensor 2004 within the time period during which a single image data frame is received from the first image sensor 2002. The trained model is thus configured to generate an output image using the single image data frame from the first image sensor 2002 and the multiple image data frames from the second image sensor 2004. The multiple image data frames from the second image sensor may include frames that together cover an area of ​​the first image sensor (e.g., when the second image area is smaller than the first image area). The multiple image data frames from the second image sensor may also include frames that together provide high-resolution data covering an area of ​​the first image sensor (e.g., when the number of pixels in the pixel array of the second image sensor is less than the number of pixels in the pixel array of the first image sensor and oversampling has been performed by the actuation of the second image sensor).

[0147] The output image is an image generated by a trained model that uses data from the second image sensor 2004 to enhance image data from the first image sensor 2002. Therefore, the output image generated by the trained model is an image with improved image quality compared to an image generated from image data from the first image sensor 2002. That is, compared to an image generated solely from the first data (from the first image sensor), data from the second image sensor 2004 can be used to enhance image brightness or image resolution. Furthermore, compared to an image generated solely from the first data (from the first image sensor), data from the second image sensor 2004 can be used to reduce the blurriness of objects in the image.

[0148] Furthermore, since the second data is generated from the second image sensor actuated by the actuator 2006 under the control of the controller 2008, the second image sensor 2004 can be made smaller. That is, the size of the second image sensor 2004 can be smaller compared to the size of the first image sensor. Therefore, image quality can be improved without increasing the physical size of the image sensor system (or the apparatus using the image sensor system).

[0149] In the examples, the methods and techniques described in this paper can be implemented at least in part using supervised machine learning models (as an example of trained models).

[0150] A supervised learning model is trained using labeled training data to learn a function that maps inputs (typically provided as feature vectors) to outputs (i.e., labels). The labeled training data consists of pairs of inputs and corresponding output labels. The output labels are typically provided by an operator and indicate the expected output for each input. The supervised learning model processes the training data to generate an inference function that can be used to map new (i.e., unseen) inputs to labels.

[0151] Input data (during training and / or inference) can include various types of data, such as numerical values, images, videos, text, or audio. Raw input data can be preprocessed to obtain appropriate feature vectors for use as input to the model; for example, features can be extracted from image or audio inputs to obtain corresponding feature vectors. It should be understood that, (if necessary) the type of input data and the techniques used for data preprocessing can be chosen based on the specific task of using the supervised learning model.

[0152] In the example, the labeled training data may include data from a first image sensor and a second image sensor; the training data may also include examples of images generated from the first image data and the second image data (where the second image data is used to improve the image quality of the first image data).

[0153] Once ready, the labeled training dataset is used to train the supervised learning model. During training, the model tunes its internal parameters (e.g., weights) to optimize (e.g., minimize) the error function, which aims to minimize the difference between the model's predicted output and the labels provided as part of the training data. In some cases, the error function may include regularization penalties to reduce overfitting of the model to the training dataset.

[0154] Supervised learning models can use one or more machine learning algorithms to learn the mapping between their inputs and outputs. Examples of suitable learning algorithms include linear regression, logistic regression, artificial neural networks, decision trees, support vector machines (SVM), random forests, and the K-nearest neighbors algorithm.

[0155] Once trained, a supervised learning model can be used for inference, that is, to predict the output for previously unseen input data. Supervised learning models can perform classification and / or regression tasks. In classification tasks, the supervised learning model predicts discrete class labels for the input data and / or assigns the input data to predetermined classes. In regression tasks, the supervised learning model predicts labels for continuous values.

[0156] Thus, in the example (once training is complete), the trained model can be used to generate output images from previously unseen input data (i.e., previously unseen data from the first and second image sensors).

[0157] In some cases, the amount of labeled data available for model training may be limited (e.g., because labeled data is costly or impractical). In such cases, supervised learning models can be extended to further utilize unlabeled data and / or generate labeled data.

[0158] Consider using unlabeled data; training data can include both labeled and unlabeled training data, and semi-supervised learning can be used to learn the mapping between the model's input and output. For example, graph-based methods such as Laplacian regularization can be used to extend the SVM algorithm to Laplacian SVM to perform semi-supervised learning on partially labeled training data.

[0159] To generate labeled data, an active learning model can be used, in which the model actively queries information sources (such as users or operators) to label data points with the desired output. Typically, only labels from a subset of the training dataset are requested, thus reducing the amount of labels required compared to fully supervised learning. The model can choose the examples of data points to request labels—for instance, it can request labels for data points that will most significantly alter the current model or minimize the model's generalization error. A semi-supervised learning algorithm can then be used to train the model on the partially labeled dataset.

[0160] Furthermore, examples of the present invention may utilize generative artificial intelligence (AI) systems and techniques. Generative AI systems may be used as examples of trained models in this disclosure.

[0161] Generative AI systems learn patterns and structures from their input training data and then generate new output data that exhibits features similar to those in the training data. Each of the input training data and the output data can include various types of data, such as images, videos, text, or audio. For example, a generative AI system can learn patterns from input training images and then generate images with similar characteristics.

[0162] Generative AI systems can generate output data based on input cues. Like training and output data, cues can include various types of data, such as images, videos, text, or audio. The cues can be the same or different data type as the model's training and / or output data. For example, input cues can include text and output data can include images (e.g., input text descriptions matching the desired image), or input cues can include images and output data can include audio data (e.g., audio with a topic matching the input image).

[0163] Generative AI systems can include generative models that are trained to learn the probability distribution of input training data and generate new output data based on this learned distribution. For example, given a set of data instances / observable variables (X) and a set of labels / target variables (Y) in a training dataset, the generative model can learn the joint probability distribution p(X, Y) of the data instances and labels, and / or the probability distribution p(X) of the data instances (e.g., in the absence of labels).

[0164] Suitable example generative models for learning the probability distribution of input training data include variational autoencoders (VAEs), Transformer-based models, diffusion models (e.g., denoising diffusion probabilistic models (DDPM)), reinforcement learning (RL), and generative adversarial networks (GANs). The choice of generative model can be based on the specific task performed by the generative AI system.

[0165] Generative models can include one or more artificial neural networks. For example, a variational autoencoder (VAE) can include a pair of neural networks acting as an encoder and decoder, respectively, traveling to and from a simplified (i.e., latent space) representation of the training data, and a generative adversarial network (GAN) can include a first “generator” neural network that generates new data and a second “discriminator” neural network that learns to distinguish the generated data from real data. The one or more component neural networks of a generative model can be trained together or individually.

[0166] During training, generative models can tune their internal parameters (e.g., neural network weights) to optimize (e.g., minimize) a loss / error function aimed at minimizing the difference between the generated output data and the desired output data. It will be understood that a specific loss function and the algorithm used to optimize it can vary depending on the nature of the generative model and its intended application. For example, the mean squared error loss function can be used for image generation tasks, and the cross-entropy loss function can be used for text generation tasks. These loss functions can be optimized using various existing optimization algorithms, such as gradient descent.

[0167] Once trained, a generative model can be used to generate new output data based on input prompts. These prompts can be provided by the user or by appropriate devices, such as using an application programming interface (API). Therefore, generative AI systems allow the generation of new content (such as images, text, or audio) based solely on prompts, without requiring detailed instructions to do so.

[0168] Therefore, in the example, a trained model can be used to generate an output image from the acquired image data. However, this disclosure is not particularly limited in this respect, and more generally, any suitable processing can be applied to the acquired image data by a controller to generate an output image.

[0169] In the example, the process of generating the output image can be performed by an external device. Alternatively, the process can be performed by the processing circuitry of an information processing device (such as an image capture device) that includes the image sensor system 2000. Or, the process can be performed by a server.

[0170] In the example, processing can be performed on the data from the image sensor before it is processed to generate image data (i.e., the image data can be preprocessed). That is, the image data acquired from the second image sensor can have a very high frame rate. Therefore, the image data generated by the second image sensor 2004 can be very large. Therefore, the controller can perform analysis on the acquired image data before processing this image data (or sending the data to an external device for processing). The analysis (preprocessing) performed by the controller can be used to identify portions of the second image data that have information that can be used to enhance the first image data. For example, analysis can be performed on areas of low image quality from the image data from the first image sensor. The controller can then reduce the second image data by filtering it to retain only the most relevant second image data (second image data that can be used to enhance the output image). This can reduce the power consumption, latency, and storage requirements of the image sensor system 2000.

[0171] In the example, during processing (or preprocessing), the controller can be configured to identify one or more regions with low image quality (e.g., low signal levels) and can be configured to control the actuation of an actuator so that a second image sensor (e.g., a SPAD) acquires additional information for the identified one or more regions. This directional actuation of the second image sensor ensures that image data from the second image sensor is acquired for the most relevant one or more regions, thereby improving image quality. Furthermore, acquiring second image data only for these most relevant image regions (rather than the entire image) also reduces power consumption, latency, and storage requirements.

[0172] In the example, during processing (or preprocessing), the controller can be configured to use the high frame rate of the second image sensor to track objects moving at high speeds in the scene (because these objects can move between frames of the second image sensor). The controller can then be configured to use motion information from the images from the second image sensor to compensate for blurring in the image data from the first image sensor caused by such high-speed moving objects (the first image sensor has a higher resolution but a lower frame rate than the second image sensor). In the example, methods such as those described with reference to this disclosure can be used. Figure 10 The described trained model applies this compensation. However, this disclosure is not particularly limited in this respect. In this way, motion blur in the image data from the first image sensor can be reduced, which further improves the image quality of the output image.

[0173] Therefore, the image sensor 2000 can generate an output image from image data acquired by the first image sensor and the second image sensor.

[0174] Flight Time

[0175] In some cases, it may be desirable to use an image sensor capable of detecting the arrival of a single photon to detect depth within the environment. That is, a sensor capable of detecting the arrival of a single photon (such as a single-photon avalanche diode, SPAD) can be used to accurately determine the photon's arrival time. Therefore, a time-of-flight sensor can be used to determine depth based on the time it takes for light emitted from a light emitter and reflected by objects in the scene to reach the SPAD. Depth is determined based on the time it takes for light to travel from the light emitter to the object and back to the SPAD.

[0176] However, as explained previously in the background section, manufacturing such an image sensor (capable of detecting the arrival time of a single photon) that meets physical size and spatial constraints can be extremely difficult. Furthermore, the cost of these image sensors can be very high. Therefore, the use of single-photon detector image sensors (such as depth sensors) is quite limited.

[0177] Thus, an image sensor system, method, and computer program are provided according to embodiments of this disclosure.

[0178] In other words, according to embodiments of this disclosure, an actuator can be provided within the image sensor system to actuate the image sensor (such as a SPAD) to move the image sensor while acquiring image data at a high frame rate. This enables small, low-resolution image sensors to be used at high-speed actuation and frame rates, thereby increasing the effective pixel count of the image sensor. Therefore, an image system that can generate higher-quality image depth data can be provided without increasing the physical size and cost of the image sensor system. Thus, an image sensor system that can be used in a wide variety of situations can be provided.

[0179] Now, referring to this disclosure Figure 11A and Figure 11B . Figure 11A and Figure 11B The configuration of an image sensor system according to an embodiment of the present disclosure is shown.

[0180] In this example, an image sensor system 1100 (time-of-flight sensor) is shown, which includes an image sensor 1104 and a light emitter 1102.

[0181] Figure 11A An image sensor 1104 in the xy plane is shown.

[0182] The light emitter 1102 of the image sensor system 1100 is configured to emit light within at least a first wavelength range over a first image region. In an example, the light emitter may be a vertical-cavity surface-emitting laser (VCSEL). However, this disclosure is not particularly limited to this specific example of a light emitter. In an example, the light emitter 1102 may be any light source capable of emitting photons that can be used to measure scene depth (based on time-of-flight). In an example, the light emitter may emit light in the wavelength range of 650 nm to 1400 nm. In fact, in an example, the light emitted by the light emitter 1102 may be light outside the visible wavelength range; this allows the light emitter of the image sensor system to be used with visible light imaging systems.

[0183] Image sensor 1104 is such as those already referred to in this disclosure. Figure 2The image sensor described is an image sensor of type 2004. Therefore, the second image sensor includes a pixel array configured to detect light within at least a first wavelength range (i.e., the wavelength of light emitted by the light emitter) in a second image region (a region where the light emitter emits light) smaller than the first image region. Furthermore, each pixel of the pixel array of the image sensor is configured to detect the arrival of a single photon. Therefore, as referenced in this disclosure… Figure 2 As described, the image sensor 1104 can be an image sensor such as a SPAD.

[0184] Actuator 1112 and controller are also configured as part of image sensor 1104. The controller is described in this disclosure. Figures 11A to 11E The example is not shown.

[0185] Actuator 1112 is configured to actuate an image sensor, and controller is configured to control the actuator to actuate a second image sensor, and to position the second image region within a portion of the first image region when depth information is acquired using the image sensor system. Details of the actuator and controller are described with reference to this disclosure. Figure 2 The examples described are the same, and for the sake of brevity in this disclosure, no further details will be provided at this stage.

[0186] Figure 11B An image sensor 1104 in the xz plane is shown.

[0187] In the xz plane, several additional elements of the image sensor system 1104 are shown. For example, a lens 1106 is disposed on the image sensor 1104; this can be used to focus light onto the image sensor 1104. In this example, the lens can be any optical element configured to focus light onto the image sensor. In this example, the lens can be set as part of an external device and not at all part of the image sensor system 1100. Similarly, in this example, a diffractive optical element 1108 is disposed above the light emitter 1102.

[0188] In this example, actuator 1112 can be configured to actuate image sensor 1104 and / or lens 1106. That is, image sensor 1104 and lens 1106 can be actuated to maintain their relative configuration. Alternatively, image sensor 1104 and lens 1106 can be actuated to change their relative configuration.

[0189] exist Figure 11B In this configuration, the image sensor system 1100 is formed on a printed circuit board. Furthermore, in... Figure 11BIn the example, actuator 1112 is positioned below and coupled to PCB 1110. Therefore, the actuator can be used to move the image sensor system 1100 as a whole (i.e., while maintaining the relative positions of the components of the image sensor system 1100). For example, the actuator can be configured to translate the image sensor system in the X and Y directions. Alternatively, the actuator can be configured to rotate the actuator along the X and Y axes. This enables spatial scanning.

[0190] By actuating the image sensor 1104, the image sensor 1104 can be used to detect the arrival time data of photons across the first image region, even if the image region covered by the image sensor (i.e., the second image region) is smaller than the region covered by the light emitter. Therefore, the controller can (using data from the image sensor 1104) generate image depth information with improved quality over the first region.

[0191] Now, for reference Figure 11C This disclosure Figure 11C Another configuration of the image sensor system 1100 according to an embodiment of the present disclosure is shown. In this example, actuator 1112 is individually coupled to image sensor 1104 and light emitter 1102. In this example, multiple independent actuators 1112 may be provided. Therefore, actuator 1112 can move the second image sensor 1104 independently of other elements of the image sensor system 1104. For example, actuator 1112 can move the second image sensor 1104 relative to lens 1106, light emitter 1102, and / or diffractive optical element 1108. Similarly, actuator 1112 can move light emitter 1102 relative to image sensor 1104, diffractive optical element 1108, and / or lens 1106. In this way, spatial scanning can be performed by changing the position of the components of the image sensor system 1100 under the control of a controller. Furthermore, since the relative positions of the components can be changed, more accurate control of spatial scanning can be performed.

[0192] Furthermore, actuation of the light emitter (such as a VCSEL) can be performed to increase the field of view and / or control the target of the light emitter, so that light is provided only in the desired area (or areas). For example, actuation of the light emitter (under the control of a controller) can be performed so that a target area (or zone) is illuminated by light from the light emitter, allowing depth information to be acquired only for that target area or zone. This can further improve efficiency because light can be emitted only for specific areas where depth information is needed.

[0193] Now, for reference Figure 11D . Figure 11D Another configuration of an image sensor system 1100 according to an embodiment of the present disclosure is shown. Figure 11D Examples and Figure 11C The difference in this example is that actuator 1112 is coupled to lens 1106 and diffractive optical element 1108 of image sensor 1104. In this example, actuator 1112 may be coupled to image sensor 1104 and lens 1106. Furthermore, in this example, actuator 1112 may be coupled to either lens 1106 or diffractive optical element 1108. Alternatively, in this example, actuator 1112 may be coupled to both lens 1106 and diffractive optical element 1108.

[0194] In this example, since actuator 1112 is coupled to lens 1106, actuator 1112 can be used to drive an aperture stop (e.g., a nine-bladed aperture stop) to change the aperture of the diffractive optical element and / or lens. Therefore, aperture control can be performed efficiently and easily.

[0195] Now, for reference Figure 11E . Figure 11E Another configuration of an image sensor system 1100 according to an embodiment of the present disclosure is shown. Figure 11E Examples and Figure 11D The difference in this example is that actuator 1112 is coupled between lens 1106 and sensor 1104 and / or between diffractive optical element 1108 and light emitter 1102. In this way, actuator 1112 can be used to change the relative position of lens 1106 and image sensor 1104 in the z-direction. Alternatively or additionally, actuator 1112 can be used to change the relative position of diffractive optical element 1108 and light emitter 1102 in the z-direction. Therefore, actuator 1112 can be controlled by a controller to change the relative z-position of these elements, and thus can be used to perform focusing of image sensor system 1100.

[0196] Furthermore, although it has been described separately Figures 11B to 11E Examples are provided, but it will be understood that these examples can be further combined to provide enhanced control over the image sensor system. For example, multiple different actuators 1112 can be provided, each coupled to an independent element of the image sensor system 1100, thus enabling the execution of multiple different movements as needed.

[0197] According to embodiments of this disclosure, the manner in which the controller uses data from the image sensor to generate depth information is not particularly limited. For example, according to this disclosure, depending on the application of embodiments of this disclosure, any suitable process or algorithm can be used to generate depth information from time-of-arrival information. However, it will be understood that from accurate time-of-arrival information (from the image sensor), the time elapsed from the emission of light by the light emitter to its detection at the image sensor after reflection from an object in the scene can be determined. This information can be used to determine the distance to the object, and thus the depth information of the scene.

[0198] Therefore, in this example, an image sensor system for obtaining depth information is provided, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range within a first image region; an image sensor, the second image sensor comprising a pixel array configured to detect light of at least the first wavelength range within a second image region smaller than the first image region, each pixel in the pixel array of the image sensor being configured to detect the arrival of a single photon; an actuator configured to actuate the image sensor; and a controller configured to control the actuator to actuate the second image sensor, and wherein the second image region is located within a portion of the first image region when depth information is obtained using the image sensor system.

[0199] In this way, an image sensor system capable of generating higher quality image depth data can be provided without increasing the physical size and cost of the image sensor system. Therefore, an image sensor system usable in a wide variety of situations can be provided.

[0200] Furthermore, in this example, the light emitter 1102 can be included in an image sensor such as image sensor 2000. That is, an image sensor with a first image sensor 2002, a second image sensor 2004, and a light emitter 1102 can be provided. However, in this example, the image sensor can be configured with a second image sensor 1104 and a light emitter 1102. Thus, the first image sensor 2002 and the light emitter 1102 can be used alone or in combination with the (actuated) second image sensor.

[0201] <Method>

[0202] Therefore, more generally, embodiments of the present disclosure provide a method for controlling an image sensor system.

[0203] Figure 12 A method according to an embodiment of this disclosure is shown. Figure 12 The method can be achieved by the control circuitry of the image sensor system (such as, as referenced in this disclosure). Figure 2The controller described in 2008 is used to execute this. Alternatively, Figure 12 The method can be implemented by control circuitry external to the image sensor system (such as, as described in this disclosure). Figure 1 The processing circuit 1002 of the described device 1000 is executed.

[0204] Figure 12 The method begins at step S1200 and proceeds to step S1202.

[0205] In step S1202, the method includes controlling an actuator to actuate a second image sensor such that the second image region is positioned within a portion of the first image region during image capture using the image sensor system.

[0206] The method proceeds to step S1204 and ends at this step.

[0207] In this way, the position of the second image region within the first image region can be changed during image capture. This allows the second image sensor to be used to acquire image data that can be used to optimize or enhance the image data from the first image sensor.

[0208] It should be understood that Figure 12 The method described herein is an example of a method for controlling an image sensor system according to embodiments of the present disclosure. Thus, the present disclosure is not particularly limited to the methods described herein. Figure 12 The example method is shown. In embodiments of this disclosure, the method for controlling the image sensor system may further include several additional method steps.

[0209] Figure 13 A method according to an embodiment of this disclosure is shown. Figure 13 The method can be achieved by the control circuitry of the image sensor system (such as, as referenced in this disclosure). Figure 2 The controller described in 2008 is used to execute this. Alternatively, Figure 13 The method can be implemented by control circuitry external to the image sensor system (such as, as described in this disclosure). Figure 1 The processing circuit 1002 of the described device 1000 is used to execute this.

[0210] Figure 13 The method begins at step S1300 and proceeds to step S1302.

[0211] In step S1302, the method includes acquiring first image data. The first image data is image data acquired using the first image sensor 2002 of the image sensor system 2000.

[0212] In step S1304, the method includes acquiring second image data. The second image data is image data acquired using the second image sensor 2004 of the image sensor system 2000.

[0213] In step S1306, the method includes controlling an actuator to actuate a second image sensor such that a second image region is positioned within a portion of a first image region during image capture using the image sensor system. In this example, the control of the actuator can be performed based on the acquired first image data. For example, analysis of the first image data from the first image sensor 2002 may indicate that a portion of the first image data from the first image sensor 2002 has image quality below a threshold (e.g., excessive blurring in a specific area of ​​the image). Therefore, the actuator can be controlled to position the second image region such that second image data (acquired by the second image sensor 2002) can be acquired for that portion of the first image region. Figure 13 The control method shown provides a feedback mechanism through which second image data is acquired for the target portion of the first image region based on the acquired first image data.

[0214] In step S1308, the acquired first image data and second image data are used to generate output image data. In this example, the first image data and second image data can be fused or combined by the controller to generate the output image. In this example, a trained model can be used to generate the output image.

[0215] Once the output image has been generated Figure 13 The example method then proceeds to step S1310 and ends with this step.

[0216] It should be understood that Figure 13 The method described herein is an example of a method for controlling an image sensor system according to embodiments of the present disclosure. Thus, the present disclosure is not particularly limited to the methods described herein. Figure 13 The example method is shown. In embodiments of this disclosure, the method for controlling an image sensor system may further include multiple additional method steps. Furthermore, according to embodiments of this disclosure, the steps in the method can be combined with… Figure 13 The different execution sequences are shown. For example, once the first output image has been generated, the method can return to step S1304 (acquisition of the first image data). Thus, in the example, the frame rate of the output image generated by the image sensor system can be the same as the frame rate of the first image sensor 2002 (where the higher frame rate of the second image sensor 2004 is used to capture image data that can be used to enhance the image data acquired from the first image sensor 2002).

[0217] Figure 14A method according to an embodiment of this disclosure is shown. Figure 14 The method can be performed by the control circuitry of the image sensor system (such as the controller of the image sensor system 1100 as described with reference to FIG. 11 of this disclosure). Alternatively, Figure 14 The method can be achieved by control circuitry external to the image sensor system (such as, as described in this disclosure). Figure 1 The processing circuit 1002 of the described device 1000 is used to execute this.

[0218] Figure 14 The method begins at step S1400 and proceeds to step S1402.

[0219] In step S1402, the method includes controlling an actuator to actuate a second image sensor such that when depth information is obtained using an image sensor system (such as image sensor system 1100 as described with reference to FIG11 of this disclosure), the second image region is located within a portion of the first image region.

[0220] The method proceeds to step S1404 and ends at this step.

[0221] In this way, the position of the second image region within the first image region can be changed while acquiring image depth information. This makes it possible to use a second image sensor to obtain higher quality depth information without increasing the size and cost of the image sensor system.

[0222] It should be understood that Figure 14 The method described herein is an example of a method for controlling an image sensor system according to embodiments of the present disclosure. Thus, the present disclosure is not particularly limited to the methods described herein. Figure 14 The example method is shown. In embodiments of this disclosure, the method for controlling the image sensor system may further include several additional method steps.

[0223] <Other Implementation Methods>

[0224] In the preceding text, the image sensor was described as having a first image region that detects light of one wavelength and a second image region smaller than the first image region that detects single photons. However, in different embodiments, this arrangement is not necessarily the case. Specifically, in other embodiments, the image sensor may only include pixels that detect light emitted by a light emitter. The image sensor may include a SPAD for detecting single photons.

[0225] In a point-based dToF system, an array of dots, formed by light emitted from a light emitter in an image sensor system, is projected onto a real-world scene. Each dot is located at a predetermined (fixed) position and reflects light captured by a corresponding pixel in the image sensor located within the image sensor system. This means that, in practice, only a small number of dots are used to measure distance in any field of view. Because the positions of the dots are fixed, it makes it difficult to select a specific point in the real-world scene, as that point cannot be illuminated by dots in a predetermined pattern. One possibility for solving this problem would be to increase the number of illuminated points in any field of view. However, this would increase the cost and size of the image sensor system in which the light emitter is mounted. The inventors aim to solve these problems.

[0226] In summary, the inventors solved this problem by moving the emitted light spot to illuminate a selected real-world location. This allows the light emitter to move the position of the emitted light spot, and therefore eliminates the need to increase the number of emitted light spots to illuminate the selected real-world location. This movement of the light emitter can be achieved by physically moving the plane of the light emitter using an actuator (e.g., by tilting the light emitter or by translating the light emitter), or by moving the emitted light spot using a lens device, as will be explained later.

[0227] Figure 15 A real-world box 1500 is shown. In this embodiment, the predetermined position of the light spot emitted by the light emitter is 1505'. However, the selected real-world position 1505 is not at the predetermined position of the light spot. In this example, the predetermined position 1505' of the light spot emitted by the light emitter corresponds to a position 4 pixels to the right of the selected real-world position 1505 on the image sensor. As will be explained later, in embodiments of this disclosure, an actuator is provided that moves the emitted light spot away from its predetermined position 1505' to illuminate the selected real-world position 1505.

[0228] As will be understood, although Figure 15 The image shows a single point of light illuminating a specific point on a real-world box, but an array of points of light could actually be set up. As will be explained later, this array could also be used to track the movement of the box over a wider range of distances.

[0229] Figure 16 The diagram shows a corresponding view of the image sensor 1600 when a light spot illuminates the upper right portion of the real-world box 1500. As will be understood, the image sensor 1600 includes a pixel array 1602, which in this embodiment is a SPAD. Figure 16The diagram shows pixel 1505', which will be activated to capture a light spot at a predetermined location. In other words, pixel 1505' within the ellipse will be activated within the image sensor to capture a light spot at the predetermined location.

[0230] However, as referenced Figure 15 To explain, the selected real-world position 1505 is equivalent to four pixels to the left of the predetermined position 1505'. Therefore, when the actuator moves the emitted light spot away from its predetermined position 1505' to illuminate the selected real-world position 1505, which is four pixels away, the activated pixels within the image sensor also shift four pixels to the left. This is because the image sensor system learns of the difference between the selected real-world position 1505 and the predetermined position 1505' of the light spot.

[0231] For reference Figure 7B The explanation is that the image sensor output in single-frame and multi-frame modes is... Figure 16 The two figures below show this.

[0232] As will be understood, although Figure 15 The light spot shown is a circular spot, but the light spot captured by the image sensor 1600 may be of a different shape, such as an ellipse.

[0233] As described above, the real-world box 1500 will actually be illuminated by an array of light spots. This array will be used to allow for a greater distance between the predetermined location of the center point of the array and the selected real-world location. This makes it possible to select a very precise real-world location for an array of a given size.

[0234] refer to Figure 17A This needs to be explained. Figure 17A The image shows a real-world box 1700. Additionally, it shows a first light spot 1705A, a second light spot 1705B, a third light spot 1705C, and a fourth light spot 1705D forming a light spot array. The position of each of the four light spots in the array is known by a controller (not shown), which controls the operation of the image sensor system, as will be explained later. Of course, although four light spots in a rectangular shape are shown, the array is not limited, and any number of light spots arranged in any shape or in any quantity is contemplated.

[0235] Additionally, a selected real-world position 1705 is shown between the first light spot 1705A and the second light spot 1705B. This selected real-world position 1705 is either selected by the user from a preview image via a viewfinder or the like, or selected by an image capture device.

[0236] As will be understood, the real-world position 1705 of the chosen box means that, in the array, the upper right corner is closer to the second light point 1705B than the first light point 1705A. Therefore, the controller does not move the light emitter to illuminate the upper right corner with the first light point 1705A, but rather... Figure 17B As shown, the controller moves the light emitter to illuminate the upper right corner (i.e., the selected real-world location shown in shadow) with the second light spot 1705B in the light spot array. This is in Figure 17B As shown in the figure, the array of light spots is relative to its position in the diagram. Figure 17A The position of the light emitter is moved to the left. In other words, the controller determines which light spot is closest to an object in the real-world scene and moves the light emitter so that the closest light spot is used to illuminate the object of interest. This reduces the amount of movement required in the light emitter. This reduces the amount of movement required in the actuator, making the actuator easier to implement.

[0237] Reference Figure 18 The document describes a flowchart explaining the operation of a controller according to an embodiment. The controller is configured to move a light emitter relative to an image sensor configured to capture the light emitted by the light emitter. This allows the light spot emitted by the light emitter to illuminate a selected real-world location on a real-world object in a point-to-point dToF system without increasing the density of light spots in the array. This reduces the complexity and power consumption of the system.

[0238] Flowchart 1800 begins at step 1805. The process then moves to step 1810, where the controller identifies a point of interest (POI) on the object to be illuminated. This POI can be selected by a user using, for example, a touchscreen on a mobile phone, or by a surgeon during surgery, or automatically by image capture software running on an image capture device (which may be a mobile phone, a single-lens reflex camera, or an endoscope, etc.). In implementations, the POI can be selected based on previous measurements received from the image sensor or other sensors within the image capture device (such as an RGB image sensor), or from the number of photons, or from metadata provided by the user, as will be understood.

[0239] In this implementation, a point of interest (POI) can be selected to correct for misalignment between the position of the illuminated point and the pixel position within the image sensor. For example, optical limitations or misalignment may exist in the light emitter or image sensor. These issues can cause the illuminated point to fall between two pixels regardless of the real-world scene, and moving the emitted light spot will resolve this problem. Furthermore, the illuminated point may fall between two pixels (regardless of any of the aforementioned misalignment). In this case, moving the illuminated point between consecutive frames will allow for better coverage of the selected real-world point. Moreover, and as described below, the selected real-world point of interest can be tracked by moving the real-world illuminated point and activating the corresponding image sensor pixel. Furthermore, the selection of the real-world location can be based on properties of the real-world location. For example, the real-world location could be a corner of an object or be highly reflective, making the detection of the illuminated point easier.

[0240] The process then moves to step 1815, where the controller determines the illuminated point closest to the selected point of interest. This is achieved when the controller learns where the light spots in the array will illuminate in the real-world scene. The controller can then determine which light spot is closest to the selected point of interest.

[0241] The process then moves to step 1820, where the controller moves the light emitter so that the nearest light spot is located at or at least closer to the selected point of interest in the real world. The controller may optionally reduce the size of the emitted array to a local area around the selected point of interest to save power. The mechanism for moving the light emitter will be described in Figure 19 below.

[0242] The process proceeds to step 1825, where the controller instructs the image sensor to activate the pixel in the image sensor corresponding to the new position of the illuminated point. This reduces the power consumption of the image sensor because only the relevant pixel is activated to receive light from the illuminated point. Of course, this disclosure is not so limited, and the lens that focuses light onto the image sensor can be configured to focus light from the new position of the illuminated point onto the same position on the image sensor as the original illuminated point. In other words, the controller can activate the same pixels as those at predetermined positions for light spots in the array (i.e., pixel positions without moving the light spot), and the lens arrangement can guide the moved illuminated point onto those same pixels. In an embodiment, the lens can remain stationary, and the actual image sensor can move such that the new position of the illuminated point is captured by the same pixels as the illuminated point at the original position.

[0243] Then, in step 1830, the process ends.

[0244] It should be noted that the foregoing describes a system in which a user or device selects a real-world location. As will be understood, this arrangement facilitates object tracking. In other words, embodiments of this disclosure can be applied to object tracking scenarios where a specific real-world point of interest (or the object itself) is selected on an object, and that point is tracked across multiple frames using point-based dToF, with the illuminated point following the point of interest. By using the actuator described above, it is not necessary to increase the size of the point-based dToF array or the density of points in the array to follow the point of interest as it moves across various capture frames.

[0245] Reference Figure 19A The image sensor system configuration according to these embodiments of the present disclosure is shown.

[0246] In this example, an image sensor system 1900 (time-of-flight sensor) is shown, which includes an image sensor 1904 (in this embodiment, only a SPAD is included) and a light emitter 1902, which may be a VCSEL. Of course, this disclosure is not so limited, and the image sensor 1904 may include other types of pixel detection, and the light emitter may be any light source capable of emitting photons that can be used to measure scene depth (based on time-of-flight). In this example, the light emitter may emit light in the wavelength range of 650 nm to 1400 nm. In fact, in this example, the light emitted by the light emitter 1902 may be light outside the visible wavelength range; this allows the light emitter of the image sensor system to be used with a visible light imaging system (not shown).

[0247] Figure 19A An image sensor system 1900 in the xy plane is shown.

[0248] Image sensor 1904 is an image sensor including a pixel array configured to detect light emitted by a light emitter. Furthermore, in an embodiment, each pixel in the pixel array of the image sensor is configured to detect the arrival of a single photon. Therefore, as referenced in this disclosure… Figure 2 (As described in relation to different implementations) the image sensor 1904 may be a pixel array configured as a SPAD array or the like.

[0249] Actuator 1912 and the aforementioned controller are configured as part of image sensor system 1900. In this disclosure... Figures 19A to 19D The example does not specifically show the controller.

[0250] Actuator 1912 is configured to actuate a light emitter, and controller is configured to control the actuator to actuate the light emitter when depth information is acquired using an image sensor system. Details of the actuator and controller are provided with reference to this disclosure. Figure 2The examples described are the same, and for the sake of brevity, no further details will be provided at this stage. However, it should be noted that the purpose of actuator 1912 is to move the light emitter relative to image sensor 1904. Specifically, in embodiments of this disclosure, actuator 1912 is arranged to tilt and move the light emitter relative to image sensor 1904.

[0251] Figure 19B An image sensor system 1900 in the xz plane is shown.

[0252] In the xz plane, several additional elements of the image sensor system 1900 are shown. For example, a lens 1906 is disposed above the image sensor 1904; this lens can be used to focus light onto the image sensor 1904. In this example, the lens can be any optical element configured to focus light onto the image sensor. In this example, the lens can be configured as part of an external device, rather than being part of the image sensor system 1900 at all. Similarly, in this example, a diffractive optical element 1908 is disposed above the light emitter 1902. The purpose of the diffractive optical element 1908 is to create an array of light spots on the object illuminated by the light emitter 1902.

[0253] exist Figure 19B In this configuration, the image sensor system 1900 is formed on the printed circuit board 1910. Furthermore, in... Figure 19B In the example, actuator 1912 is positioned below light emitter 1902 and is mounted on and connected to PCB 1910. Therefore, as described above, the actuator can be used to move light emitter 1902 relative to image sensor 1904.

[0254] Now, for reference Figure 19C This disclosure Figure 19C Another configuration of the image sensor system 1900 according to an embodiment of the present disclosure is shown. In this example, the actuator 1912 is separately coupled to the image sensor 1904 and the light emitter 1902. Therefore, the actuator 1912 can be used to move the image sensor 1904 independently of other components of the image sensor system 1904. For example, the actuator 1912 can move the image sensor 1904 relative to the lens 1906, the light emitter 1902, the PCB 1910, and / or the diffractive optical element 1908. Similarly, the actuator 1912 can move the light emitter 1902 relative to the image sensor 1904, the diffractive optical element 1908, and / or the lens 1906. In this way, by referring to... Figure 18 The position of the components of the image sensor system 1900 can be changed under the control of the controller, so that only the pixels on the image sensor required for detecting the light emitted by the light emitter 1912 can be activated.

[0255] Now, for reference Figure 19D . Figure 19D Another configuration of an image sensor system 1900 according to an embodiment of the present disclosure is shown. Figure 19D Examples and Figure 19C The difference in the example is that the actuator 1912 is coupled to the lens 1906 and the diffractive optical element 1908 of the image sensor 1904. In the example, the actuator 1912 can be coupled to both the image sensor 1904 and the lens 1906. Furthermore, in the example, the actuator 1912 can be coupled to either the lens 1906 or the diffractive optical element 1908. Alternatively, in the example, the actuator 1912 can be coupled to both the lens 1906 and the diffractive optical element 1908.

[0256] Since the actuator 1912 is coupled to the lens 1906 in this example, the actuator 1912 can be used to adjust the focal position of the light captured from the light emitter on the image sensor 1904.

[0257] Furthermore, although it has been described separately Figures 19B to 19D Examples are provided, but it will be understood that these examples can be further combined to provide enhanced control over the image sensor system. For example, multiple different actuators 1112 can be provided, each coupled to an independent element of the image sensor system 1100, thus enabling the execution of multiple different movements as needed.

[0258] According to embodiments of this disclosure, the manner in which the controller uses data from the image sensor to generate depth information is not particularly limited. For example, according to this disclosure, depending on the application of embodiments of this disclosure, any suitable process or algorithm can be used to generate depth information from time-of-arrival information. However, it will be understood that from accurate time-of-arrival information (from the image sensor), the time elapsed from the emission of light by the light emitter to its detection at the image sensor after reflection from an object in the scene can be determined. This information can be used to determine the distance to the object, and thus the depth information of the scene.

[0259] While the foregoing describes arranging a light spot at a selected point of interest in a real-world location, this disclosure is not so limited. Specifically, the light spot can be of any shape or size, or in fact, it can be a region. The purpose of the light spot (or region) is to illuminate the selected point of interest so that the illuminated area can be detected by an image sensor, thereby enabling distance measurement.

[0260] Similarly, although the above <Other Embodiments> are specifically described with respect to point-type dToF, this disclosure is not so limited. The principles of the <Other Embodiments> apply to dToF systems. In embodiments with dToF systems, the actuator will move the illuminator (e.g., by tilting or translating the actuator), while the image sensor side may move or not move as needed. This allows for a reduction in the basic illumination field of view of the light emitter and the field of view of the image sensor. This has the advantage of reduced cost because the illuminator size is small (meaning fewer emitters (such as VCSELs)) and the image sensor size is smaller. Furthermore, the illumination / field of view can be expanded and power consumption reduced.

[0261] In both point-based dToF and dTOF applications, actuation of the light emitter (such as a VCSEL) can be performed to increase the field of view and / or control the target of the light emitter so that light is provided only to one or more desired areas. For example, actuation of the light emitter can be performed (under the control of a controller) so that a target area (or region) is illuminated by light from the light emitter, allowing depth information to be acquired only for that target area or region. This can further improve efficiency because light can be emitted only for specific areas where depth information is needed.

[0262] Computer Programs

[0263] Furthermore, it should be understood that the methods of this disclosure can be executed on conventional hardware (such as the hardware previously described herein) that is suitably adapted for application by means of software instructions or by including or replacing dedicated hardware. Therefore, the required adaptation to existing portions of a conventional equivalent device can be implemented in the form of a computer program product comprising processor-executable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory, or any combination of these or other storage media, or implemented in hardware as an ASIC (Application-Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or other configurable circuitry suitable for adapting a conventional equivalent device. In addition, such a computer program can be transmitted via data signals from a network such as Ethernet, wireless networks, the Internet, or any combination of these or other networks.

[0264] <item>

[0265] Alternatively, embodiments of this disclosure can be arranged according to the following numbered items: 1) A dToF measurement system, comprising: A light emitter is configured to emit at least one light region at a predetermined location to illuminate a portion of a real-world object; The controller is configured to select a real-world location different from a predetermined location for illumination; and The actuator is configured to move the area of ​​light to be emitted under the control of the controller to illuminate a selected real-world location.

[0266] 2. The system according to claim 1, wherein the light emitter is configured as an array of light emitting regions, each region being located at a predetermined position, and the controller is configured to select the region in the array whose position is closest to the selected real-world position, and the actuator is configured to move the selected region in the array to the selected real-world position.

[0267] 3. The system according to item 1 or 2, wherein the actuator is configured to move the light emitter to move the area of ​​light to be emitted.

[0268] 4. The system according to claim 1 or 2, wherein the light emitter includes a diffractive optical element, and the actuator is configured to move the diffractive optical element to move the region of light to be emitted.

[0269] 5. The system according to any of the preceding items, wherein the controller is configured to select a real-world location based on user input.

[0270] 6. The system according to any of the preceding items, comprising: An image sensor includes a plurality of pixels configured to receive light regions from a predetermined location and a selected real-world location, wherein the image sensor is configured to read light received only from the selected real-world location.

[0271] 7. The system according to any one of items 1 to 5, comprising: An image sensor includes multiple pixels configured to receive light from a predetermined location; and The second actuator is configured to direct light received from a selected real-world location onto multiple pixels.

[0272] 8. The system according to claim 7, wherein the second actuator is configured to move the image sensor to direct light received from a selected real-world location onto a plurality of pixels.

[0273] 9. The system according to any of the preceding items, wherein the dToF measurement system is a point-type dToF measurement system and the optical region is a light spot.

[0274] 10. A dToF measurement method, comprising: At least one area of ​​light is emitted at a predetermined location to illuminate a portion of a real-world object; Irradiation was performed at a real-world location different from the predetermined location; and The actuator is moved so that the area of ​​light to be emitted illuminates the selected real-world location.

[0275] 11. The method of claim 10, comprising an array of light-emitting regions, each region located at a predetermined position, and selecting a region in the array whose position is closest to the selected real-world position, and a motion actuator causing the selected region in the array to move to the selected real-world position.

[0276] 12. The method according to item 10 or 11 includes moving the light emitter to move the area of ​​light to be emitted.

[0277] 13. The method according to item 10 or 11, wherein the light emitter includes a diffractive optical element, and the method includes moving the diffractive optical element to move the region of light to be emitted.

[0278] 14. The method according to any one of items 10 to 13, comprising selecting a real-world location based on user input.

[0279] 15. The method according to any one of items 10 to 14, comprising: The image sensor reads light received only from selected real-world locations, and the image sensor includes multiple pixels configured to receive light regions from predetermined locations and selected real-world locations.

[0280] 16. The method according to any one of items 10 to 14, comprising: Receives light from a predetermined area from an image sensor comprising multiple pixels; and A second actuator is used to direct light received from a selected real-world location onto multiple pixels.

[0281] 17. The method of claim 16, comprising using a second actuator to move an image sensor to direct light received from a selected real-world location onto the plurality of pixels.

[0282] 18. The method according to any one of items 10 to 17, wherein the dToF measurement system is a point-type dToF measurement system and the light region is a light spot.

[0283] 19. A computer program including computer-readable instructions, which, when loaded onto a computer, configure the computer to perform the method according to any one of claims 10 to 18.

[0284] 20. A computer program product configured to store the computer program according to claim 19.

[0285] Furthermore, it should be understood that numerous modifications and variations of this disclosure are possible in light of the foregoing teachings. Therefore, it should be understood that this disclosure may be practiced in ways other than those specifically described herein, within the scope of the appended claims.

[0286] Since embodiments of this disclosure have been described as being implemented at least in part by a data processing device controlled by software, it should be understood that non-transitory machine-readable media (such as optical discs, magnetic disks, semiconductor memories, etc.) carrying such software are also considered to represent embodiments of this disclosure.

[0287] It should be understood that, for clarity, the above description has referenced different functional units, circuits, and / or processors in describing the implementation. However, it will be apparent that any suitable allocation of functions among the different functional units, circuits, and / or processors can be used without diminishing the implementation.

[0288] The described embodiments can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The described embodiments can optionally be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. Elements and components of any embodiment can be implemented physically, functionally, and logically in any suitable manner. In practice, the function can be implemented in a single unit, in multiple units, or as part of other functional units. Thus, the disclosed embodiments can be implemented in a single unit or can be physically and functionally distributed among different units, circuits, and / or processors.

[0289] Although this disclosure has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. Furthermore, while features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments can be combined in any manner suitable for implementing the present technology.

Claims

1. A dToF measurement system, comprising: A light emitter is configured to emit at least one light region at a predetermined location to illuminate a portion of a real-world object; The controller is configured to select a real-world location different from the predetermined location for illumination; as well as An actuator is configured to move the area of ​​light to be emitted, under the control of the controller, to illuminate the selected real-world location.

2. The system according to claim 1, wherein, The light emitter is configured as an array of light-emitting regions, each region being located at a predetermined position, and the controller is configured to select the region in the array whose position is closest to the selected real-world position, and the actuator is configured to move the selected region in the array to the selected real-world position.

3. The system according to claim 1, wherein, The actuator is configured to move the light emitter to move the area of ​​light to be emitted.

4. The system according to claim 1, wherein, The light emitter includes a diffractive optical element, and the actuator is configured to move the diffractive optical element to move the region of light to be emitted.

5. The system according to claim 1, wherein, The controller is configured to select the real-world location based on user input.

6. The system according to claim 1, comprising: An image sensor includes a plurality of pixels configured to receive light regions from the predetermined location and a selected real-world location, wherein the image sensor is configured to read light received only from the selected real-world location.

7. The system according to claim 1, comprising: An image sensor includes a plurality of pixels configured to receive a light region from the predetermined location; as well as A second actuator is configured to direct light received from the selected real-world location onto the plurality of pixels.

8. The system according to claim 7, wherein, The second actuator is configured to move the image sensor to direct the light received from the selected real-world location onto the plurality of pixels.

9. The system according to claim 1, wherein, The dToF measurement system is a point-type dToF measurement system, and the light region is a light spot.

10. A dToF measurement method, comprising: At least one area of ​​light is emitted at a predetermined location to illuminate a portion of a real-world object; Irradiation was performed at a real-world location different from the predetermined location; as well as The actuator is moved so that the area of ​​light to be emitted illuminates the selected real-world location.

11. The method of claim 10, comprising an array of light-emitting regions, each region located at a predetermined position, and selecting a region in the array whose position is closest to the selected real-world position; and moving the actuator such that the selected region in the array moves to the selected real-world position.

12. The method of claim 10, further comprising moving the light emitter to move the region of light to be emitted.

13. The method according to claim 10, wherein, The light emitter includes a diffractive optical element, and the method includes moving the diffractive optical element to move the region of light to be emitted.

14. The method of claim 10, further comprising selecting the real-world location based on user input.

15. The method of claim 10, comprising: The image sensor reads light received only from the selected real-world location, the image sensor comprising a plurality of pixels configured to receive a region of light from the predetermined location and the selected real-world location.

16. The method of claim 10, comprising: Receive light from the predetermined location from an image sensor comprising multiple pixels; as well as A second actuator is used to direct light received from the selected real-world location onto the plurality of pixels.

17. The method of claim 16, further comprising using a second actuator to move the image sensor to direct light received from the selected real-world location onto the plurality of pixels.

18. The method according to claim 10, wherein, The dToF measurement system is a point-type dToF measurement system, and the light region is a light spot.

19. A computer program including computer-readable instructions that, when loaded onto a computer, configure the computer to perform the method of claim 10.

20. A computer program product configured to store the computer program according to claim 19.

Images