Image sensor system, control method, computer program product, and storage medium

Abstract

The invention relates to an image sensor system, a control method, a computer program product, and a storage medium. The image sensor system includes: a first image sensor including a pixel array configured to detect light of at least a first wavelength range within a first image region; a second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, where a number of pixels of the pixel array of the second image sensor is less than a number of pixels of the pixel array of the first image sensor and / or the second image area is less than the first image area, each pixel in the pixel array of the second image sensor is configured to detect arrival of a single photon; an actuator configured to actuate the second image sensor; and a controller configured to control the actuator to actuate the second image sensor such that the second image region is positioned within a portion of the first image region during image capture with the image sensor system.

Description

Technical Field

[0001] The present disclosure relates to an image sensor system, a control method, and a computer program. Background Art

[0002] The "background technology" description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent described in the background technology section and in aspects of the description that may not be prior art at the time of filing, is neither explicitly nor implicitly admitted to be prior art with respect to the present invention.

[0003] Image data may be acquired using an image sensor. The image data may include still image data (such as a photograph) or moving image data (such as a video). The image sensor may be included as part of an image processing device (such as an image capture device). The image sensor may also be used to obtain image data related to depth information in a scene.

[0004] One type of image sensor includes a charge accumulation sensor. Charge accumulation sensors (including, for example, charge coupled devices and complementary metal oxide semiconductor sensors) are used in a variety of different devices. However, under certain image conditions, these sensors may not provide a desired level of image quality. For example, these types of image sensors may not provide a desired level of image quality in lower light conditions or for fast moving objects and scenes.

[0005] Another type of image sensor includes a single photon detector (such as a single photon avalanche diode) that can detect the presence of a single photon. However, while single photon detectors are very sensitive, it can be very difficult to produce such image sensors that meet the constraints of physical size and space. In addition, the cost of these image sensors can be very high. Therefore, the use of single photon detector image sensors is quite limited and restricted.

[0006] Therefore, it may be difficult to provide an image sensor that can acquire high-quality image data under certain image capture conditions.

[0007] The present disclosure aims to address these problems. Summary of the invention

[0008] The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure.

[0009] Embodiments of the present disclosure are defined by the independent claims. Other aspects of the present disclosure are defined by the dependent claims.

[0010] With embodiments of the present disclosure, an actuator is used to change the position of a second image sensor of an image sensor system during image capture and / or when obtaining image depth information. Thus, a second image sensor can be used to obtain high-quality image data, which can be used to optimize or enhance image data from a first image sensor and / or obtain more accurate depth information of a scene. Furthermore, the second image sensor can be implemented in an image sensor system without increasing the physical size (form factor) or cost of the image sensor system. Alternatively or additionally, an actuator can be used to change the position of an image sensor and / or a light emitter. Thus, higher quality image depth data can also be provided without increasing the physical size and cost of the image sensor system.

[0011] The present disclosure is not particularly limited to these advantageous technical effects. Other technical effects will become apparent to those skilled in the art when reading the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A more complete appreciation of the present disclosure and many of its attendant advantages will be readily obtained as they become better appreciated by reference to the following detailed description when considered in conjunction with the accompanying drawings, wherein:

[0013] Figure 1 An apparatus according to an embodiment of the present disclosure is shown;

[0014] Figure 2 An image sensor system according to an embodiment of the present disclosure is shown;

[0015] Figure 3 shows a pixel array of an image sensor system according to an embodiment of the present disclosure;

[0016] Figure 4 shows a first image region and a second image region according to an embodiment of the present disclosure;

[0017] Figure 5 shows a first image region and a second image region according to an embodiment of the present disclosure;

[0018] Figure 6 shows a first image region and a second image region according to an embodiment of the present disclosure;

[0019] 7A to 7D illustrates an example actuation of a second image sensor according to an embodiment of the present disclosure;

[0020] Figure 8 shows an example configuration of a second image sensor according to an embodiment of the present disclosure;

[0021] Fig. 9shows an example filter pattern according to an embodiment of the present disclosure;

[0022] Fig.10 An example of generating an output image according to an embodiment of the present disclosure is shown;

[0023] FIG. 11A to FIG. 11E shows a configuration of an image sensor system according to an embodiment of the present disclosure;

[0024] Fig.12 An example method according to an embodiment of the present disclosure is shown;

[0025] Fig.13 An example method according to an embodiment of the present disclosure is shown; and

[0026] Fig.14 An example method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] refer to Figure 1 , shows a device 1000 (an example of an information processing device) according to an embodiment of the present disclosure. Typically, the device 1000 according to an embodiment of the present disclosure is a computer device, such as a personal computer, an entertainment system, a terminal connected to a server. In fact, in an embodiment, the device may also be a server. A microprocessor or other processing circuit 1002 is used to control the device 1000. In some examples, the device 1000 may be a portable computing device, such as a mobile phone (e.g., a so-called "smart phone"), a laptop computer, or a tablet computing device. More generally, the computer device may include any other type or portable computing device.

[0028] Processing circuit 1002 may be a microprocessor that executes computer instructions or may be an application specific integrated circuit. Computer instructions are stored on storage medium 1004, which may be a magnetically readable medium, an optically readable medium, or a solid-state type circuit system. Storage medium 1004 may be integrated into device 1000, or may be separate from device 1000 and connected to the device using a wired or wireless connection. Computer instructions may be embodied as computer software containing computer readable code that, when loaded onto processor circuit system 1002, configures processor circuit system 1002 to perform a method according to an embodiment of the present disclosure.

[0029] In addition, an optional user input device 1006 is shown connected to the processing circuit system 1002. The user input device 1006 can be a touch screen or can be a mouse or designer type input device. The user input device 1006 can also be a keyboard, a controller, or any combination of these devices. In some examples, the user input device 1006 can be a microphone or other device. The user can then provide input via sound or voice.

[0030] A network connection 1008 may optionally be coupled to the processor circuit system 1002. The network connection 1008 may be a connection to a local area network or a wide area network, such as the Internet or a virtual private network, etc. The network connection 1008 may be connected to a server that allows the processor circuit system 1002 to communicate with another device in order to obtain or provide relevant data. The network connection 1008 may be behind a firewall or some other form of network security.

[0031] In addition, display device 1010 is shown coupled to processing circuit system 1002. Display device 1010, although shown as integrated into device 1000, may otherwise be separate from device 1000 and may be a monitor or some device (e.g., a display screen or head mounted display) that allows a user to visualize the operation of the system. In addition, display device 1010 may be a printer, a projector, or some other device that allows a user or a third party to view relevant information generated by device 1000.

[0032] As explained in the background, image sensors can be used to collect image data. However, the type of image sensor used can have a significant impact on the image quality of the images obtained. For example, certain types of image sensors (such as charge accumulation sensors) may have difficulty producing images with a desired level of image quality under certain image capture conditions (e.g., lower light conditions). Other types of image sensors, such as single photon detectors, are very sensitive, but it can be very difficult to produce such image sensors that meet physical size and space constraints. In addition, the cost of these image sensors can be very high. In particular, due to the above-mentioned constraints on image sensors (which in turn constrain the die size of the image sensors), the cost of these image sensors can be very high.

[0033] Therefore, according to the present disclosure, an image sensor system, a control method, and a computer program are provided.

[0034] <Image Sensor System>

[0035] The present disclosure Figure 2 An example image sensor system is shown in accordance with an embodiment of the present disclosure.

[0036] The present disclosure Figure 2The image sensor system 2000 includes a first image sensor 2002 , a second image sensor 2004 , an actuator 2006 , and a controller 2008 .

[0037] The first image sensor 2002 includes a pixel array configured to detect light of at least a first wavelength range within a first image area.

[0038] The second image sensor 2004 includes a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels in the pixel array of the second image sensor is smaller than the number of pixels in the pixel array of the first image sensor and / or the second image area is smaller than the first image area, and each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon.

[0039] Actuator 2006 of image sensor system 2000 is configured to actuate second image sensor 2004 .

[0040] Finally, controller 2008 is configured to control actuator 2006 to actuate the second image sensor such that the second image area is positioned within a portion of the first image area during image capture using image sensor system 2000 .

[0041] In this way, the image sensor system 2000 can change the position of the second image area within the first image area during image capture. This enables the use of the second image sensor 2004 to obtain image data that can be used to optimize or enhance the image data from the first image sensor 2002. This will be described in more detail later.

[0042] In an example, the controller 2008 may be provided in the same device as the first image sensor 2002, the second image sensor 2004, and the actuator 2006. In an example, the controller 2008 may be provided in a different device than the first image sensor 2002, the second image sensor 2004, and the actuator 2006. For example, the controller 2008 may be provided in a separate device and provide instructions to perform control of the actuator 2006 using any suitable wired or wireless communication.

[0043] In an example, controller 2008 may be processing circuitry of image sensor system 2000. For example, controller 2008 may be a microprocessor that executes computer instructions or may be an application specific integrated circuit.

[0044] In an example, a power source (as a source of power) may be provided as part of the image sensor system. In an example, the power source may include a battery. In an example, the power source may be provided external to the image sensor system; the image sensor system may then use power from the power source external to the image sensor system. For example, when the image sensor system is included in an information processing device (such as a camera), the image sensor system may use the power source of the information processing system.

[0045] The type of actuator used according to the embodiments of the present disclosure is not particularly limited. In fact, the type of actuator used may depend at least in part on the situation to which the embodiments of the present disclosure are applied. In an example, the actuator may include: at least one item in the list including a mechanical actuator, a piezoelectric actuator, an electric actuator, a shape memory alloy actuator and / or an ultrasonic actuator. In an example, a combination of different actuators may be used to provide different ranges of motion. In an example, a combination of different actuators may be used to provide motion along different axes.

[0046] In an example, the first image sensor may include an accumulation image sensor. In an example, the accumulation image sensor may include a charge coupled device. In an example, the accumulation sensor may include a complementary metal oxide semiconductor detector.

[0047] In an example, the second image sensor is an image sensor configured to detect the arrival of a single photon. In an example, the second image sensor may include a single photon avalanche diode (SPAD). SPAD is capable of detecting the arrival of a single photon, and thus the arrival of the single photon may be used to increase the sensitivity of the image sensor.

[0048] Further details of image sensor system 2000 will now be provided.

[0049] <Pixel Array>

[0050] Now consider the present disclosure Figure 3 . Figure 3 A pixel array of an image sensor system according to an embodiment of the present disclosure is shown.

[0051] Figure 3 The image sensor system is shown in FIG. The image sensor system may be such as that disclosed in the present disclosure. Figure 2 The described image sensor system 2000 includes a first image sensor 2002 and a second image sensor 2004 .

[0052] Figure 3 The image sensor system 2000 also includes the Figure 2The actuator 2006 and controller 2008 described herein are not disclosed herein. Figure 3 Shown in.

[0053] The first image sensor 2002 includes a pixel array configured to detect light of at least a first wavelength range within a first image area. Figure 3 Pixel array 2002A of the first image sensor 2002 is shown in FIG.

[0054] exist Figure 3 In the example, the pixel array of the first image sensor 2002 is a uniform pixel array. However, the present disclosure is not particularly limited in this regard. In the example, the pixel array of the first pixel sensor 2002 may be a non-uniform pixel array.

[0055] Each pixel of the first image sensor 2002 is configured to detect light of at least a first wavelength range. For example, the pixels of the first image sensor 2002 may be configured to detect visible light (e.g., light in a wavelength range of 380 to 700 nm). Different filters may be set on different pixels of the first image sensor 2000 so that the first image sensor 2002 may capture image data about light of different wavelengths in the wavelength range. For example, the first image sensor may be configured to obtain red, green, and blue image data (which may be used to generate a color image). This will be described in more detail later.

[0056] As previously explained, the pixels of the first image sensor 2002 may include a light accumulation detector. In this example, the pixels of the first image sensor are complementary metal oxide semiconductor (CMOS) sensors. Therefore, in this example, each pixel includes a plurality of different transistors. The CMOS sensor converts light into charge. In particular, the CMOS sensor converts photons into electrons. Each pixel of the CMOS sensor can be read out individually (the charge corresponds to the light intensity of the pixel).

[0057] The number of pixels in the pixel array 2002A of the first image sensor 2002 is not particularly limited to the number of pixels in the present disclosure. Figure 3 In particular, it should be understood that the number of pixels in the first image sensor 2002 pixel array 2002A may be significantly greater than the number of pixels in the present disclosure. Figure 3 For example, the first image sensor 2002 pixel array 2002A may include millions of pixels. However, the present disclosure is not particularly limited to any particular number of pixels in the pixel array.

[0058] The pixels of the first image sensor 2002 capture an image within the first image area. This will be referred to in the present disclosure. Figure 4Described in more detail. The frame rate of the first image sensor may be a first frame rate (such as 60 frames per second).

[0059] The second image sensor 2004 is also Figure 3 The second image sensor 2004 includes a pixel array configured to detect light of at least a second wavelength range within a first image area (an image area in which pixels of the first image sensor are configured to detect light), wherein the number of pixels in the pixel array of the second image sensor is smaller than the number of pixels in the pixel array of the first image sensor, and / or the second image area is smaller than the first image area. Figure 3 Pixel array 2004A of second image sensor 2004 is shown in FIG.

[0060] exist Figure 3 In the example, the pixel array of the first image sensor 2004 is a uniform pixel array. However, the present disclosure is not particularly limited in this regard. In the example, the pixel array of the second pixel sensor 2004 may be a non-uniform pixel array.

[0061] As explained, each pixel of the second image sensor 2004 is configured to detect light of at least a second wavelength range. In an example, the second wavelength range may be the same as the first wavelength range (i.e., visible light, such as light in a wavelength range of 380nm to 700nm, for example). However, in some examples, the wavelength range in which the pixels of the second image sensor 2004 are configured to detect light may be different from the first wavelength range. For example, when used as part of a time-of-flight sensor, the wavelength range of the second image sensor 2004 may be a wavelength range such as a range of 650nm to 1400nm. In an example, the wavelength range of the second image sensor 2004 as part of a time-of-flight sensor may be 940nm. However, the present disclosure is not particularly limited to these example wavelength ranges of the second image sensor, and any suitable wavelength range may be used as desired. The pixels of the second image sensor may also be provided with different filters so that the second image sensor may be configured to obtain red, green, and blue image data (which may be used to generate a color image). This will be described in more detail later.

[0062] As previously described, the pixels of the second image sensor 2004 may include single photon detectors (i.e., pixels that can detect the arrival of a single photon). In this example, the pixels of the second image sensor 2002 pixel array 2004A are single photon avalanche diode detectors (SPADs). SPADs include a photodiode having a reverse bias voltage that is higher than the breakdown voltage of the photodiode. A single photon incident on the SPAD can trigger a self-sustaining avalanche. Therefore, SPADs are very sensitive and have the ability to detect single photons.

[0063] However, SPADs can be very expensive to produce. Furthermore, SPADs can be very difficult to produce within image sensors of compact physical size (small form factor). Therefore, as explained in the background, the use of SPADs in image sensors is very limited.

[0064] However, in the present disclosure, the second image sensor is provided with a lower number of pixels. That is, the number of pixels in the second image sensor may be smaller than the number of pixels in the first image sensor and / or the second image area may be smaller than the first image area. Thus, the second pixel sensor is a low resolution array and may be made small and compact. In practice, the second pixel sensor may be able to detect light in only a portion of the first image area at a given instance in time. Alternatively, the second pixel sensor may be able to detect light over the entire first image area, but at a lower resolution.

[0065] In the present disclosure, the provision of a second image sensor having a lower number of pixels may include a second image sensor having a lower absolute number of pixels. Alternatively, the provision of a second image sensor having a lower number of pixels may include a second image sensor having a lower relative number of pixels (compared to the number of pixels of the first image sensor). For example, the second image sensor may be 1 megapixel (MP) while the first image sensor may include millions of pixels. However, the present disclosure is not particularly limited to any particular number of pixels on the second image sensor, as long as the number of pixels in the second image sensor is less than the number of pixels in the first image sensor (enabling the second image sensor to be made small and compact).

[0066] Actuator 2006 ( Figure 3 The second image sensor may be configured to actuate the second image sensor so that the second image sensor may be positioned to detect light from the second image area within a portion of the first image area during image capture with the image sensor. Thus, the second image sensor may be used to obtain image data that may be used to optimize or enhance image data from the first image sensor.

[0067] In an example, the frame rate of the second image sensor 2004 can be much higher than the frame rate of the first image sensor. For example, when the pixels of the second image sensor 2004 are SPAD detectors, the frame rate of the SPAD can exceed 1000 frames per second. This is because the SPAD detector can be read out very quickly. Specifically, a SPAD-based quantum image sensor (QIS) can generate images with a low bit count and can operate with very fast readout. In an example, these SPAD detectors can operate at a rate of 1kfps or higher.

[0068] Thus, in an example, the frame rate of the image data from the second image sensor may be higher than the frame rate of the image data from the first image sensor.

[0069] Thus, in an example, the actuator can move the second sensor 2004 to acquire rapid single photon image data that can be used to supplement the image data from the first image sensor. That is, even if the resolution of the pixel array 2004A of the second pixel sensor is low, actuation of the second pixel sensor (combined with the high frame rate of the pixels of the second pixel sensor (compared to the pixels of the first image sensor 2002)) can be used to provide enhanced image data (such as super-resolution imaging).

[0070] The present disclosure Figure 4 A first image region and a second image region are shown according to an embodiment of the present disclosure.

[0071] In this example, an image captured by a first sensor 2002 and an image captured by a second sensor 2004 are shown. That is, pixel array 2002A of the first image sensor 2002 detects light in a first image area, and the second image sensor 2004 detects light in a second image area. In this example, the second image area is smaller than the first image area. In the example, the image area is the field of view of the image sensor (this is the extent of the observable world visible to the image sensor at any instance in time).

[0072] Thus, the first image sensor 2002 captures image data having a first image area I1, and the second image sensor 2004 captures image data having a second image area I2. In this example, the second image area I2 is a smaller area than the first image area I1; this is because the second image sensor 2004A has a smaller pixel array 2004A (and has a lower spatial resolution) than the first image sensor 2002A.

[0073] In this disclosure Figure 4 In the example of , the second image region I2 is the region at the center of the first image region I1. Therefore, image data from the first sensor is acquired from the entire image region I1; and data from the first sensor 2002 and the second sensor 2004 are acquired within the image region I2.

[0074] However, the position of the second image region I2 within the image region I1 is not limited to Figure 4That is, the image sensor system 2000 includes an actuator 2006 configured to actuate the second image sensor. Thus, the actuator 2006 (under the control of the controller 2008) is capable of changing the position of the second image sensor 2004. This means that the controller can control the actuator to actuate the second image sensor and thus position the second image area I2 within a portion of the first image area I1 during image capture within the image sensor system 2000.

[0075] exist Figure 4 In the example of FIG. 2 , the controller 2008 is configured to control the actuator 2006 to move the second image sensor 2004 and thus can change the position of the second image area I2 within the first image area I1. In this example, the image area I2 can be moved in the vertical and horizontal directions (by actuation of the second image sensor 2004) (directions orthogonal to the plane parallel to the second image sensor 2004).

[0076] However, although the present disclosure Figure 4 The movement of the second image sensor (and therefore the second image area I2) in the example of is limited to movement in the horizontal and vertical image directions, but the present disclosure is not particularly limited in this regard. In the example, it is also possible to move the second image sensor (and therefore the second image area I2) in a direction orthogonal to the reference Figure 4 The movement is set in the third dimension of both the horizontal (x) and vertical (y) directions described above. In fact, in the example, the actuator can be configured to actuate the second image sensor in the z direction (also a direction orthogonal to the plane of the image sensor) which is orthogonal to the horizontal and vertical directions. This provides improved control when moving the image sensor to position the second image area I2 at a specific location of the first image area I1.

[0077] Additionally, in an example, the actuator may tilt or rotate the second image sensor 2004. Thus, in an example, linear and / or rotational actuation may be provided by the actuator 2006.

[0078] In this way, the controller can move the second image area I2 to a position other than the center position (which can be a default or initial position in an example). This will change the portion of the first image data I1 for which the second image data (from the second image sensor 2004) is being captured (i.e., the overlapping portion of the first image area and the second image area). Thus, the second image area I2 can be changed during image capture by the image sensor system 2000; thus, data from the second image sensor can be used to optimize or enhance the image data from the first image sensor.

[0079] As previously mentioned, in an example, the second image sensor 2004 can have a higher frame rate than the first image sensor. For example, when the second image sensor 2004 is a SPAD, the frame rate (the rate at which data can be read out) of the second image sensor can be greater than 1000 frames per second. This can be much higher than the frame rate of the first image sensor 2002.

[0080] In an example, the controller may be configured to control the actuator to change the position of the second image sensor (and thus change the portion of the first image region I1 at which the second image region I2 is positioned (or aligned)) multiple times during a single frame of image capture from the first image sensor 2002. For example, if the frame rate of the second image sensor 2004 is 10 times higher than the frame rate of the first image sensor 2002, the position of the second image sensor may be changed approximately 10 times during a single frame of image capture by the first image sensor. In this example, this would enable image data from the second image sensor 2004 to be obtained from 10 different regions across the first image region I1 for a single frame of image data from the first image sensor 2002.

[0081] In an example, the controller may be configured to control the actuator to change the position of the second image sensor (and thus change the portion of the first image area I1 at which the second image area I2 is positioned (or aligned)) multiple times corresponding to a single frame of image capture from the first image sensor 2002. That is, the controller may control the actuator so that the second image sensor acquires multiple frames of image capture from multiple different positions corresponding to a single image frame of image capture from the first image sensor 2002. Multiple frames of image capture of the second image sensor corresponding to the frame of image capture from the first sensor may be captured within a time period of performing the first frame of image capture (due to the higher frame rate of the second image sensor). In an example, multiple frames of image capture of the second image sensor may be obtained in a time period before or after the time period of performing the image capture from the first image sensor. Therefore, the present disclosure is not specifically limited to the case where a frame from the second image sensor is acquired during the time of performing the image capture from the first image sensor. More generally, a frame of image capture from the second image sensor corresponding to the image capture from the first image sensor may be captured at a different time from the capture from the first image sensor.

[0082] Thus, in an example, multiple frames from the second image sensor may correspond to a single frame of the first image sensor.

[0083] Therefore, in an example, the controller is configured to control the actuator to actuate the second image sensor such that the second image area is positioned a plurality of times in association with one image frame of the first image sensor.

[0084] Thus, in an example, the actuator can move the second sensor 2004 to acquire rapid single photon image data that can be used to supplement the image data from the first image sensor. That is, even if the resolution of the pixel array 2004A of the second pixel sensor is low, actuation of the second pixel sensor (combined with the high frame rate of the pixels of the second pixel sensor (compared to the pixels of the first image sensor 2002)) can be used to provide enhanced image data (such as super-resolution imaging).

[0085] The present disclosure is not particularly limited to Figure 4 An example of a first image region I1 and a second image region I2 is described. For example, the relative sizes of the first image region and the second image region may differ greatly from this example. In fact, in the example, the first image region I1 may be much larger than the second image region I2.

[0086] In addition, in the example, the first image area I1 and the second image area I2 can be the same size (wherein the second image sensor has a smaller number of pixels than the first image sensor). In this example, the frame of the image data of the second image sensor provides lower resolution image data across the same image area as the first image sensor. However, in the example, the frame rate of the second image sensor is higher than the frame rate of the first image sensor. Therefore, the controller is configured to control the actuator to actuate the second image sensor so that the second image area is positioned within a portion of the first image area during image capture using the image sensor system. This means that oversampling can be performed using the second image sensor, thereby increasing the effective spatial resolution of the second image sensor (described in more detail with reference to FIG. 7 of the present disclosure). Therefore, the second image sensor can be used to obtain high-quality image data, which can be used to optimize or enhance the image data from the first image sensor.

[0087] Furthermore, the movement of the second image area I2 is not limited to the type of movement described with reference to this example.

[0088] Reference will now be made to the Figure 5 and Figure 6 A further example of movement of the second image area 12 (by actuation of the second image sensor 2004) during image capture is described. 7A to 7D Further examples of movement of the second image sensor are also provided (described in more detail later).

[0089] Now consider the present disclosure Figure 5 .and Figure 4 Similarly, the present disclosure Figure 51 and 2 show a first image region and a second image region according to an embodiment of the present disclosure. Figure 4 As depicted, the first image area I1 is an area covered by the first image sensor 2002 , and the second image area I2 is an area covered by the second image sensor 2004 .

[0090] In this example, the initial position of the second image area I2 is the upper left corner of the first image area I1. For example, this can be the position of the second image area I2 at the beginning of image capture. Then, during image capture using the image capture sensor 2000 (i.e., a single frame of image capture of the first image sensor 2002), the second image area I2 is moved (by actuation by the actuator 2006 under the control of the controller 2008). In this example, the movement of the second image area I2 follows the path P so that the second image area I2 is positioned to cover all areas of the first image area I1 for one frame of image data from the first image sensor 2002. This means that data from the second image sensor 2004 is available for all areas of the first image area I1. For example, the first image sensor can acquire a single high-resolution frame, while the second image sensor (actuated by the actuator under the control of the controller) acquires multiple frames at a high frame rate (each frame individually at a lower spatial resolution and a lower number of bits). Thus, in this example, image data from the second image sensor 2004 (which may include multiple frames of data) may be used to supplement image data from the first image sensor 2002 for each frame of image data acquired by the first image sensor 2002 across the entire first image area I1.

[0091] In addition, data from the second image sensor can be used to supplement image data from the first image sensor acquired for multiple frames of data from the first image sensor (including, for example, burst photography or video data). In addition, data from the second image sensor can be used to supplement image data from a single (partial) area or range of image data acquired by the first image sensor.

[0092] Path P can be Figure 5 That is, in the example, when covering the entire first image area (such as Figure 5 ), any suitable path of the second image sensor 2004 may be used. For example, the second image area I2 may move in vertical columns across the first image area, the second image area I2 may move in horizontal rows across the first image area, the second image area may move in diagonal lines across the first image area, the second image area may move in concentric circles across the first image area, the second image area may move in a sweeping path across the first image area, and so on.

[0093] Therefore, the present disclosure Figure 5 An example is provided in which the controller is configured to control the actuator according to a predetermined pattern. Figure 5 ), the predetermined pattern may be a pattern in which the second image area covers the entire first image area over several frames of image data of the second image sensor.

[0094] Although the example of covering the entire first image area within a single frame of image data of the first image sensor 2002 using the movement of the second image area has been described, Figure 5 , but the present disclosure is not particularly limited in this regard. In an example, targeted movement of the second image area can be provided (by actuation of the second image sensor 2004).

[0095] Now consider the present disclosure Figure 6 .and Figure 4 and Figure 5 similar, Figure 6 1 and 2 show a first image region and a second image region according to an embodiment of the present disclosure. Figure 4 and Figure 5 As depicted, the first image area I1 is an area covered by the first image sensor 2002 , and the second image area I2 is an area covered by the second image sensor 2004 .

[0096] However, with the present disclosure Figure 5 In contrast, in this example, the movement of the second image area I2 is a targeted movement.

[0097] A targeted movement can be performed to provide the second image area I2 to be aligned with a portion of the first image area I1 for which data from the first image sensor 2002 is most improved when compensated with image data from the second image sensor 2004 (i.e., the portion of the first image area I1 for which image data from the second image sensor 2004 will be most relevant).

[0098] As explained, in an example, the first image sensor 2002 may be an image sensor such as an accumulation type image sensor (such as a CMOS sensor). This type of sensor may suffer in image quality under certain image conditions (e.g., low light image conditions) or for certain types of images (e.g., images with fast moving objects). Therefore, there may be areas where the image quality of the image data from the first image sensor 2002 is lower (e.g., areas with fast moving objects); the data from the second image sensor 2004 will be most relevant to this image area.

[0099] Thus, in an example, the controller may be configured to use data from one or more previous frames of image data from the first image sensor or the second image sensor in order to control the actuator (and thus the position of the second image area). For example, the controller may be configured to analyze the first image data to identify a portion of the first image area where the image quality is below a predetermined threshold, and control the actuator to actuate the second image sensor and position the second image area within the identified portion of the first image area.

[0100] In an example, a test frame of image data may be acquired, wherein the test frame of image data is used in order to identify and determine a targeted region of an image region for which second image data is desired.

[0101] exist Figure 6 In the example of FIG. 1 , the image area I2 is at an initial position within the image area I1. The controller then identifies a target area T of the first image area I1 that is particularly advantageous for having data from the second image sensor 2004. Therefore, the controller is configured to control the actuator 2006 to actuate the second image sensor to move along the Figure 6 The path P shown in 2 moves the second image sensor so that the second image area I2 is positioned at the target area T and is thus aligned with the portion of the first image area I1 .

[0102] In this way, the use of the second image sensor 2004 can be optimized so that data from the second image sensor 2004 is acquired for those areas that need it most. This can reduce the image data from the second image sensor 2004 (obtained only for its most relevant areas), which improves the data efficiency of the image sensor system 2000.

[0103] The predetermined threshold value may vary depending on the situation to which the embodiments of the present disclosure are applied. For example, different predetermined threshold values ​​may be set depending on the type of image being captured, the mode of image capture, and / or the intended use of the image that has been captured. For example, if an information processing device (such as an image capture device) provided with an image sensor 2000 is used in a "sports mode", it may be determined that the reduction of blur in the image is most relevant. Therefore, the predetermined threshold value of image quality related to the blur level in the image may be made lower in order to ensure that the second image data is ensured in a manner that enables the blur level to be reduced.

[0104] In fact, in an example, the image quality may include a brightness level in the first image data, a blur level in the first data, and / or an image detail level in the first image data (indicating that higher resolution imaging will be required for areas with high image detail levels). However, the present disclosure is not particularly limited to these examples of image quality, and according to embodiments of the present disclosure, the controller may use any suitable indicator of image quality to identify a target area for locating the second image area I2.

[0105] In an example, the movement of the second image area I2 (by actuation of the second image sensor 2004) may result in a large change in position. For example, the second image area I2 may be located at a first position, such as the upper left corner of the first image area I1, and may then be moved to a second position, such as the lower right corner of the second image area I2. Thus, the change in position of the second image area I2 may be large (i.e., the movement may exceed the pixel size of the second image sensor).

[0106] However, in other examples, movement of the second image area 12 (via actuation of the second image sensor 2004) may result in a small change in position; in this case, the small change in position is a change in position corresponding to a movement amplitude that is less than the pixel size of the second image sensor.

[0107] Now consider the present disclosure Fig. 7A The present disclosure Fig. 7A An example actuation of the second image sensor is shown in accordance with an embodiment of the present disclosure.

[0108] In this example, several pixels P1, P2 and P3 of a first image sensor are shown. The pixels of the first image sensor 2002 may be pixels of an image sensor such as a CMOS sensor. In addition, a pixel P4 of a second image sensor 2004 is shown. The pixels of the second image sensor may be a SPAD detector or the like. The pixel size of the second image sensor 2004 is larger than the pixel size of the pixels of the first image sensor 2002.

[0109] According to an embodiment of the present invention, the image sensor system 2000 includes an actuator 2006 configured to actuate the second image sensor 2004. Therefore, the second image sensor 2004 can be as disclosed in the present invention. Fig. 7A Move as shown (in this example, horizontally).

[0110] The initial position of the pixel P4 of the first image sensor P4 is determined by the present disclosure. Fig. 7AIn this position, pixel P4 is aligned with pixels P1 and P2 of the first image sensor. Then, after actuation, pixel P4 moves to the second position (by actuation of the second image sensor 2004). The new position of pixel P4 after actuation is determined by the present invention. Fig. 7A In this new position, pixel P4 is aligned with pixels P2 and P3 of first image sensor 2002. Therefore, the movement of second image sensor 2004 can be less than the pixel size of the second image sensor (i.e., in this example, the movement is less than the size of pixel P4).

[0111] In this way, movement of the second image sensor 2004 by the actuator 2006 can be performed with a quantization of movement that is less than the size of a single pixel of the second image sensor 2004; precise movement of the second image sensor 2004 in this manner enables efficient use of data from the second image sensor 2004 when generating an output image of the image sensor system (even when the size of the pixel of the second image sensor 2004 is larger than the size of the pixel of the first image sensor 2000).

[0112] Now consider the present disclosure Figure 7B . Figure 7B An example actuation of a second image sensor, which is an image sensor that can detect the arrival of a single photon, such as a SPAD, is shown in accordance with an embodiment of the present disclosure.

[0113] Figure 7B The first plane of shows a detector array 7000 of a second image sensor. The detector array 7000 comprises a pixel array 7002, wherein, in this example, each pixel of the array 7000 is a SPAD pixel.

[0114] An example light distribution 7004 is shown falling on the detector array 7000. This is the light from the object being imaged (e.g., focused onto the detector by a lens).

[0115] The pixels of the detector array are SPAD pixels; they are capable of detecting the arrival of single photons of light. Therefore, at a certain instance in time (a certain frame of the image), photons may arrive at some pixels in the detector array but not at other pixels in the detector array. That is, pixels of the detector array outside the light distribution 7004 will not detect the arrival of any photons. Pixels within the light distribution 7004 can detect the arrival of photons for a certain frame. However, in different frames, pixels within the light distribution 7004 may not detect the arrival of any photons (because photons will not be continuously received within the light distribution).

[0116] therefore, Figure 7BThe second plane shows an example output of the image sensor when operating in 1-bit (single frame) mode. Here, a pixel 7006 within the light distribution area 7004 has detected the arrival of a photon (within that image frame).

[0117] It can therefore be seen that the effective resolution of the image sensor's frames can be quite low.

[0118] Figure 7B The third plane shows an example output of the image sensor when operating in multi-bit (multi-frame) mode. Here, the output of the SPAD pixels is accumulated and combined across multiple frames. Therefore, even if a pixel does not receive a photon at a certain time instance, the same pixel will still record a value in the output if it is received at a different time instance (different frame) within the multiple frames being combined. The more frames in which photons are detected, the higher the value output by the pixel for the combined multi-frame image will be. Therefore, it can be seen that by using a multi-frame image mode, the effective resolution of the image sensor's frames can be increased. However, the resolution is still quite low.

[0119] However, as previously explained, embodiments of the present disclosure utilize actuation of the second image sensor in order to provide improved image quality. For example, actuation of the second image sensor may be used in order to increase the effective resolution of the output.

[0120] Now consider the present disclosure Figure 7C . Figure 7C An example output of an image sensor according to an embodiment of the present disclosure is shown. Specifically, in this example, sub-pixel size shifts in the x-direction and the y-direction are performed by an actuator between image frames of the second image sensor.

[0121] Figure 7C The top plane shows the shifting of the pixel array between image frames.

[0122] The pixel array of the second image sensor is shown at a first position 7000 at a first time instance (e.g., a first image frame). The same pixel array is then actuated by actuator 2006 under the control of controller 2008 to cause a shift in the x-direction and the y-direction. After the shift, the same pixel array is located at a second position 7010 at a second time instance (e.g., a second image frame). In this example, the shift in the x-direction and the shift in the y-direction are less than the pixel size of the pixel array of the second image sensor (similar to the pixel size of the pixel array of the second image sensor as described in reference to the present disclosure). Fig. 7A described).

[0123] Although in Figure 7COnly two time instances are shown in the top plane of , but it should be understood that the shift in the x direction and the shift in the y direction can be performed for a greater number of time instances (e.g., for more image frames). The present disclosure is not particularly limited in this regard. For example, the number of shifts of the pixel array can depend on the frame rate of the second image sensor and the nature of the object being imaged. However, it should be understood that the frame rate of the pixels of the second image sensor can be very high (typically, SPAD pixels can have a frame rate of more than 1000 frames per second). Therefore, it can be considered that the light distribution 7004 remains in a constant position between image frames. Therefore, since the light distribution 7004 can be considered to remain in a constant position, the relative position of the light distribution 7004 and the pixel array will be changed by actuation of the pixel array. In this way, due to the actuation of the pixel array for different time instances, the light distribution will fall on different parts of the pixel array (and therefore on different pixels of the pixel array) at different time instances.

[0124] Now consider the present disclosure Figure 7C The second plane of FIG. 7012 shows the output of the pixel array at three different time instances 7012A, 7012B, and 7012C. For each of these different time instances, the position of the light distribution on the pixel array is slightly different; this is because the pixel array has been shifted in the x-direction and the y-direction between image frames.

[0125] The output of the array at the first time instance 7012A shows the pixels that have detected photons at the first time instance. The output of the array at the second time instance 7012B shows the pixels that have detected photons at the second time instance. The output of the array at the third time instance shows the pixels that have detected photons at the third time instance. Thus, each of 7012A, 7012B, and 7012C shows a possible 1-bit frame sensor output at a different XY sampling position (where the actuator has shifted the x and y positions of the pixel array between different time instances).

[0126] By sampling the light distribution using actuation of the pixel array in this manner, the spatial resolution of the second image sensor can be improved. That is, a high spatial resolution image can be reconstructed by the controller using different image data (i.e., 7012A, 7012B, and 7012C in this example) that have been acquired (at different sampling locations).

[0127] The present disclosure Figure 7CThe third plane of shows a high spatial resolution image that has been reconstructed from the image data that has been acquired. The constructed image has a virtual resolution of N×M pixels compared to the n×m actual pixels in the pixel sensor array (where N>>n and M>>m). This is because oversampling of the image can be performed by shifting the position of the pixel array and acquiring image data (sampling) at each of the different positions of the pixel array. A person skilled in the art can use any suitable signal processing to perform reconstruction of the image based on the spatial oversampling that has been performed.

[0128] Thus, actuation of the second image sensor may be used to increase the spatial resolution of the output image. Thus, the second image sensor may be used to obtain high quality image data, which may be used to optimize or enhance the image data from the first image sensor. Furthermore, the second image sensor may be implemented in an image sensor system without increasing the physical size (form factor) or cost of the image sensor system.

[0129] Although the present disclosure has been described with reference to a position shift smaller than the pixel size of the second image sensor, Figure 7C The actuation described above is described above, but the present disclosure is not particularly limited in this regard. In other examples, the size of the displacement caused by the actuation of the sensor may be larger than the pixel size of the second image sensor.

[0130] Now consider the present disclosure Fig.7D . Fig.7D An example output of an image sensor according to an embodiment of the present disclosure is shown. Specifically, in this example, a sub-pixel size shift in the x-direction and the y-direction is performed by an actuator between image frames of a second image sensor. However, in this example, the size of the pixel shift exceeds the size of a pixel of the second image sensor.

[0131] In fact, in this example, the actuator is controlled to perform a shift to a random or pseudo-random position. In addition, the SPAD sensor mask of the activated pixels is also randomly or pseudo-randomly generated and changes over time. This means that both the position of the pixel array and the pattern of activated SPAD pixels change over time.

[0132] therefore, Fig.7D The first plane shows the shift in the x-position and y-position of the pixel array for two different time instances; again, it can be assumed that the light distribution 7004 remains in the same absolute position. Therefore, the change in the position of the pixel array changes the portion of the pixel array that is incident with the light distribution. It is worth noting that although Fig.7DThe top plane of shows a certain size of x displacement and a certain size of y displacement, but it should be understood that the present disclosure is not particularly limited in this regard. Instead, the size and direction of the displacement will change as the actuator is controlled to perform a displacement to a random or pseudo-random position.

[0133] Fig.7D The second plane shows the output of the image sensor at two different time instances. Fig.7D On the left hand side of the second plane of , the output of the array at a first time instance is shown. Here, the array is in an initial position so that the light distribution 7004 is incident on a first area of ​​the pixel array. In addition, several pixels of the SPAD array are activated. The activated pixels are shown as shown for pixel 7016. Therefore, in the pixel array, only one pixel is activated, exists in the area of ​​the light distribution 7004, and a photon is detected at this first time instance.

[0134] exist Fig.7D On the right hand side of the second plane of , the output of the array at the second time instance is shown. Here, the array is in a different position, having been shifted to a random or pseudo-random position by actuation of the actuator. In addition, the mask of the activated pixels has also been randomly or pseudo-randomly generated and changed at the second time instance (compared to the first time instance). Therefore, at the second time instance, the light distribution is incident on a second area of ​​the pixel array (different from the first area of ​​the pixel array). In addition, two pixels are activated, exist within the area of ​​the light distribution 7004, and photons are detected at the second time instance.

[0135] Pseudo-random sampling of the light distribution 7004 can be performed by actuating the pixel array to random or pseudo-random positions and by changing the random or pseudo-random pattern of enabled pixels. This can be performed over several image frames; the number of frames can be greater than the reference image. Fig.7D The two frames described. Therefore, random (or pseudo-random) oversampling of the image can be performed by shifting the position of the pixel array and acquiring image data (samples) at each of the different positions of the pixel array. A person skilled in the art can use any suitable signal processing to perform reconstruction of the image from the spatial oversampling that has been performed. This provides an output image with a much higher resolution than the pixel array.

[0136] Thus, random (or pseudo-random) actuation of the second image sensor can be used to improve the spatial resolution of the output image. Thus, the second image sensor can be used to obtain high-quality image data, which can be used to optimize or enhance the image data from the first image sensor. Furthermore, the second image sensor can be implemented in an image sensor system without increasing the physical size (form factor) or cost of the image sensor system.

[0137] <Actuation of Second Image Sensor>

[0138] The movement of the second image area 12 by actuation of the second image sensor 2004 has been described. However, further details of the actuation of the second image sensor 2004 will now be described.

[0139] The image sensor system 2000 includes an actuator 2006 for actuation of the second image sensor 2004. In an example, the actuator 2006 may be configured to actuate the second image sensor 2004 as a whole (so that the second image sensor 2004 changes its position as a unit). In an example, the actuator 2006 may be configured to actuate the second image sensor 2004 by actuation of individual parts or components of the second image sensor. This enables different parts or components of the second image sensor to move as needed, thereby providing improved control over the movement of the second image sensor (and the second image area I2).

[0140] Now consider the present disclosure Figure 8 . Figure 8 An example configuration of a second image sensor according to an embodiment of the present disclosure is shown.

[0141] In this example, image sensor 2004 includes pixel array 2004A (such as a SPAD array) and lens 2104. Lens 2104 is an example of a refractive element that can be used in an image sensor such as image sensor 2004. That is, lens 2104 receives light from a scene (which may have passed through an image filter) and focuses the light on pixel array 2004A so that pixel array 2004A can acquire image data of the scene. As an alternative to a refractive element (such as a lens), a reflective element or a diffractive element can be provided to focus the light onto the image sensor. In examples, a combination of optical elements (including refractive, reflective, and diffractive elements) can be provided.

[0142] In an example, actuator 2006 may be configured to actuate image sensor 2004 as a whole (e.g., the actuator may be coupled to a housing of image sensor 2004). In this manner, the actuator may cause image sensor 2004 to undergo rotational and / or linear actuation. The relative positions of components of image sensor 2004 (i.e., the relative positions between pixel array 2004 and lens 2104) may be maintained.

[0143] In an example, the actuator 2006 can be configured to actuate one or more individual parts of the image sensor 2004. For example, the actuator can be configured to actuate the pixel array 2004A. In an example, the actuator can be configured to actuate the lens 2104. In this way, the actuator can cause the pixel array 2004A and / or the lens 2104 to undergo rotational and / or linear actuation. The relative position of the components of the image sensor 2004 (i.e., the relative position between the pixel array 2004 and the lens 2104 can therefore be changed). This can provide an increased level of control when changing the position of the second image area within the first image area.

[0144] In an example, the actuator 2006 may be configured to adjust the position of one or more elements of the second image sensor 2004 as part of active alignment of the image sensor. For example, a shift of one or more elements of the image sensor in the z-direction of the second image sensor 2004 may be performed. Thus, the actuator 2006 may be controlled by the controller 2008 to perform active alignment of the image sensor based on one or more external factors (such as temperature, etc.) to maintain the configuration of the image sensor 2004 in an optimal configuration.

[0145] Although the second image sensor 2004 has been described with reference to Figure 8 , it should be understood that the present invention is not particularly limited to the internal configuration of the second image sensor 2004 illustrated in this example. Figure 8 Only a single lens 2104 is shown, but it should be understood that multiple lenses (or optical elements) may be provided. In addition, although only a single actuator 2006 is shown, multiple different actuators may be provided. Different actuators may be configured for different types of movement (e.g., linear or rotational), may be configured for actuation of different elements within the second image sensor (e.g., pixel array 2004A and lens 2014), and may be of different types (e.g., piezoelectric actuators and ultrasonic actuators or shape memory alloy actuators).

[0146] <Filter>

[0147] The first image sensor 2002 and the second image sensor 2004 of the image sensing system 2000 have been described. Each of the first image sensor 2002 and the second image sensor 2004 includes a pixel array (a pixel array 2002A for the first image sensor and a pixel array 2004A for the second image sensor). The pixels of the first image sensor are configured to detect light in at least a first wavelength band, and the pixels of the second image sensor are configured to detect light in at least a second wavelength band. In an example, the first wavelength band and the second wavelength band may be the same wavelength band, wherein each image sensor is configured to detect light in a visible wavelength range (e.g., light in a wavelength range of 380 to 700 nm).

[0148] However, in an example, to provide a color image, a filter may be provided; the filter may be configured such that a portion of the pixels of the image sensor receive light of a first wavelength band within a wavelength range (e.g., red light), a second portion of the pixels of the image sensor receive light of a second wavelength band within the wavelength range (e.g., green light), and a third portion of the pixels of the image sensor receive light of a third wavelength band within the wavelength range (e.g., blue light). In this way, the image sensor may be used to provide a color image (e.g., composed of separate red, green, and blue image data from the image sensor).

[0149] In an example, the filter may be a filter such as a color filter array. In an example, the color filter array may be a Bayer filter.

[0150] Now consider the present disclosure Fig. 9 . Fig. 9 Example filter patterns according to embodiments of the present disclosure are shown.

[0151] Fig. 9 An example filter array of may be a filter array for the first image sensor 2002. Thus, Fig. 9 The pixel array 2002A of the first image sensor 2002 in three different planes is shown. In this example, the first pixel array is a 6 by 6 pixel array (however, the present disclosure is not particularly limited in this regard). Fig. 9 The first plane R of shows the configuration of the filter corresponding to the portion of the filter enabling transmission of light in the red wavelength range (so that the pixel in this position can be used to generate image data for red light). Fig. 9 The second plane G of shows a configuration corresponding to a filter that enables transmission of a portion of light in the green wavelength range (so that the pixel in this position can be used to generate image data for green light). Fig. 9The third plane B shows a configuration corresponding to a filter that enables transmission of a portion of light in a blue wavelength range (so that the pixel in this position can be used to generate image data for blue light).

[0152] Using such Fig. 9 With the filters shown, the first image sensor 2002 may be used to acquire image data that may produce a color image.

[0153] Although the first image sensor 2002 has been described with reference to Fig. 9 , but it should be understood that the second image sensor 2004 may also be provided with a filter such as a color array so that the second image sensor 2004 can acquire image data of different colors, which can then be used to supplement or enhance the data acquired by the first image sensor.

[0154] Furthermore, although the color filter arrays such as the Bayer filter have been described above, Fig. 9 , but it should be understood that the present disclosure is not particularly limited in this regard. More generally, when imaging is performed using the image sensor system 2000 as needed according to the situation in which the embodiments of the present disclosure are applied, any suitable filter or filter type material can be used.

[0155] <Generation of output image>

[0156] like Figure 2 The depicted image sensor system 2000 may be configured to use image data from the first image sensor 2000 and the second image sensor 2002 in order to generate an output image.

[0157] That is, in an example, the controller may use image data from the second image sensor 2004 (which has been actuated to align the second image area with a position within the first image area) to enhance the image data from the first image sensor. For example, when the first image sensor is a sensor such as a CMOS sensor, the image quality of the image data acquired from the first image sensor may be low in certain circumstances (such as at low brightness levels). Therefore, data from the second image sensor (which can detect the arrival of a single photon) may be used to enhance the image data from the first image sensor, thereby providing an output image from the image system 2000 with improved image quality (such as better image quality under lower light conditions).

[0158] There is no particular limitation on the manner in which the image data from the first image sensor 2000 is combined with the image data from the second image sensor 2004 to generate an output image from the image system 2000. Any suitable algorithm or process for combining or fusing image data known in the art may be used as needed. In an example, a spatial domain fusion method may be used. In an example, a transform domain fusion method may be used. In an example, a fusion method such as principal component analysis may be used (an example of a spatial domain fusion method).

[0159] Furthermore, in examples, a trained model may be used in order to generate output images.

[0160] Now consider the present disclosure Fig.10 The present disclosure Fig.10 An example of generating an output image according to an embodiment of the present disclosure is shown.

[0161] In this example, a training model 9000 is provided. The training model 9000 has been trained on image data (simulated or historical) to generate output images using image data from several image sensors. The training of the training model is described in more detail later.

[0162] The training model receives image data 1 from the first image sensor 2002. In addition, the training model receives image data 2 from the second image sensor 2004. The second image sensor 2004 may have a much higher frame rate than the first image sensor 2002. Therefore, the training model may receive multiple frames of image data from the second image sensor 2004 in the time period of receiving one frame of image data from the first image sensor 2002. The training model is thus configured to use a single frame of image data from the first image sensor 2002 and multiple frames of image data from the second image sensor 2004 in order to generate an output image. The multiple frames of image data from the second image sensor may include frames that together cover an area of ​​the first image sensor (such as when the second image area is smaller than the first image area). The multiple frames of image data from the second image sensor may include frames that together provide high-resolution data covering an area of ​​the first image sensor (such as when the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and oversampling has been performed by actuation of the second image sensor).

[0163] The output image is an image in which the data from the second image sensor 2004 has been used by the training model to enhance the image data from the first image sensor 2002. Therefore, the output image generated by the training model is an image with improved image quality compared to an image generated from the image data from the first image sensor 2002. That is, the data from the second image sensor 2004 can be used to enhance the image brightness or image resolution compared to an image generated from only the first data (from the first image sensor). In addition, the data from the second image sensor 2004 can be used to reduce the blur level of objects in the image compared to an image generated from only the first data (from the first image sensor).

[0164] In addition, because the second data is generated according to the second image sensor actuated by the actuator 2006 under the control of the controller 2008, the second image sensor 2004 can be made smaller. That is, the size of the second image sensor 2004 can be made smaller compared to the size of the first image sensor. Therefore, the image quality can be improved without increasing the physical size of the image sensor system (or the device in which the image sensor system is used).

[0165] In an example, the methods and techniques herein can be implemented at least in part using a supervised machine learning model (as an example of a training model).

[0166] A supervised learning model is trained using labeled training data to learn a function that maps inputs (usually provided as feature vectors) to outputs (i.e., labels). The labeled training data includes pairs of inputs and corresponding output labels. The output labels are usually provided by an operator to indicate the expected output for each input. The supervised learning model processes the training data to produce an inferred function that can be used to map new (i.e., unseen) inputs to labels.

[0167] Input data (during training and / or inference) can include various types of data, such as numerical values, images, video, text, or audio. The raw input data can be preprocessed to obtain appropriate feature vectors for use as input to the model, for example, features of an image or audio input can be extracted to obtain corresponding feature vectors. It should be understood that the type of input data and the techniques for preprocessing the data (if necessary) can be selected based on the specific task for which the supervised learning model is used.

[0168] In an example, the labeled training data may include data from a first image sensor and a second image sensor; the training data may also include examples of images that have been generated from the first image data and the second image data (where the second image data has been used to improve the image quality of the first image data).

[0169] Once prepared, the labeled training data set is used to train a supervised learning model. During training, the model adjusts its internal parameters (e.g., weights) in order to optimize (e.g., minimize) an error function, aiming to minimize the difference between the model's predicted output and the labels provided as part of the training data. In some cases, the error function may include a regularization penalty to reduce overfitting of the model to the training data set.

[0170] A supervised learning model may use one or more machine learning algorithms in order to learn the mapping between its inputs and outputs. Example suitable learning algorithms include linear regression, logistic regression, artificial neural networks, decision trees, support vector machines (SVMs), random forests, and K-nearest neighbor algorithms.

[0171] Once trained, supervised learning models can be used for inference, i.e., for predicting outputs for previously unseen input data. Supervised learning models can perform classification tasks and / or regression tasks. In classification tasks, supervised learning models predict discrete class labels for input data and / or assign input data to predetermined categories. In regression tasks, supervised learning models make predictions for labels that are continuous values.

[0172] Thus, in the example (once trained), the trained model can be used to generate output images from previously unseen input data (ie, previously unseen data from the first image sensor and the second image sensor).

[0173] In some cases, a limited amount of labeled data may be available for training the model (e.g., because labeling of the data is expensive or impractical). In this case, the supervised learning model can be extended to further use unlabeled data and / or generate labeled data.

[0174] Considering the use of unlabeled data, the training data may include both labeled and unlabeled training data, and semi-supervised learning may be used to learn the mapping between the input and output of the model. For example, a graph-based method such as Laplace regularization may be used to extend the SVM algorithm to Laplace SVM to perform semi-supervised learning on partially labeled training data.

[0175] Considering the generation of labeled data, an active learning model can be used, in which the model actively queries an information source (such as a user or operator) to label data points with the desired output. Labels are typically requested for only a subset of the training dataset, thus reducing the amount of labeling required compared to fully supervised learning. The model can select the examples for which it requests labels, for example, the model can request labels for data points that most change the current model or that most reduce the generalization error of the model. Semi-supervised learning algorithms can then be used to train the model based on the partially labeled dataset.

[0176] In addition, examples of the present invention may use generative artificial intelligence (AI) systems and techniques. Generative artificial intelligence systems may be used as examples of training models in the present disclosure.

[0177] A generative AI system learns patterns and structures in its input training data in order to then generate new output data that exhibits similar characteristics to the training data. Each of the input training data and the output data may include various types of data, such as images, video, text, or audio. For example, a generative AI system may learn patterns in an input training image and then generate an image with similar characteristics.

[0178] A generative AI system can generate output data based on an input prompt. As with training and output data, prompts can include various types of data, such as images, video, text, or audio. Prompts can be of the same or different data types as the model's training and / or output data. For example, the input prompt can include text and the output data can include an image (e.g., matching the input text description of the desired image), or the input prompt can include an image and the output data can include audio data (e.g., having a subject that matches the input image).

[0179] A generative AI system may include a generative model that is trained to learn a probability distribution of input training data and to generate new output data based on the learned distribution. For example, for a set of data instances / observable variables (X) and a set of labels / target variables (Y) in a training dataset, a generative model may learn a joint probability distribution p(X,Y) of the data instances and labels, and / or a probability distribution of the data instances p(X) (e.g., when no labels are available).

[0180] Example suitable generative models for learning the probability distribution of input training data include variational autoencoders (VAEs), transformation-based models, diffusion models (e.g., denoising diffusion probability models (DDPMs)), reinforcement learning (RL), and generative adversarial networks (GANs). The choice of a generative model can depend on the specific task being performed by the generative AI system.

[0181] A generative model may include one or more artificial neural networks. For example, a variational autoencoder (VAE) may include a pair of neural networks that act as an encoder and decoder, respectively, to or from a reduced (i.e., latent space) representation of the training data, and a generative adversarial network (GAN) may include a first "generator" neural network that generates new data and a second "discriminator" neural network that learns to distinguish between the generated data and real data. One or more of the constituent neural networks of a generative model may be trained together or separately.

[0182] During training, the generative model can adjust its internal parameters (e.g., neural network weights) in order to optimize (e.g., minimize) a loss / error function, aiming to minimize the difference between the generated output data and the desired output data. It should be understood that the specific loss function and the algorithm used to optimize the function may vary depending on the nature of the generative model and its intended application. For example, a mean squared error loss function may be used for image generation tasks, and a cross entropy loss function may be used for text generation tasks. These loss functions can be optimized using various existing optimization algorithms (such as gradient descent).

[0183] Once trained, a generative model can be used to generate new output data based on an input prompt. The input prompt can be provided by a user or by an appropriate means (e.g., using an application programming interface (API)). Thus, a generative AI system allows new content (e.g., images, text, or audio) to be generated based solely on a prompt, and does not require detailed instructions for doing so.

[0184] Thus, in an example, a training model may be used to generate an output image from already acquired image data. However, the present disclosure is not particularly limited in this regard, and more generally, any suitable processing may be applied by the controller to already acquired image data to generate an output image.

[0185] In an example, the process of generating an output image may be performed by an external device. In an example, the process of generating an output image may be performed by a processing circuit system of an information processing device (such as an image capture device) including the image sensor system 2000. In an example, the process of generating an output image may be performed by a server.

[0186] In an example, processing may be performed on the data from the image sensor before processing to generate image data (i.e., the image data may be pre-processed). That is, the image data acquired from the second image sensor may have a very high frame rate. Therefore, the image data generated by the second image sensor 2004 may be very large. Therefore, before processing the image data (or sending the data to an external device for processing), the controller may perform an analysis of the image data that has been acquired. The analysis (pre-processing) performed by the controller may be used to identify portions of the second image data that have information that can be used to enhance the first image data. For example, analysis may be performed for areas of low image quality of the image data from the first image sensor. The controller may then reduce the second image data by filtering the second image data to retain only the most relevant second image data (second image data that can be used to enhance the output image). This may reduce power consumption, latency, and storage requirements of the image sensor system 2000.

[0187] In an example, during processing (or pre-processing), the controller may be configured to identify one or more areas of low image quality (e.g., low signal level, etc.), and may be configured to control actuation of the actuator so that the second image sensor (e.g., SPAD) acquires additional information for the identified one or more such areas. Such targeted actuation of the second image sensor may ensure that image data from the second image sensor is acquired for the most relevant one or more areas, thereby improving image quality. In addition, when the second image data is acquired only for these most relevant image areas (rather than the entire image), power consumption, latency, and storage requirements may also be reduced.

[0188] In an example, during processing (or pre-processing), the controller may be configured to use the high frame rate of the second image sensor to track objects that are moving at high speed in the scene (because these objects may move between frames of the second image sensor). The controller may then be configured to use information about the movement of fast-moving objects in the images from the second image sensor to compensate for blurring caused by such fast-moving objects in image data from the first image sensor (which has a higher resolution than the second image sensor, but a lower frame rate). In an example, the controller may use a method such as that described in reference to the present disclosure. Fig.10 The compensation is applied by the training model described. However, the present disclosure is not particularly limited in this regard. In this way, motion blur in the image data from the first image sensor can be reduced, which further improves the image quality of the output image.

[0189] Therefore, the image sensor 2000 may generate an output image based on the image data acquired by the first image sensor and the second image sensor.

[0190] <Flight time>

[0191] In some cases, it may be desirable to detect depth within an environment using an image sensor that is capable of detecting the arrival of a single photon. That is, a sensor that is capable of detecting the arrival of a single photon (such as a single photon avalanche diode, SPAD) may be used in order to accurately determine the time of arrival of the photon. Thus, a time-of-flight sensor may be used to determine depth based on the time it takes for light emitted from a light emitter and reflected by an object in the scene to arrive at the SPAD. Depth is determined based on the time it takes for light to travel from the light emitter to the object and then back to the SPAD.

[0192] However, as previously explained in the background, it can be very difficult to produce such image sensors (capable of detecting the arrival time of a single photon) that meet the constraints of physical size and space. In addition, the cost of these image sensors can be very high. Therefore, the use of single photon detector image sensors (such as depth sensors) is quite limited.

[0193] Therefore, according to embodiments of the present disclosure, image sensor systems, methods and computer programs are provided.

[0194] That is, according to an embodiment of the present invention, an actuator may be provided within an image sensor system to actuate an image sensor (such as a SPAD) to move the image sensor while acquiring image data at a high frame rate. This enables a small, low-resolution image sensor to be used at high-speed actuation and frame rates to increase the effective pixel count of the image sensor. Thus, an image system may be provided that can produce higher quality image depth data without increasing the physical size and cost of the image sensor system. Thus, an image sensor system that can be used in a wide range of situations may be provided.

[0195] Now consider the present disclosure Fig.11A and Fig. 11B . Fig.11A and Fig. 11B A configuration of an image sensor system according to an embodiment of the present disclosure is shown.

[0196] In this example, an image sensor system 1100 (a time-of-flight sensor) is shown that includes an image sensor 1104 and a light emitter 1102 .

[0197] Fig.11A Image sensor 1104 is shown in the xy plane.

[0198] The light emitter 1102 of the image sensor system 1100 is configured to emit light of at least a first wavelength range within a first image region. In an example, the light emitter can be a vertical cavity surface emitting laser (VCSEL). However, the present disclosure is not particularly limited to this specific example of a light emitter. In an example, the light emitter 1102 can be any light source that can emit photons that can be used to measure depth in a scene (based on time of flight). In an example, the light emitter can emit light in a wavelength range of 650nm to 1400nm. In fact, in an example, the light emitted by the light emitter 1102 can be light outside the visible wavelength range; this enables the light emitter of the image sensor system to be used with a visible light imaging system.

[0199] Image sensor 1104 is such as that already described in reference to the present disclosure. Figure 2 The image sensor of the image sensor 2004 described above. Thus, the second image sensor includes a pixel array configured to detect light of at least a first wavelength range (i.e., the wavelength of light emitted by the light emitter) within a second image area (the area over which the light emitter emits light) that is smaller than the first image area. In addition, each pixel of the pixel array of the image sensor is configured to detect the arrival of a single photon. Thus, as described with reference to the present disclosure Figure 2 As depicted, image sensor 1104 may be an image sensor such as a SPAD.

[0200] An actuator 1112 and a controller are also provided as part of the image sensor 1104. FIG. 11A to FIG. 11E The controller is not shown in the example.

[0201] The actuator 1112 is configured to actuate the image sensor, and the controller is configured to control the actuator to actuate the second image sensor and position the second image area within a portion of the first image area while obtaining depth information with the image sensor system. The details of the actuator and the controller are similar to those of the present disclosure. Figure 2 The examples are the same as described in , and for the sake of brevity of this disclosure, no further details will be provided at this stage.

[0202] Fig. 11B Image sensor 1104 is shown in the xz plane.

[0203] In the xz plane, several additional elements of the image sensor system 1104 are shown. For example, a lens 1106 is provided on the image sensor 1104; this can be used to focus light on the image sensor 1104. In an example, the lens can be any optical element configured to focus light on the image sensor. In an example, the lens can be provided as part of an external device and not part of the image sensor system 1100 at all. Likewise, in this example, a diffractive optical element 1108 is provided above the light emitter 1102.

[0204] In an example, actuator 1112 may be configured to actuate image sensor 1104 and / or lens 1106. That is, image sensor 1104 and lens 1106 may be actuated to maintain the relative configuration of image sensor 1104 and / or lens 1106. Alternatively, image sensor 1104 and lens 1106 may be actuated to change the relative configuration of image sensor 1104 and / or lens.

[0205] exist Fig. 11B In the embodiment, the image sensor system 1100 is formed on a printed circuit board. Fig. 11B In the example of , the actuator 1112 is disposed below and coupled to the PCB 1110. Thus, the actuator can be used to move the image sensor system 1100 as a whole (i.e., while maintaining the relative positions of the elements of the image sensor system 1100). For example, the actuator can be configured to translate the image sensor system in the X and Y directions. Alternatively, the actuator can be configured to rotate the actuator along the X and Y axes. This enables spatial scanning to be performed.

[0206] By actuation of the image sensor 1104, the image sensor 1104 can be used to detect arrival time data of photons across the first image area, i.e., such that the image area covered by the image sensor (i.e., the second image area) is smaller than the area covered by the light emitter. Thus, the controller can generate (using data from the image sensor 1104) image depth information having improved quality across the first area.

[0207] Now consider Fig. 11C The present disclosure Fig. 11CAnother configuration of the image sensor system 1100 according to an embodiment of the present disclosure is shown. In this example, the actuator 1112 is coupled to the image sensor 1104 and the light emitter 1102 separately. In an example, a plurality of independent actuators 1112 may be provided. Thus, the actuator 1112 may be used to move the second image sensor 1104 independently of the other elements of the image sensor system 1104. For example, the actuator 1112 may move the second image sensor 1104 relative to the lens 1106, the light emitter 1102, and / or the diffractive optical element 1108. Similarly, the actuator 1112 may move the light emitter 1102 relative to the image sensor 1104, the diffractive optical element 1108, and / or the lens 1106. In this way, spatial scanning may be performed by changing the position of the components of the image sensor system 1100 under the control of the controller. In addition, since the relative positions of the components may be changed, more precise control of the spatial scanning may be performed.

[0208] In addition, actuation of the light emitter (such as a VCSEL) can be performed to increase the field of view and / or control the target of the light emitter so that light is provided only in the area (or multiple areas) where light is needed. For example, actuation of the light emitter can be performed (under the control of the controller) so that a target domain (or area) is illuminated by light from the light emitter so that depth information can be acquired only for that target domain or area. This can further improve efficiency because light can be emitted only for specific areas where depth information is needed.

[0209] Now consider Fig.11D . Fig.11D Another configuration of an image sensor system 1100 according to an embodiment of the present disclosure is shown. Fig.11D Examples with Fig. 11C The example of the embodiment of the present invention differs in that the actuator 1112 is coupled to the lens 1106 of the image sensor 1104 and the diffractive optical element 1108. In the example, the actuator 1112 may be coupled to the image sensor 1104 and the lens 1106. Furthermore, in the example, the actuator 1112 may be coupled to one of the lens 1106 or the diffractive optical element 1108. Alternatively, in the example, the actuator 1112 may be coupled to both the lens 1106 and the diffractive optical element 1108.

[0210] In this example, since the actuator 1112 is coupled to the lens 1106, the actuator 1112 can be used to drive a stop (such as a nine-blade, for example) to change the aperture of the lens and / or the diffractive optical element. Therefore, aperture control can be performed efficiently and easily.

[0211] Now consider Fig.11E . Fig.11EAnother configuration of an image sensor system 1100 according to an embodiment of the present disclosure is shown. Fig.11E Examples with Fig.11D The example of 1100 differs in that the actuator 1112 is coupled between the lens 1106 and the sensor 1104 and / or between the diffractive optical element 1108 and the light emitter 1102. In this way, the actuator 1112 can be used to change the relative position of the lens 1106 and the image sensor 1104 in the z-direction. Alternatively or additionally, the actuator 1112 can be used to change the relative position of the diffractive optical element 1108 and the light emitter 1102 in the z-direction. Therefore, the actuator 1112 can be controlled by the controller to change the relative z-position of these elements, and can therefore be used to perform focusing of the image sensor system 1100.

[0212] Furthermore, although it has been described separately FIG. 11B to FIG. 11E 1100, but it should be understood that these examples can be further combined to provide enhanced control of the image sensor system. For example, a plurality of different actuators 1112 can be provided, each actuator coupled to an independent element of the image sensor system 1100, which thus enables a plurality of different movements to be performed as desired.

[0213] According to an embodiment of the present disclosure, the manner in which the controller uses data from the image sensor to generate depth information is not particularly limited. For example, according to the present disclosure, any suitable process or algorithm may be used to generate depth information from arrival time information, depending on the situation to which the embodiment of the present disclosure is applied. However, it should be understood that, based on accurate arrival time information (from the image sensor), the time taken from the light emitter emitting light to the light being detected at the image sensor after being reflected from an object in the scene can be determined. This information can be used to determine the distance to the object, and thereby determine the depth information of the scene.

[0214] Therefore, in an example, an image sensor system for obtaining depth information is provided, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range within a first image area; an image sensor comprising a pixel array configured to detect light of at least the first wavelength range within a second image area smaller than the first image area, each pixel in the pixel array of the image sensor being configured to detect the arrival of a single photon; an actuator configured to actuate the image sensor; and a controller configured to control the actuator to actuate the image sensor and position the second image area within a portion of the first image area when obtaining depth information using the image sensor system.

[0215] In this way, an image sensor system can be provided that can generate higher quality image depth data without increasing the physical size and cost of the image sensor system. Thus, an image sensor system that can be used in a wide range of situations can be provided.

[0216] Furthermore, in an example, the light emitter 1102 may be included in an image sensor such as the image sensor 2000. That is, an image sensor having the first image sensor 2002, the second image sensor 2004, and the light emitter 1102 may be provided. However, in an example, the image sensor may be provided with the second image sensor 1104 and the light emitter 1102. Therefore, the first image sensor 2002 and the light emitter 1102 may be used alone or in combination with the (actuated) second image sensor.

[0217] <Method>

[0218] Therefore, more generally, methods of controlling an image sensor system are provided according to embodiments of the present disclosure.

[0219] Fig.12 A method according to an embodiment of the present disclosure is shown. Fig.12 The method may be implemented by a control circuit system of an image sensor system (such as, for example, as described in reference to the present disclosure) Figure 2 Alternatively, Fig.12 The method may be implemented by a control circuit system external to the image sensor system (such as, for example, as described in reference to the present disclosure) Figure 1 The processing circuitry 1002 of the device 1000 is described as executing.

[0220] Fig.12 The method starts at step S1200 and proceeds to step S1202.

[0221] In step S1202 , the method includes controlling an actuator to actuate a second image sensor so that a second image area is positioned within a portion of the first image area during image capture using the image sensor system.

[0222] The method proceeds to step S1204 and ends with this step.

[0223] In this way, the position of the second image area within the first image area can be changed during image capture.This enables the use of the second image sensor to obtain image data that can be used to optimize or enhance the image data from the first image sensor.

[0224] It should be understood that Fig.12The method is an example illustration of a method of controlling an image sensor system according to an embodiment of the present disclosure. Therefore, the present disclosure is not particularly limited to the method of controlling an image sensor system according to an embodiment of the present disclosure. Fig.12 In the embodiment of the present disclosure, the method of controlling the image sensor system may further include a plurality of additional method steps.

[0225] Fig.13 A method according to an embodiment of the present disclosure is shown. Fig.13 The method may be implemented by a control circuit system of an image sensor system (such as, for example, as described in reference to the present disclosure) Figure 2 The controller 2008 described above is executed. Alternatively, Fig.13 The method may be implemented by a control circuit system external to the image sensor system (such as, for example, as described in reference to the present disclosure) Figure 1 The processing circuitry 1002 of the device 1000 is described as executing.

[0226] Fig.13 The method starts at step S1300 and proceeds to step S1302.

[0227] In step S1302 , the method includes acquiring first image data. The first image data is image data acquired using the first image sensor 2002 of the image sensor system 2000 .

[0228] In step S1304 , the method includes acquiring second image data. The second image data is image data acquired using the second image sensor 2004 of the image sensor system 2000 .

[0229] In step S1306, the method includes controlling an actuator to actuate a second image sensor so that a second image area is positioned within a portion of the first image area during image capture using the image sensor system. In this example, control of the actuator may be performed based on acquired first image data. For example, analysis of first image data from the first image sensor 2002 may indicate that a portion of the first image data from the first image sensor 2002 has an image quality below a threshold (e.g., an excessive blur level in the image in a certain image portion). Thus, the actuator may be controlled to position the second image area so that second image data (acquired by the second image sensor 2002) may be acquired for the portion of the first image area. Thus, Fig.13 The control method shown provides a feedback mechanism by which second image data (based on the acquired first image data) is acquired for a target portion of a first image area.

[0230] In step S1308, the output image data is generated using the first image data and the second image data that have been acquired. In an example, the first image data and the second image data may be fused or combined by the controller to generate the output image. In an example, a training model may be used to generate the output image.

[0231] Once the output image has been generated, Fig.13 The example method proceeds to step S1310 and ends with this step.

[0232] It should be understood that Fig.13 The method is an example illustration of a method of controlling an image sensor system according to an embodiment of the present disclosure. Therefore, the present disclosure is not particularly limited to the method of controlling an image sensor system according to an embodiment of the present disclosure. Fig.13 In the embodiment of the present disclosure, the method of controlling the image sensor system may further include a plurality of additional method steps. In addition, according to the embodiment of the present disclosure, the steps of the method may be similar to Fig.13 2002). For example, once the first output image has been generated, the method may return to step S1304 (acquisition of first image data). Thus, in an example, the frame rate of the output image generated by the image sensor system may be the same as the frame rate of the first image sensor 2002 (where the higher frame rate of the second image sensor 2004 is used in order to capture image data that can be used to enhance the image data acquired from the first image sensor 2002).

[0233] Fig.14 A method according to an embodiment of the present disclosure is shown. Fig.14 The method may be performed by a control circuit system of an image sensor system (such as a controller of the image sensor system 1100 as described with reference to FIG. 11 of the present disclosure). Alternatively, Fig.14 The method may be implemented by a control circuit system external to the image sensor system (such as, for example, as described in reference to the present disclosure) Figure 1 The processing circuitry 1002 of the device 1000 is described as executing.

[0234] Fig.14 The method starts at step S1400 and proceeds to step S1402.

[0235] In step S1402 , the method includes controlling an actuator to actuate a second image sensor so that the second image area is positioned within a portion of the first image area when depth information is obtained using an image sensor system (such as, for example, image sensor system 1100 as described with reference to FIG. 11 of the present disclosure).

[0236] The method proceeds to step S1404 and ends with this step.

[0237] In this way, the position of the second image area within the first image area can be changed when obtaining image depth information. This enables the use of the second image sensor to obtain higher quality depth information without increasing the size and cost of the image sensor system.

[0238] It should be understood that Fig.14 The method is an example illustration of a method of controlling an image sensor system according to an embodiment of the present disclosure. Therefore, the present disclosure is not particularly limited to the method of controlling an image sensor system according to an embodiment of the present disclosure. Fig.14 In the embodiment of the present disclosure, the method of controlling the image sensor system may further include a plurality of additional method steps.

[0239] <Computer Program>

[0240] In addition, it should be understood that the method of the present disclosure can be performed on conventional hardware (such as the hardware previously described herein) that is appropriately adapted to be applicable through software instructions or by including or replacing dedicated hardware. Therefore, the required adaptation of existing parts of conventional equivalent devices can be implemented in the form of a computer program product, which includes processor-implemented instructions stored on a non-transitory machine-readable medium such as a floppy disk, an optical disk, a hard disk, a PROM, a RAM, a flash memory, or any combination of these or other storage media, or implemented in hardware as an ASIC (application-specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program can be transmitted via a data signal on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0241] <Terms>

[0242] Additionally, embodiments of the present disclosure are arranged by the following numbered clauses:

[0243] 1) An image sensor system for image capture, the image sensor system comprising:

[0244] a first image sensor comprising an array of pixels configured to detect light of at least a first wavelength range within a first image area;

[0245] a second image sensor, the second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and / or the second image area is smaller than the first image area, and each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon;

[0246] an actuator configured to actuate the second image sensor; and

[0247] A controller is configured to control the actuator to actuate the second image sensor so that the second image area is positioned within the portion of the first image area during image capture using the image sensor system.

[0248] 2) The image sensor system according to clause 1, wherein a frame rate of the image data from the second image sensor is higher than a frame rate of the image data from the first image sensor.

[0249] 3) The image sensor system according to clause 2, wherein the controller is configured to control the actuator to actuate the second image sensor so that the second image area is positioned a plurality of times in association with one image frame of the first image sensor.

[0250] 4) An image sensor system according to any preceding clause, wherein the controller is configured to control the actuator according to a predetermined pattern.

[0251] 5) An image sensor system according to any preceding clause, wherein the predetermined pattern is a pattern that covers, for the second image area, an entire portion of the first image area over several frames of image data of the second image sensor.

[0252] 6) An image sensor system according to any preceding clause, wherein the controller is configured to control the actuator based on image data from at least one of the first image sensor and / or the second image sensor.

[0253] 7) An image sensor system according to any preceding clause, wherein the controller is configured to analyze the first image data to identify a portion of the first image area where the image quality is below a predetermined threshold, and to control the actuator to actuate the second image sensor and position the second image area within the identified portion of the first image area.

[0254] 8) The image sensor system according to clause 6, wherein the image quality comprises at least one of: a brightness level in the first image data, a blur level in the first image data and / or a level of image detail in the first image data.

[0255] 9) An image sensor system according to any preceding clause, wherein the actuator is configured to actuate the second image sensor in: an x-direction parallel to the plane of the image sensor system, a y-direction orthogonal to the x-direction and parallel to the plane of the image sensor system, and a z-direction orthogonal to the plane of the image sensor system.

[0256] 10) The image sensor system according to any preceding clause, wherein the actuator is configured to actuate at least one of a pixel array of the second image sensor and / or an optical element of the second image sensor.

[0257] 11) An image sensor system according to any preceding clause, wherein the controller is configured to generate an output image using image data from the first image sensor and the second image sensor.

[0258] 12) The image sensor system of clause 11, wherein the controller is configured to enhance the image data from the first image sensor using the image data from the second image sensor when generating the output image.

[0259] 13) An image sensor system according to clause 11 or clause 12, wherein the controller is configured to use the trained model when generating the output image.

[0260] 14) The image sensor system according to any preceding clause, wherein the image sensor system further comprises a light emitter coupled to the second image sensor, the light emitter being configured to emit light in a second wavelength range.

[0261] 15) An image sensor system according to clause 14, wherein the light emitter comprises a vertical cavity surface emitting laser.

[0262] 16) Image sensor system according to clause 15, wherein the second wavelength range is a range of 650 nm to 1400 nm.

[0263] 17) The image sensor system of clause 14, wherein the controller is configured to calculate the depth within the second image region using image data from the second image sensor.

[0264] 18) An image sensor system according to any preceding clause, wherein the pixel array of the first image sensor and / or the pixel array of the second image sensor has a Bayer filter pattern.

[0265] 19) An image sensor system according to any preceding clause, wherein the first wavelength range is the same as the second wavelength range; or an image sensor system according to clause 1, wherein the first wavelength range is different from the second wavelength range.

[0266] 20) An image sensor system according to any preceding clause, wherein pixels of the second image sensor are single photon avalanche diodes, and / or wherein pixels of the first image sensor are complementary metal oxide semiconductor detectors.

[0267] 21) An information processing device comprising an image sensor system according to any preceding clause.

[0268] 22) An image sensor system according to any preceding clause, wherein the pixel size of the second image sensor is larger than the pixel size of the first image sensor, and wherein the amount of movement of the actuator by the controller is less than the pixel size of the second image sensor.

[0269] 23) A method of controlling an image sensor system for image capture, the image sensor system comprising: a first image sensor, the first image sensor comprising a pixel array configured to detect light of at least a first wavelength range within a first image area; a second image sensor, the second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and / or the second image area is smaller than the first image area, each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon; and an actuator configured to actuate the second image sensor; the method comprising:

[0270] The actuator is controlled to actuate the second image sensor such that the second image area is positioned within the portion of the first image area during image capture using the image sensor system.

[0271] 24) A computer program comprising instructions which, when implemented by a computer, cause the computer to perform a method of controlling an image sensor system for image capture, the image sensor system comprising: a first image sensor comprising a pixel array configured to detect light of at least a first wavelength range within a first image area; a second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and / or the second image area is smaller than the first image area, and each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon; the method comprising:

[0272] The actuator is controlled to actuate the second image sensor such that the second image area is positioned within the portion of the first image area during image capture using the image sensor system.

[0273] 25) A non-transitory computer-readable storage medium storing a computer program according to clause 24.

[0274] 26) An image sensor system for obtaining depth information, the image sensor system comprising:

[0275] a light emitter configured to emit light of at least a first wavelength range within a first image region;

[0276] an image sensor comprising an array of pixels configured to detect light of at least a first wavelength range within a second image area that is smaller than the first image area, each pixel in the array of pixels of the image sensor being configured to detect the arrival of a single photon;

[0277] an actuator configured to actuate the image sensor; and

[0278] A controller is configured to control the actuator to actuate the image sensor so that the second image area is positioned within a portion of the first image area when obtaining depth information using the image sensor system.

[0279] 27) A method of controlling an image sensor system for obtaining depth information, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range in a first image area; an image sensor comprising a pixel array configured to detect light of at least the first wavelength range in a second image area smaller than the first image area, each pixel in the pixel array of the image sensor being configured to detect the arrival of a single photon; and an actuator configured to actuate the image sensor; the method comprising:

[0280] The actuator is controlled to actuate the second image sensor such that the second image area is positioned within the portion of the first image area when obtaining depth information using the image sensor system.

[0281] 28) A computer program comprising instructions, which when implemented by a computer, causes the computer to perform a method of controlling an image sensor system for obtaining depth information, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range in a first image area; an image sensor comprising a pixel array configured to detect light of at least the first wavelength range in a second image area smaller than the first image area, each pixel in the pixel array of the image sensor being configured to detect the arrival of a single photon; and an actuator configured to actuate the image sensor; the method comprising:

[0282] The actuator is controlled to actuate the image sensor such that the second image area is positioned within a portion of the first image area when obtaining depth information using the image sensor system.

[0283] 29) A non-transitory computer-readable storage medium storing a computer program according to clause 28.

[0284] Additionally, it will be appreciated that numerous modifications and variations of the present disclosure are possible in light of the above teachings.Therefore, it will be appreciated that within the scope of the appended claims, the present disclosure may be practiced otherwise than as specifically described herein.

[0285] To the extent that embodiments of the present disclosure have been described as being implemented at least in part by a data processing device controlled by software, it will be understood that non-transitory machine-readable media (such as optical disks, magnetic disks, semiconductor memories, etc.) carrying such software are also considered to represent embodiments of the present disclosure.

[0286] It will be appreciated that for clarity, the above description has described embodiments with reference to different functional units, circuit systems and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuit systems and / or processors may be used without departing from the embodiments.

[0287] The described embodiments can be implemented in any suitable form, including hardware, software, firmware or any combination of these. The described embodiments can be optionally implemented at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment can be implemented physically, functionally and logically in any suitable manner. In fact, the function can be implemented in a single unit, in multiple units or as part of other functional units. Therefore, the disclosed embodiments can be implemented in a single unit, or can be physically and functionally distributed between different units, circuit systems and / or processors.

[0288] Although the present disclosure has been described in conjunction with some embodiments, the present disclosure is not intended to be limited to the specific form set forth herein. In addition, although features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined in any manner suitable for implementing the present technology.

Claims

1. An image sensor system for image capture, the image sensor system comprising: a first image sensor comprising an array of pixels configured to detect light of at least a first wavelength range within a first image area; a second image sensor, the second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and / or the second image area is smaller than the first image area, and each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon; an actuator configured to actuate the second image sensor; and A controller is configured to control the actuator to actuate the second image sensor so that the second image area is positioned within a portion of the first image area during image capture using the image sensor system.

2. The image sensor system according to claim 1, wherein: The frame rate of the image data from the second image sensor is higher than the frame rate of the image data from the first image sensor.

3. The image sensor system according to claim 2, wherein: The controller is configured to control the actuator to actuate the second image sensor so that the second image area is positioned a plurality of times in association with one image frame of the first image sensor.

4. The image sensor system according to claim 1, wherein: The controller is configured to control the actuator according to a predetermined pattern.

5. The image sensor system according to claim 4, wherein: The predetermined pattern is a pattern that covers an entire portion of the first image area over several frames of image data of the second image sensor for the second image area.

6. The image sensor system according to claim 1, wherein: The controller is configured to control the actuator based on image data from at least one of the first image sensor and / or the second image sensor.

7. The image sensor system according to claim 6, wherein: The controller is configured to analyze the first image data to identify a portion of the first image area having an image quality below a predetermined threshold, and control the actuator to actuate the second image sensor and position the second image area within the identified portion of the first image area.

8. The image sensor system according to claim 7, wherein: The image quality comprises at least one of: a brightness level in the first image data, a blur level in the first image data, and / or an image detail level in the first image data.

9. The image sensor system according to claim 1, wherein: The actuator is configured to actuate the second image sensor in the following directions: an x-direction parallel to the plane of the image sensor system, a y-direction orthogonal to the x-direction and parallel to the plane of the image sensor system, and a z-direction orthogonal to the plane of the image sensor system.

10. The image sensor system of claim 1, wherein: The actuator is configured to actuate at least one of a pixel array of the second image sensor and / or an optical element of the second image sensor.

11. The image sensor system of claim 1 , wherein: The controller is configured to generate an output image using image data from the first image sensor and the second image sensor.

12. The image sensor system according to claim 11, wherein: The controller is configured to enhance image data from the first image sensor using image data from the second image sensor in generating the output image.

13. The image sensor system according to claim 11, wherein: The controller is configured to use a trained model when generating the output image.

14. The image sensor system of claim 1, wherein: The image sensor system also includes a light emitter coupled to the second image sensor, the light emitter configured to emit light in the second wavelength range.

15. The image sensor system of claim 14, wherein: The controller is configured to calculate a depth within the second image area using image data from the second image sensor.

16. The image sensor system of claim 1, wherein: The pixel array of the first image sensor and / or the pixel array of the second image sensor has a Bayer filter pattern.

17. The image sensor system of claim 1, wherein: The first wavelength range is the same as the second wavelength range; or the image sensor system according to claim 1, wherein the first wavelength range is different from the second wavelength range.

18. The image sensor system of claim 1, wherein: The pixels of the second image sensor are single photon avalanche diodes, and / or wherein, The pixels of the first image sensor are complementary metal oxide semiconductor detectors; and / or wherein the pixel size of the second image sensor is larger than the pixel size of the first image sensor, and wherein the amount of movement of the actuator by the controller is less than the pixel size of the second image sensor.

19. A method for controlling an image sensor system for image capture, the image sensor system comprising: a first image sensor comprising an array of pixels configured to detect light of at least a first wavelength range within a first image area; a second image sensor, the second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and / or the second image area is smaller than the first image area, each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon; and an actuator configured to actuate the second image sensor; the method comprising: The actuator is controlled to actuate the second image sensor such that the second image area is positioned within a portion of the first image area during image capture using the image sensor system.

20. A computer program product comprising instructions which, when implemented by a computer, cause the computer to perform a method of controlling an image sensor system for image capture, the image sensor system comprising: a first image sensor comprising an array of pixels configured to detect light of at least a first wavelength range within a first image area; a second image sensor, the second image sensor comprising a pixel array configured to detect light of at least a second wavelength range within a second image area, wherein the number of pixels of the pixel array of the second image sensor is smaller than the number of pixels of the pixel array of the first image sensor and / or the second image area is smaller than the first image area, and each pixel in the pixel array of the second image sensor is configured to detect the arrival of a single photon; the method comprising: An actuator is controlled to actuate the second image sensor such that the second image area is positioned within a portion of the first image area during image capture using the image sensor system.

21. A non-transitory computer-readable storage medium storing the computer program product according to claim 20.

22. An image sensor system for obtaining depth information, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range within a first image region; an image sensor comprising an array of pixels configured to detect light of at least the first wavelength range within a second image area smaller than the first image area, each pixel in the array of pixels of the image sensor being configured to detect the arrival of a single photon; an actuator configured to actuate the image sensor; as well as A controller is configured to control the actuator to actuate the image sensor so that the second image area is positioned within a portion of the first image area when obtaining depth information using the image sensor system.

23. A control method for an image sensor system for obtaining depth information, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range within a first image region; an image sensor comprising an array of pixels configured to detect light of at least the first wavelength range within a second image area smaller than the first image area, each pixel in the array of pixels of the image sensor being configured to detect the arrival of a single photon; and an actuator configured to actuate the image sensor; the method comprising: The actuator is controlled to actuate the image sensor such that the second image area is positioned within a portion of the first image area when obtaining depth information using the image sensor system.

24. A computer program product comprising instructions which, when implemented by a computer, cause the computer to perform a method of controlling an image sensor system for obtaining depth information, the image sensor system comprising: a light emitter configured to emit light of at least a first wavelength range within a first image region; an image sensor comprising an array of pixels configured to detect light of at least the first wavelength range within a second image area smaller than the first image area, each pixel in the array of pixels of the image sensor being configured to detect the arrival of a single photon; and an actuator configured to actuate the image sensor; the method comprising: The actuator is controlled to actuate the image sensor such that the second image area is positioned within a portion of the first image area when obtaining depth information using the image sensor system.

25. A non-transitory computer-readable storage medium storing the computer program product according to claim 24.

Images