Systems, methods and media for generating digital images using low bit depth image sensor data - Patents.com

Abstract

According to some embodiments, a system, method, and media are provided for generating a digital image using low bit-depth image sensor data. In some embodiments, the system includes an image sensor and a processor, the processor being programmed to receive a sequence of low bit-depth frames from the image sensor, provide the low bit-depth image information to a trained machine learning model including a 3D convolutional layer, a 2D convolutional LSTM layer, a concatenation layer configured to generate a tensor including an output of the 2D convolutional LSTM layer and the low bit-depth image information, and a 2D convolutional layer configured to generate an output based on the tensor, and generate a high bit-depth image of the scene based on the output of the 2D convolutional layer.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is related to and claims priority to U.S. Provisional Patent Application No. 17 / 687,390, filed March 4, 2022, which is incorporated by reference herein for all purposes.

[0002] (STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT) This invention was made with Government support under award 1943149 from the National Science Foundation. The Government has certain rights in this invention. [Background technology]

[0003] In general, image sensors capable of detecting the occurrence of individual photons, which may also be referred to as quantum image sensors, can generate data with relatively low read noise, relatively high temporal granularity, and relatively high dynamic range. However, raw binary frames from quantum image sensors contain high shot noise due to their short duration.

[0004] Therefore, new systems, methods, and media for generating digital images using low bit depth image sensor data are desired. Summary of the Invention

[0005] In accordance with some embodiments of the disclosed subject matter, systems, methods, and media are provided for generating digital images using low bit-depth image sensor data.

[0006] In accordance with some embodiments of the disclosed subject matter, a system for generating a digital image is provided, the system including an image sensor configured to generate low bit-depth frames, and at least one processor programmed to receive a sequence of low bit-depth frames from the image sensor and provide low bit-depth image information based on the sequence of low bit-depth frames to a trained machine learning model, the trained machine learning model including a 3D convolutional layer, a 2D convolutional long-short-term memory (LSTM) layer configured to receive an output of the 3D convolutional layer, a concatenation layer configured to generate a tensor including a concatenation of an output of the 2D convolutional layer LSTM layer and the low bit-depth image information, and a 2D convolutional layer configured to generate an output based on the tensor generated by the concatenation layer and configured to generate a high bit-depth image of the scene based on the output of the 2D convolutional layer.

[0007] In some embodiments, the image sensor includes a plurality of single photon avalanche diodes.

[0008] In some embodiments, the series of low bit depth frames comprises a series of binary frames.

[0009] In some embodiments, pixels in a low bit depth frame are represented using 4 bits or less.

[0010] In some embodiments, a pixel in a high bit depth image is represented using at least 8 bits.

[0011] In some embodiments, the 2D convolutional LSTM layer is a bidirectional 2D convolutional LSTM layer.

[0012] In some embodiments, the trained machine learning model further includes three LSTM blocks, each including two 3D convolutional layers, a bidirectional 2D convolutional LSTM layer configured to receive the output of the second 3D convolutional layer, and a concatenation layer configured to concatenate inputs to the LSTM blocks and outputs of the bidirectional 2D convolutional LSTM layer, wherein the 3D convolutional layer, the 2D convolutional LSTM layer, and the concatenation layer are included in a third LSTM block.

[0013] In some embodiments, the trained machine learning model includes multiple input channels, each corresponding to a color channel.

[0014] In accordance with some embodiments of the disclosed subject matter, a method for generating a range digital image is provided, the method including receiving a sequence of low bit-depth frames from an image sensor, providing the low bit-depth image information based on the sequence of low bit-depth frames to a trained machine learning model including a 3D convolutional layer, a 2D convolutional long short-term memory layer configured to receive an output of the 3D convolutional layer, a concatenation layer configured to generate a tensor including a concatenation of an output of the 2D convolutional LSTM layer and the low bit-depth image information, and a 2D convolutional layer configured to generate an output based on the tensor generated by the concatenation layer, and generating a high bit-depth image of the scene based on the output of the 2D convolutional layer.

[0015] According to some embodiments of the disclosed subject matter, a non-transitory computer-readable medium is provided that includes computer-executable instructions that, when executed by a processor, cause the processor to perform a method for generating a range digital image, the method including: receiving a series of low bit-depth frames from an image sensor; providing the low bit-depth image information based on the series of low bit-depth frames to a trained machine learning model including a 3D convolutional layer, a 2D convolutional long-short-term memory (LSTM) layer configured to receive an output of the 3D convolutional layer, a concatenation layer configured to generate a tensor including a concatenation of an output of the 2D convolutional LSTM layer and the low bit-depth image information, and a 2D convolutional layer configured to generate an output based on the tensor generated by the concatenation layer; and generating a high bit-depth image of the scene based on the output of the 2D convolutional layer.

[0016] The various objects, features and advantages of the disclosed subject matter can be more fully appreciated by reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements and in which: [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 shows an example of a low noise, high bit depth frame of a scene that can be generated using a high noise, low bit depth frame and a low bit depth frame of a scene, in accordance with some embodiments of the disclosed subject matter. [Diagram 2] FIG. 2 illustrates an example of a system for generating a digital image using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter. [Diagram 3] FIG. 3 illustrates an example of hardware that can be used to implement the image data source, computing device, and server illustrated in FIG. 2, according to some embodiments of the disclosed subject matter. [Figure 4]FIG. 4 illustrates an example of a convolutional neural network topology that can be used to implement a mechanism for generating a digital image using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter. [Diagram 5] FIG. 5 illustrates another example topology of a convolutional neural network that can be used to implement a mechanism for generating a digital image using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter. [Figure 6] FIG. 6 illustrates an example of a process for generating a digital image using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter. [Figure 7] FIG. 7 shows an example of low bit-depth frames of a scene generated from still images with simulated motion, predicted low-noise high bit-depth frames of the scene generated from a stream of low bit-depth frames using the mechanisms described herein, and a ground truth image of the scene. [Figure 8] FIG. 8 shows an example of low bit-depth frames based on various multiple binary frames generated from still images with simulated motion, predicted low-noise high bit-depth frames of a scene generated from a stream of low bit-depth frames using the mechanisms described in this specification, and a ground truth image of the scene. [Figure 9] FIG. 9 shows an example of low bit-depth frames of a scene generated from video data, predicted low-noise high bit-depth frames of the scene generated from a stream of low bit-depth frames using the mechanisms described herein, and a ground truth image of the scene. [Figure 10] FIG. 10 shows a high bit-depth frame of a scene generated from a setup binary frame using another technique, a predicted low-noise high bit-depth frame of the scene generated from a stream of low bit-depth frames using the mechanisms described in this specification, and an example of a ground truth image of the scene. [Figure 11] FIG. 11 shows an example of the results of an object detection operation performed on low bit-depth frames of a scene generated from video data, predicted low-noise high bit-depth frames of the scene generated from a stream of low bit-depth frames using the mechanisms described in this specification, and a ground truth image of the scene. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] According to various embodiments, mechanisms (which may include, for example, systems, methods, and media) are provided for generating digital images using low bit-depth image sensor data.

[0019] In some embodiments, the mechanisms described herein can be used to generate relatively low noise images at a relatively high bit depth (e.g., at least 8 bits per pixel per color channel) using relatively high noise frames at a relatively low bit depth (e.g., 1 bit per pixel per color channel, 2 bits per pixel per color channel, 3 bits per pixel per color channel, 4 bits per pixel per color channel, or less than 4 bits per pixel per color channel). In some embodiments, the mechanisms described herein can generate high bit depth images at a rate comparable to the rate at which frames (or a relatively short series of frames) are output by a quantum image sensor.

[0020] For example, single photon avalanche diodes (SPADs) are a class of single photon image sensors that can be operated at very high frame rates (e.g., on the order of 100,000 frames per second). Conventional image sensors (e.g., CMOS image sensors, CCD image sensors) generally return an integer value from each pixel that is proportional to the brightness of the scene at that pixel. When set in frame readout mode, image sensors implemented with SPADs (and / or other QIS technologies) return mostly binary values. For example, a pixel implemented with SPADs can output a 1 if at least one photon is detected, and a 0 otherwise. SPADs offer several advantages over conventional sensors, including low read noise, high temporal granularity, and high dynamic range. However, raw binary frames from SPADs (or any quantum image sensor) contain high shot noise due to their short duration.

[0021] In some embodiments, the mechanisms described herein can be implemented to recover relatively low noise and high bit depth images from the low bit depth image data output by the QIS. For example, a noisy frame {b 0、 b 1、 … 、 b t}, the mechanism described herein can find frames {f0, f1, ..., f t} low noise frame sequence corresponding to the true sequence TIFF2025507996000002.tif7150. In some embodiments, the mechanisms described herein can leverage the high frame rate of the input to produce an output at a high frame rate (e.g., a uniform high frame rate) while merging intensity information from multiple frames to reduce noise.

[0022] In some embodiments, the mechanisms described herein use information from before and after the frame to determine the frame TIFF2025507996000003.tif7150 can be evaluated (e.g., from b≦i and from b≧i, if such future information is available). Additionally or alternatively, in some embodiments, the mechanisms described herein can provide real-time or near real-time frame You can evaluate TIFF2025507996000004.tif7150 (for example, b i Evaluation immediately after arrival TIFF2025507996000005.tif7150, b t Evaluation just before the arrival of TIFF2025507996000006.tif7150, the mechanism described herein is frame {b 0、 b 1、 … 、 b i} to The use of any future information can also be excluded, as can TIFF2025507996000007.tif7150.

[0023] In some embodiments, the mechanisms described herein can use a recurrent convolutional neural network (RCNN) to compensate for motion during high-noise frames (e.g., output from a QIS). Convolutional neural networks can be used for many computer vision tasks and are effective for processing spatially localized structures in images. Recurrent convolutional neural networks can be used for temporal sequence processing and can process and generate sequences of any length. In some embodiments, the mechanisms described herein can leverage RCNN to process spatially localized structures in a series of high-noise frames to generate a series of low-noise frames.

[0024] In some embodiments, the mechanisms described herein can utilize one or more convolutional long-short-term memory (LSTM) layers to recover relatively low noise and high depth images from low bit depth image data output by the QIS. LSTMs can be configured to form long-term dependencies by adaptively updating their internal states. Convolutional LSTMs can replace dense operations in normal LSTMs with convolutional operations. As described below, models implemented using one or more bidirectional LSTMs that allow information to flow both forwards and backwards in time achieved the best performance. To utilize bidirectional LSTMs, all frames to be analyzed are captured and stored before processing. Alternatively, the model can be implemented using one or more unidirectional LSTMs (processing frames only in the forward direction of time). For example, such models can be implemented for real-time applications and / or for use with devices with low memory and / or processing resources.

[0025] In some embodiments, the mechanisms described herein can be implemented in a variety of different implementations with different computational costs. For example, in applications with limited resources (e.g., memory and / or computing resources), the number of layers and the size of each layer can be reduced. In general, a trade-off space can be expected between smaller, more efficient networks and larger, high-fidelity networks.

[0026] In some embodiments, the mechanisms described herein can be implemented to take in new frames in real time and output high bit depth predicted frames at a similar rate. For example, after an RNN implemented as per the mechanisms described herein uses a new frame to update its internal state, the frame can be discarded (note that this is only the case for unidirectional RNNs). Other techniques (e.g., Quantaburst photography) involve capturing and storing all frames that are to be used prior to performing the analysis.

[0027] In some embodiments, the mechanisms described herein can be implemented on a graphics processing unit (GPU), with high inherent parallelism facilitating efficient execution. For example, preliminary results show computation times several orders of magnitude shorter compared to quantum burst photography.

[0028] In some embodiments, the mechanisms described herein can be used to implement models (e.g., convolutional RNN models) that can be trained end-to-end with one or more downstream neural networks configured to perform image processing and / or machine vision tasks. For example, the mechanisms described herein can be used to implement a convolutional RNN that can be trained simultaneously with a CNN configured to perform object detection. Such end-to-end training can facilitate the convolutional RNN and downstream networks to take full advantage of their parameters relative to one another, facilitating higher quality output with noisier inputs.

[0029] In some embodiments, convolutional RNNs implemented according to the mechanisms described herein can learn statistical priors over video structure, allowing such convolutional RNNs to significantly outperform traditional, untrained algorithms on videos with extremely high noise.

[0030] FIG. 1 illustrates a low-noise, high-bit-depth frame of a scene that can be generated using a high-noise, low-bit-depth frame and a low-bit-depth frame of a scene, according to some embodiments of the disclosed subject matter. As shown in FIG. 1, the binary frames that can be output by a quanta image sensor are generally very noisy. To obtain a usable non-binary image, information from multiple consecutive binary frames can be merged. Binary frame merging can be considered a special case of a more general problem: motion image processing and high-level vision. When a scene contains motion, an imaging system can either shorten the exposure time during exposure to reduce the amount of captured motion (which leads to more noise) or increase the exposure time to process the noise (which leads to a blurred image). In either case, the result is a low-quality image. If this low-quality image is then fed to a downstream model (e.g., a machine learning system for object detection), the model will suffer from reduced accuracy.

[0031] FIG. 2 illustrates an example system 200 for generating digital images using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter. As illustrated in FIG. 2, a computing device 210 can receive images from image data source(s) 202. In some embodiments, the computing device 210 can execute at least a portion of an image processing system 204 to perform image processing tasks, such as generating a high bit-depth image from a low bit-depth image using a neural network, training a neural network to generate a high bit-depth image from a low bit-depth image, etc. Additionally, in some embodiments, the computing device 210 can execute at least a portion of a machine vision system (not shown) to perform machine vision tasks, such as image classification, object detection, image segmentation, object tracking, and / or any other suitable computer vision task. For example, the machine vision system can receive one or more images (e.g., high bit-depth images, etc.) generated by the image processing system 204 and can perform machine vision task(s) based on the image(s) received from the image processing system 204.

[0032] Additionally or alternatively, in some embodiments, computing device 210 can communicate data received from image data source 202 to server 220 via communications network 208, which can then perform at least a portion of image processing system 204 and / or at least a portion of the machine vision system. In such embodiments, server 220 can return information to computing device 210 (and / or any other suitable computing device) indicative of the output of image processing tasks performed by image processing system 204 and / or computer vision tasks performed by the computer vision system. In some embodiments, image processing system 204 can perform portions of one or more processes 600 described below with respect to FIG.

[0033] In some embodiments, computing device 210 and / or server 220 can be any suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a computing device integrated into a vehicle (e.g., an autonomous vehicle), a camera, a robot, a virtual machine running on a physical computing device, etc.

[0034] In some embodiments, image data source 202 can be any suitable source of low bit-depth image data (e.g., implemented with single photon pixels, implemented with a combination of conventional and single photon pixels) and / or other data that can be used to generate high bit-depth image data (e.g., depicting a scene in the physical environment of image data source 202) as described herein. For example, image data source 202 can be implemented using one or more digital cameras that generate and / or output image data indicative of the arrival time of single photons. In a more specific example, image data source 202 can include an imaging device (e.g., using an avalanche photodiode) configured to detect the arrival of individual photons, such as the imaging device described in U.S. Patent Application Serial No. 16 / 844,899, filed April 9, 2020, and entitled "Systems, Methods, and Media for High Dynamic Range Quantum Burst Imaging." As yet another specific example, the image data source 202 can include an imaging device configured to detect the arrival of individual photons (e.g., a Jot-based detector), such as the imaging devices described in Fossum et al., “Quanta Image Sensor: Every Photon Counts,” Sensor (2016).

[0035] In some embodiments, the image data source 202 can be local to the computing device 210. For example, the image data source 202 can be incorporated into the computing device 210 (e.g., the computing device 210 can be configured as part of the device for capturing, storing, and / or processing image data). As another example, the image data source 202 can be connected to the computing device 210 by a cable, a direct wireless link, etc. Additionally or alternatively, in some embodiments, the image data source 202 can be located locally and / or remotely from the computing device 210 and communicate image data (e.g., single photon sensor image data, etc.) to the computing device 210 (and / or server 220) via a communications network (e.g., communications network 208).

[0036] In some embodiments, communication network 208 can be any suitable communication network or combination of communication networks. For example, communication network 208 can include a WiFi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc., conforming to any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, NR, etc.), a wired network, etc. In some embodiments, communication network 208 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university Internet), any other suitable network type, or any suitable combination of networks. Each of the communication links illustrated in FIG. 2 can be any suitable communication link or combination of communication links, such as a wired link, a fiber optic link, a WiFi link, a Bluetooth link, a cellular link, etc.

[0037] 3 illustrates an example of hardware that may be used to implement the image data source 202, the computing device 210, and / or the server 220 illustrated in FIG. 2, according to some embodiments of the disclosed subject matter. As illustrated in FIG. 3, in some embodiments, the computing device 210 may include a processor 302, a display 304, one or more inputs 306, one or more communication systems 308, and / or a memory 310. In some embodiments, the processor 302 may be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a microcontroller (MCU), etc. In some embodiments, the display 304 may include any suitable display device, such as a computer monitor, a touch screen, a television, an infotainment screen, etc. In some embodiments, the input 306 may include any suitable input device and / or sensor that may be used to receive user input, such as a keyboard, a mouse, a touch screen, a microphone, etc.

[0038] In some embodiments, communications system 308 may include any suitable hardware, firmware, and / or software for communicating information over communications network 208 and / or any other suitable communications network. For example, communications system 308 may include one or more transceivers, one or more communications chips and / or chipsets, etc. In more particular examples, communications system 308 may include hardware, firmware, and / or software that may be used to establish a WiFi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0039] In some embodiments, memory 310 may include any suitable storage device or devices that may be used to store image data, instructions, values, etc. that may be used by processor 302, for example, to perform image processing tasks, perform machine vision tasks, provide content using display 304, communicate with server 220 via communication system(s) 208, etc. Memory 310 may include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 310 may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), one or more flash devices, one or more hard disks, one or more solid-state drives, one or more optical drives, etc. In some embodiments, memory 310 may further encode computer programs to control the operation of computing device 210. For example, in such an embodiment, the processor 302 may at least partially execute a computer program to perform one or more image processing tasks described herein and / or to perform one or more machine vision tasks based on output generated by the image processing tasks described herein, may present content (e.g., an image, information about objects included in the image data, information about distances to one or more points in a scene, etc.), may receive information and / or content from the image data source 202, may transmit information to the image data source 202, may receive information and / or content from the server 220, may transmit information to the server 220, etc. As another example, the processor 302 may at least partially execute a computer program for implementing the image processing system 204 and / or a machine vision system. As yet another example, the processor 302 may at least partially execute a process 600 described below with respect to FIG. 6.

[0040] In some embodiments, server 220 may include a processor 312, a display 314, one or more inputs 316, one or more communication systems 318, and / or memory 320. In some embodiments, processor 312 may be any suitable hardware processor or combination of processors, such as a CPU, GPU, ASIC, FPGA, DSP, MCU, etc. In some embodiments, display 314 may include any suitable display device, such as a computer monitor, touch screen, television, etc. In some embodiments, input 316 may include any suitable input device and / or sensor that may be used to receive user input, such as a keyboard, mouse, touch screen, microphone, etc.

[0041] In some embodiments, communications system 318 may include any suitable hardware, firmware, and / or software for communicating information over communications network 208 and / or any other suitable communications network. For example, communications system 318 may include one or more transceivers, one or more communications chips and / or chipsets, etc. In more particular examples, communications system 318 may include hardware, firmware, and / or software that may be used to establish a WiFi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0042] In some embodiments, memory 320 may include any suitable storage device or devices that may be used to store instructions, values, etc. that may be used by processor 312, for example, to provide content using display 314, to communicate with one or more computing devices 210, to communicate with one or more image data sources 202, etc. Memory 320 may include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 320 may include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory 320 may encode server programs for controlling the operation of server 220 as well. For example, in some embodiments, the processor 312 may at least partially execute a server program to perform one or more image processing tasks described herein and / or perform one or more machine vision tasks based on output generated by the image processing tasks described herein, may provide content (e.g., an image, information about objects contained in the image data, information about distances to one or more points in the scene, etc.), may receive information and / or content from the image data source 202, may transmit information to the image data source 202, may receive information and / or content from the computing device 210, may transmit information to the computing device 210, etc. As another example, the processor 312 may at least partially execute a server program for implementing the image processing system 204 and / or a machine vision system. As yet another example, the processor 312 may at least partially execute a process 600 described below with respect to FIG. 6.

[0043] As shown, the image data source 202 may include an image sensor 322 (e.g., an area sensor including an array of single photon detectors, such as a SPAD array or an array of Jots, e.g., as described in U.S. patent application Ser. No. 16 / 844,899), optics 324 (e.g., may include one or more lenses, one or more filters, attenuation elements such as a lens aperture, and / or any other suitable optical elements such as a beam splitter, etc.), a processor 326 for controlling the operation of the image data source 202, which may include any suitable hardware processor (which may be a CPU, GPU, FPGA, ASIC, DSP, MCU, etc.) or combination of hardware processors, and input device(s) 328 for accepting input from a user and / or from the environment (e.g., shutters, etc.). The image data source 202 may include a display 330 (e.g., a touch screen, a liquid crystal display, a light emitting diode display, etc.) for providing information (e.g., images, a user interface, etc.) for consumption by a user, a memory 332, a signal generator 334 for generating one or more signals for controlling operation of the image sensor 322, and a communication system or systems 336 for facilitating communication between the image data source 202 and other devices, such as a smartphone, a wearable computer, a tablet computer, a laptop computer, a personal computer, a server, an embedded computer (e.g., for controlling an autonomous automobile, a robot, etc.), etc., via a communication link. In some embodiments, the memory 332 may store image data and / or any other suitable data. The memory 332 may include a storage device (e.g., RAM, ROM, EEPROM, one or more flash devices, one or more hard disks, one or more solid state drives, one or more optical drives, etc.) for storing computer programs for controlling the processor 326.In some embodiments, the memory 332 may include instructions for causing the processor 326 to perform processes associated with the mechanisms described herein, such as the process 600 described below with respect to FIG.

[0044] In some embodiments, the image sensor 322 can include an image sensor implemented at least in part using an array of SPAD detectors (sometimes referred to as Geiger-mode avalanche photodiodes) and / or one or more other detectors configured to detect the arrival time of a solid photon (e.g., a jot). In some embodiments, one or more elements of the single-photon image sensor 322 can be configured to generate data indicative of the arrival time of a photon from a scene via optics 324. For example, in some embodiments, the image sensor 322 can be an array of multiple SPAD detectors. As yet another example, the image sensor 322 can be a hybrid array including a SPAD detector and one or more conventional photodetectors (e.g., CMOS-based pixels).

[0045] In some embodiments, the image data source 202 can include additional optics. For example, while the optics 324 are shown as a single lens, the optics 324 can be implemented as a compound lens or a combination of lenses. Note that while the mechanisms described herein are generally described as using a SPAD-based detector, this is only an example of a single photon detector. As described above, other single photon detectors, such as Jot-based image sensors, can be used.

[0046] In some embodiments, signal generator 334 can be one or more signal generators that can generate signals to control image sensor 322. For example, in some embodiments, signal generator 334 can provide signals to enable and / or disable one or more pixels of image sensor 322 (e.g., by controlling gating signals of SPADs used to implement the pixels). As another example, signal generator 334 can provide signals to control readout of image signals from image sensor 322 (e.g., to memory 332, processor 326, cache memory associated with image sensor 322, etc.).

[0047] In some embodiments, the image data source 202 can communicate with remote devices over a network using the communication system(s) 336 and communication links. Additionally or alternatively, the image data source 202 can be incorporated as part of and / or integrated as part of another device (e.g., computing device 210), such as a smartphone, tablet computer, laptop, autonomous automobile, robot, etc. Portions of the image data source 202 can be shared with the device in which the image data source 202 is integrated. For example, if the image data source 202 is integrated with an autonomous automobile, the processor 326 can be the processor of the autonomous automobile and can be used to control the operation of the image data source 202.

[0048] In some embodiments, display 330 may be used to display images and / or video generated by image data source 202 and / or by another device (e.g., computing device 210, server 220, etc.), display a user interface, etc. In some embodiments, display 330 may be implemented using any suitable device or combination of devices and may include one or more inputs, such as a touch screen.

[0049] FIG. 4 illustrates an example topology 400 of a machine learning model that can be used to implement a mechanism for generating a digital image using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter. In some embodiments, a machine learning model based on topology 400 can receive input generated by a quanta image sensor (e.g., implemented with an array of SPADs). Such input can be formatted in various ways. For example, the input can be received as a sequence (e.g., a stream) of binary low bit-depth frames (e.g., 1-bit frames). As another example, the input can be received as a sequence of non-binary low bit-depth frames (e.g., 2-bit frames, 3-bit frames, etc.). In a more specific example, a multi-bit (e.g., 2-bit, 3-bit, 4-bit) frame includes a value corresponding to a total number of multiple binary frames at each pixel. In such an example, the 2-bit frames can total up to 4 binary frames, the 3-bit frames can total up to 8 binary frames, etc. As yet another example, the input can be received as a sequence of averaged low bit-depth frames. In such an example, data from each pixel location in a frame can be averaged over multiple frames (e.g., two frames, four frames, eight frames, or any other suitable number of frames). In one particular example, the average of eight frames can be calculated by determining the total number of pixel detections at each pixel location (for 1-bit frames, values ​​from 0 to 8) and dividing by 8. Note that multiple frames can include information from multiple binary frames, and the averaged low bit depth frame can be generated from one or more multi-bit frames.For example, data from four binary frames can be used to generate a 2-bit frame, and the two 2-bit frames can be used to generate an average of eight frames by determining the total number of pixel values ​​in the two frames and dividing by eight.

[0050] In some embodiments, topology 400 can utilize data from multiple color channels and output high bit depth color image data. For example, topology 400 can be modified to include multiple input channels (e.g., three image channels corresponding to red, green, and blue color data, or more than three channels for color image data including more than three color channels) and multiple output channels (e.g., corresponding to colors associated with the input data).

[0051] In some embodiments, topology 400 can include one or more 3D convolutional layers. Such 3D convolutions can generate features that reflect spatiotemporal patterns in the input. For example, topology 400 can include a single 3D convolutional layer prior to a convolutional LSTM layer. As another example, topology 400 can include multiple 3D convolutional layers prior to a convolutional LSTM layer. In some embodiments, each 3D convolutional layer can use any suitable kernel. For example, a 3D convolutional layer can use a 3×3×3 kernel. As another example, a 3D convolutional layer can use a 1×1×1 kernel, a 5×5×5 kernel, a 7×7×7 kernel, or any other suitable kernel. In some embodiments, each 3D convolutional layer can include any suitable number of output channels, resulting in a corresponding number of kernels being trained. For example, each 3D convolutional layer can generate 128 output channels using 128 kernels. As another example, each 3D convolutional layer may generate 128 output channels or more (e.g., 256 output channels, 512 output channels, 1024 output channels, etc.). In some embodiments, each 3D convolutional layer may utilize any suitable activation function. For example, a 3D convolutional layer may utilize rectified linear unit (ReLU) activation with any suitable leakage rate (e.g., a leakage rate of 0.3, or any other suitable leakage rate).

[0052] In some embodiments, one or more convolutional LSTM layers can be implemented using unidirectional convolutional LSTMs. Additionally or alternatively, one or more convolutional LSTM layers can be implemented using a bidirectional convolutional LSTM. For example, a unidirectional convolutional LSTM layer can utilize a single convolutional LSTM cell and can sequentially receive the outputs of a 3D convolutional layer as inputs. As another example, a bidirectional convolutional LSTM layer can utilize multiple convolutional LSTM cells. In such an example, an LSTM cell can sequentially receive the outputs of a 3D convolutional layer as inputs in the order in which the outputs are generated, and another LSTM cell can sequentially receive the outputs of a 3D convolutional layer as inputs in the reverse order from which the outputs are generated.

[0053] In some embodiments, topology 400 can include one or more skip connections. For example, topology 400 can include a skip connection between the input and the output of the LSTM layer. In such examples, the input to the model and the output of the LSTM layer can be concatenated (e.g., via a concatenation layer) using any suitable technique or combination of techniques. In some embodiments, the concatenation can be performed along a channel axis. For example, an input corresponding to a frame at time t can be concatenated with an output corresponding to a frame at time t. For example, if the input has 128 channels and the output has 128 channels, the concatenated output can have 256 channels. As another example, topology 400 can include a skip connection between the output of an LSTM layer and the output of a next LSTM layer. In such examples, the input to the model, the output of the first LSTM layer, and the output of the next LSTM layer can be concatenated (e.g., via a concatenation layer) using any suitable technique or combination of techniques. As yet another example, using skip connections and concatenation, a block l (e.g., including one or more 3D convolutional layers and a convolutional LSTM layer) can receive the input to block l-1 and also the inputs to all previous blocks (e.g., including the inputs of the first model).

[0054] In some embodiments, a model implemented according to the mechanisms described herein (e.g., using one or more bidirectional LSTM layers, using only a unidirectional LSTM layer) may: When evaluating TIFF2025507996000008.tif7150, frame f i information collected before f (e.g., using frames with index b≦i) and iFor example, both bidirectional LSTMs and 3D convolutions (which can exploit the receptive field to extend a fixed distance both into the past and future in the stream of input frames) can leverage information from before and after a particular frame. As another example, in some applications where the output is generated before the entire input sequence is received as input (e.g., real-time applications), convolutional LSTMs can be implemented as unidirectional LSTMs and 3D convolution ... i Receiving and Evaluation A fixed delay can be introduced between producing TIFF2025507996000009.tif7150.

[0055] In some embodiments, topology 400 may include at least one 2D convolutional layer capable of generating an output frame. For example, the 2D convolutional layer may receive a concatenated output (e.g., including at least the output of the convolutional LSTM layer and the input data). In some embodiments, the 2D convolutional layer may use any suitable kernel. For example, the 2D convolutional layer may use a 1×1 kernel. The 2D convolutional layer may operate independently on the output associated with each time step (e.g., for a particular input frame b i In some embodiments, a two-dimensional convolutional layer may include any suitable number of output channels, which may correspond to output frames (e.g., TIFF2025507996000010.tif7150). For example, a 2D convolutional layer may have a single output channel. In some embodiments, the output of a 2D convolutional layer may be in a particular range (e.g., from 0 to 1). For example, outputs outside a particular range may be restricted to a particular range (e.g., values ​​less than zero may be set to zero, and values ​​equal to or greater than one may be set to one). As another example, an activation function (e.g., a sigmoid activation function) may be used to generate outputs in a particular range. In some embodiments, the output of a 2D convolutional layer may be mapped to a high bit depth image (e.g., using any suitable tone mapping technique). For example, values ​​may be converted to integer values ​​in a range inclusive of 0 and 255. In such examples, the mapping may be linear or nonlinear.

[0056] In some embodiments, a model implemented as per the mechanisms described herein can be trained using any suitable technique or combination of techniques (e.g., based on topology 400, based on topology 500 described below with respect to FIG. 5, etc.) and / or using any suitable training data. For example, the model can be trained using data including synthetic motion and / or data including video with real-world motion. In a more specific example, some results described below were generated using a model trained using two datasets: a synthetic motion dataset and a real video dataset. Both datasets encompass full-depth frames captured by conventional cameras and were used as ground truth during training. In some embodiments, the frames used as input for training can be generated by randomly applying Poisson binarization to the full-depth frames.

[0057] In some embodiments, frames with synthetic motion can be generated using any suitable technique or combination of techniques. For example, a computing device (e.g., a computing device performing process 600) can extract a moving sequence of bounding boxes from a still image. The computing device can use a random walk in each coordinate (e.g., x, y, rotation, and scale) of the bounding box. The steps of the random walk can be calculated using the relationship:

number

[0058] In some embodiments, frames with real motion can be generated using any suitable technique or combination of techniques. For example, a computing device (e.g., a computing device performing process 600) can extract low bit depth frames from a high bit depth video. For example, the low bit depth frames were generated from real videos from the Need for Speed ​​(NFS) dataset and used during training of a model implemented as described herein based on the topology 500 described below with respect to FIG. 5. The NFS dataset includes 100 videos with a total of 380,000 frames captured at 240 frames per second (FPS). The frames have a resolution of 1280×720 and were reduced in size to 320×180 pixels to speed up training and reduce the amount of motion between frames (measured in pixels per frame). Bounding boxes of objects associated with the videos were not used.

[0059] In some embodiments, a computing device (e.g., a computing device performing process 600) can generate frames simulating frames captured by a low bit-depth sensor from still image data and / or from frames of video data using any suitable technique or combination of techniques. For example, the computing device can convert still image data or frames of video to grayscale and reduce the bit-width of the image (e.g., generating a binarized image). In a more particular example, the computing device can generate the binarized image using Poisson statistics. In such an example, pixels with intensity i ∈ [0,1] can be binarized using the relationship:

number

[0060] In many practical applications, the motion can be much less than one pixel per binary frame. Thus, performing reconstruction at the frame rate at which low bit depth frames (e.g., binary frames) are generated by the image sensor can often be computationally wasteful, leading to very high frame rates (e.g., 100,000 FPS or greater). In some embodiments, multiple frames (2, 3, 4, 5, 6, 7, 8, etc.) can be averaged, and the averaged frames can be provided as input to a model (e.g., to train and / or train a model). For example, the input to the model can be averaged over an 8-frame block. In such an example, the motion rate in the synthetic data can be adjusted to give approximately one pixel of motion per 8-frame block. In a practical video dataset, each frame can be repeated 8 times before binarization and 8-frame averaging. This 8-frame repetition makes the amount of motion between binary frames more manageable, and can reduce the number of disk reads per sequence from 512 to 64.

[0061] FIG. 5 illustrates another example topology 500 of a convolutional neural network that can be used to implement a mechanism for generating digital images using low bit depth image sensor data, according to some embodiments of the disclosed subject matter. As illustrated in FIG. 5, the topology 500 can include three blocks, each including two 3D convolutional layers and a bidirectional convolutional LSTM layer, each of which can be followed by concatenation. The 3D convolutional layer can use a 3×3×3 kernel with 128 output channels, and the bidirectional convolutional LSTM layer can include two convolutional LSTM cells, each of which can use a 5×5 kernel and each of which can have 64 output channels. In some embodiments, the topology 500 can include a 2D convolutional layer that can generate an output frame. The 2D convolutional layer can use one 1×1 kernel with one output channel. In some embodiments, each 3D convolutional layer can utilize rectified linear unit (ReLU) activation with a leakage rate of 0.3.

[0062] In some embodiments, a machine learning model implemented as per the mechanisms described herein (e.g., using topology 400 and / or topology 500) can be trained using any suitable optimizer (e.g., RMSProp optimizer), any suitable learning rate (e.g., a learning rate of 10 minus 4), any suitable batch size (e.g., a batch size of 1), and any suitable loss function (e.g., an L2 loss function). In some embodiments, a training epoch can include any suitable number of sequences (e.g., 2048 sequences), each of which (after any averaging) can include any suitable number of input frames of any suitable size (e.g., 64 frames of size 32×32). In some embodiments, any suitable number of test sequences can be used to evaluate the performance of the trained machine learning model. For example, a test sequence including 64 frames of size 128×128 can be obtained from a separate partition of the source dataset and used to evaluate the performance of the trained machine learning model.

[0063] FIG. 6 illustrates an example process 600 for generating a digital image using low bit-depth image sensor data, according to some embodiments of the disclosed subject matter.

[0064] At 602, process 600 may receive a high bit depth still image (e.g., captured with a conventional image sensor such as a CMOS image sensor or a CCD image sensor). In some embodiments, process 600 may receive the high bit depth still image from any suitable source or combination of sources. For example, process 600 may receive the high bit depth image from a server (e.g., server 220), from a computing device (e.g., computing device 210), from a memory (e.g., memory 310, memory 320), etc.

[0065] At 604, process 600 can generate a series of simulated frames from each of the still image frames using any suitable technique or combination of techniques, along with frame-to-frame motion (e.g., simulating translation, rotation, axial motion, etc.) For example, as described above with respect to equation (1), process 600 can determine a position, orientation, and / or scale of a bounding box, and can generate the simulated frames based on the position, orientation, and / or scale of the bounding box.

[0066] At 606, process 600 can generate a simulated low bit depth frame from the series of simulated frames using any suitable technique or combination of techniques. For example, process 600 can generate a low bit depth image from each simulated frame using the technique described above with respect to equation (2). As another example, process 600 can generate multiple low bit depth images from each simulated frame. As described above with respect to FIG. 4, in some embodiments, process 600 can combine information from multiple simulated low bit depth frames, for example, by averaging blocks of multiple frames.

[0067] In some embodiments, process 600 can omit 602 through 606. For example, process 600 can generate simulated low-depth frames from the video data as described below with respect to 608 and 610.

[0068] At 608, process 600 can receive high bit depth video data (e.g., captured with a conventional image sensor such as a CMOS image sensor or a CCD image sensor). In some embodiments, process 600 can receive the high bit depth video data from any suitable source or combination of sources. For example, process 600 can receive high bit depth images from a server (e.g., server 220), from a computing device (e.g., computing device 210), from a memory (e.g., memory 310, memory 320), etc.

[0069] At 610, process 600 may generate simulated low bit depth frames from the frames of video data using any suitable technique or combination of techniques. For example, process 600 may generate a low bit depth image from each frame of the video using a technique such as described above with respect to equation (2). As another example, process 600 may generate multiple low bit depth images from each frame of the video. As described above with respect to FIG. 4, in some embodiments, process 600 may combine information from multiple simulated low bit depth frames, for example, by averaging blocks of multiple frames.

[0070] At 612, process 600 may use intensities from the low bit depth frames (e.g., generated at 606 and / or 610) and corresponding frames of still image and / or video data to train a machine learning model (e.g., having a topology as described above with respect to FIGS. 4 and / or 5). In some embodiments, process 600 may use any suitable technique or combination of techniques to train the machine learning model. For example, process 600 may use the techniques described above with respect to FIGS. 4 and 5 to train the machine learning model. For example, as described above, in some embodiments, process 600 may use an L2 loss function and may use an RMSProp optimizer.

[0071] At 614, the process 600 can receive a series of low bit depth frames from any suitable source. For example, the process 600 can receive SPAD image sensor data (e.g., 1-bit SPAD image sensor data, multi-bit SPAD image sensor data) of a scene. As another example, the process 600 can receive Jot image sensor data (e.g., 1-bit Jot image sensor data, multi-bit Jot image sensor data) of the same scene.

[0072] At 616, the process 600 may combine information from the multiple frames received at 614. For example, as described above with respect to FIG. 4, multiple frames (e.g., 2 frames, 3 frames, 4 frames, 5 frames, 6 frames, 7 frames, 8 frames, etc.) may be averaged and the averaged frames may be provided as inputs to the trained machine learning model. In some embodiments, each frame may be included in a single average. For example, 8 frames (e.g., frames 1 through 8) may be averaged to generate a first input, and another 8 frames (e.g., frames 9 through 16) may be averaged to generate a second input. In some embodiments, combining information from multiple frames may be omitted. For example, if a single low bit depth frame is provided as an input to the trained machine learning model, combining information from multiple frames may be omitted.

[0073] At 618, the process 600 can provide the low bit depth image information as an input to a trained machine learning model. In some embodiments, the process 600 can provide a single sequence of low bit depth frames as an input to a trained machine learning model. In some embodiments, the process 600 can combine information from multiple low bit depth frames to generate an aggregated low bit depth frame and provide a sequence of aggregated low bit depth frames as an input to the trained machine learning model.

[0074] At 620, the process 600 can receive relatively high bit depth (e.g., full depth) frames as output from the trained machine learning model. For example, the trained machine learning model as described above with respect to FIG. 4 or 5 can generate a high bit depth frame for each input provided to the trained machine learning model (e.g., for each aggregated low bit depth frame for each low bit depth frame provided as an input to the trained machine learning model). As a further specific example, the process 600 can receive the output in real time or near real time after at least a predetermined number of inputs have been provided to the machine learning model (e.g., generated by a trained machine learning model implemented with one or more single convolutional LSTM layers). In such an example, the output can be generated after a delay corresponding to a number of frames in the acceptance domain of the trained machine learning model. In such an example, using a convolution with a 3×3×3 kernel, the acceptance domain can cover 1+2m frames, where m corresponds to the number of three-dimensional convolutional layers used to implement the machine learning model. In a specific example, using an average of eight frames for each input frame at a frame rate of about 100,000 frames per second, the delay can be on the order of about 1 millisecond (ms) for a machine learning model implemented using topology 500. In yet another specific example, if temporal padding is applied before the input of the first frame, process 600 can receive output in real time with substantially no delay (e.g., no delay after the first frame is received at 614 and / or combined at 616). In yet another specific example, process 600 can receive output with a relatively short delay after a predetermined number of inputs are provided to the machine learning model (e.g., generated by a trained machine learning implemented with one or more bidirectional convolutional LSTM layers). In such an example, the output can be generated after a delay corresponding to a number of frames within the processing window.In such an example, if the input is processed in 128 frame blocks, the delay may correspond to the time taken to acquire 128 frames. In a specific example, with a frame rate of about 100,000 per second, and using an average of 8 frames for each input frame, the delay may be on the order of about 10 ms.

[0075] In some embodiments, the process 600 (or any other suitable process) may perform additional image processing on the output received at 620. For example, the process 600 (or any other suitable process) may perform noise removal on the output received at 620. As another example, the process 600 (or any other suitable process) may perform blur removal on the output received at 620. In some embodiments, such image processing may be used with respect to the output of a machine learning model that is relatively lightweight (e.g., having fewer layers, having fewer channels, etc.). For example, a machine learning model implemented using topology 500 may output a relatively high quality image from further image processing that may be less useful (or may not be useful at all). As another example, a machine learning model implemented using fewer layers, fewer channels, etc. may utilize fewer computing resources than a machine learning model implemented using topology 500 and may generate an image that is relatively lower quality (e.g., may benefit from the use of additional image processing). In such examples, additional image processing may or may not be applied prior to utilizing the output of the trained machine learning model as input for a downstream task (e.g., for use in a machine vision task).

[0076] At 622, process 600 can display the one or more output images (e.g., using a display), provide the one or more output images to another device, or provide the one or more output images to a machine vision system (e.g., to perform a computer vision task). For example, process 600 can display the one or more images via a display (e.g., display 330, display 304, display 314). As another example, process 600 can use the one or more images in a computer vision application (e.g., object detection or object recognition).

[0077] FIG. 7 shows examples of low bit depth frames of a scene generated from still images of simulated motion, predicted low noise high bit depth frames of a scene generated from a stream of low bit depth frames using the mechanisms described herein, and a ground truth image of the scene. The results shown in FIG. 7 to FIG. 11 were generated using a trained machine learning model implemented using the topology described above with respect to FIG. 5. The machine learning model was trained using an RMSProp optimizer, a learning rate of 10-4, a batch size of 1, and an L2 loss function. Each training epoch contained 2048 sequences, each of which contained 64 frames of size 32×32 (after any averaging). The test sequences contained 64 frames of size 128×128 and were taken from a separate partition of the source dataset. The core model was trained for 100 epochs on the synthetic motion dataset over an average of 8 binary frames. A peak signal-to-noise ratio (PSNR) of 32.10 and a structural similarity index (SSIM) of 0.8878 were achieved. The predicted image in Figure 7 was generated using the core model.

[0078] FIG. 8 shows examples of low bit-depth frames based on a variety of multiple binary frames generated from still images of simulated motion, predicted low-noise high bit-depth frames of a scene generated from a stream of low-bit-depth frames using the mechanisms described herein, and a ground truth image of the scene.

[0079] In some scenes, non-negligible motion may occur between each binary frame. As such, averaging may result in blurring and loss of information. Figure 8 shows a comparison of four models whose inputs were averaged over 1, 2, 4, and 8 binary frames. The speed of motion between binary frames was inversely scaled by the number of averaged frames. In this way, the same amount of motion was achieved between averaged frames so that the model's behavior could be analyzed while the signal-to-motion ratio was reduced.

[0080] Four models were each trained from scratch for 50 epochs. The 1, 2, 4, and 8 frame models achieved PSNRs of 27.95, 29.37, 30.11, and 31.68, respectively, and SSIM ods of 0.7448, 0.8090, 0.8305, and 0.8710, respectively. Performance generally decreases as the signal-to-motion ratio increases. However, models implemented and trained according to the mechanisms described herein performed very satisfactorily in the extreme case where the motion was on the order of one pixel per binary frame.

[0081] FIG. 9 shows examples of low bit-depth frames of a scene generated from video data, predicted low-noise high bit-depth frames of the scene generated from a stream of low bit-depth frames using the mechanisms described herein, and a ground truth image of the scene.

[0082] A machine learning model implemented as per the topology described above with respect to Figure 5 was initialized with the core model weights and trained for 50 epochs using training data generated from the videos. The model achieved a PSNR of 34.90 and a SSIM of 0.9238. Note that this is better than the synthetic motion performance, likely due to the presence of large static regions in many real videos that facilitate high performance (e.g., based on reduced motion in the test data).

[0083] A baseline technique involving averaging all input frames together to attempt to generate a high bit depth image (e.g., averaging pixel values ​​using 64 binary frames without motion correction) was performed as a basis for comparison. This resulted in a PSNR of 20.52 and a SSIM of 0.6877 on the real video dataset. Another baseline technique involved performing motion averaging over 64 binary frames (or 8 averaging frames). This resulted in a PSNR of 20.83 and a SSIM of 0.5868 on the real video dataset.

[0084] FIG. 10 shows examples of high bit-depth frames of a scene generated from a setup binary frame using another technique, predicted low-noise high bit-depth frames of the scene generated from a stream of low bit-depth frames using the mechanisms described herein, and a ground truth image of the scene.

[0085] Frames generated from real video data were used to generate high bit depth images using QuantaBurst Photography (QBP) techniques (e.g., as described in U.S. Patent No. 11,170,549). Results such as those shown in FIG. 10 were generated using frames generated from real video data, without frame repetition (not intentional), resulting in 8 times faster motion than used in the training dataset used to train the model described above with respect to FIG. 9; the "predicted image" in FIG. 10 was generated using the same model.

[0086] Both the QBP technique and the trained model in Fig. 9 were used to generate high bit depth images from 100 example sequences of low bit depth images. Since QBP recovers only a single frame from each sequence, the corresponding frame was extracted from the output of the trained model for comparison. QBP and QBP-BM3D (a variant of QBP with BM3D denoiser) achieved PSNR of 28.50 and 29.23 and SSIM of 0.7449 and 0.8160, respectively. The trained model yielded a PSNR of 29.71 and a SSIM of 0.8489. Although unintentional, the fact that the test motion speed was much higher than in the training dataset proves the generalizability of the mechanism described herein to various motion speeds.

[0087] FIG. 11 shows an example of the results of an object detection operation performed on low bit-depth frames of a scene generated from video data, predicted low-noise high bit-depth frames of the scene generated from a stream of low bit-depth frames using the mechanisms described herein, and ground truth images of the scene.

[0088] For many applications, the high bit depth frames generated using the mechanisms described herein can be used by downstream processing applications. For example, a user may wish to run an object detector or compute optical flow and / or object recognition tasks. FIG. 11 shows results generated by running an off-the-shelf object detector (YOLOv3, e.g., as described in Redmon et al., "YOLOv3: Incremental Improvements," arXiv:1804.02767 (2018)) using an 8-frame average of binary frames generated from frames of the video, the output of the trained machine learning model described above with respect to FIG. 9, and the corresponding frames of the video (classified as "ground truth"). As shown in FIG. 11, the results of the object detection were similar when run on the original video frames and on the images output by the machine learning model, although no objects were detected in the 8-frame average.

[0089] In some embodiments, any suitable computer-readable medium can be used to store instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer-readable media can be transitory or non-transient. For example, non-transient computer-readable media can include media such as magnetic media (such as hard disks, floppy disks, etc.), optical media (such as compact disks, digital video disks, Blu-ray disks, etc.), semiconductor media (such as RAM, flash memory, Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.), any suitable media that is not transitory or does not have any appearance of permanence during transmission, and / or any suitable tangible media. As another example, a transitory computer-readable medium can include wires, conductors, optical fibers, signals in circuits, on a network, or any suitable media that is transitory and does not have any appearance of permanence during transmission, and / or any suitable intangible media.

[0090] It should be noted that, as used herein, the term mechanism can encompass hardware, software, firmware, or any suitable combination thereof.

[0091] It should be understood that the above-described steps of the process of Figure 6 can be executed or performed in any suitable order or sequence, not limited to the order or sequence shown and described in the figures. Also, some of the above-described steps of the process of Figure 6 can be executed or performed substantially simultaneously or in parallel, where appropriate, to reduce wait and processing times.

[0092] While the invention has been described and illustrated in the above illustrative embodiments, it will be understood that the disclosure is made by way of example only, and that numerous changes in the details of the implementation of the invention can be made without departing from the spirit and scope of the invention, which is limited only by the claims that follow. Features of the disclosed embodiments can be combined and rearranged in various ways.

Claims

1. 1. A system for generating a digital image having a bit depth higher than a bit depth of an image frame detected by an image sensor, comprising: a communication system configured to receive image frames detected by the image sensor; at least one processor, receiving, via the communication system, a time series of image frames including an image sensor bit depth detected by the image sensor for a predetermined scene; providing data from a plurality of image frames of the time series of image frames based on a bit depth of the image sensor to a trained machine learning model having a topology structured to include a feature analysis block trained to process spatiotemporal relationships between pixels across input image frames, and an output block trained to generate estimated pixel intensity values ​​having a bit depth higher than the bit depth of the image sensor based on learned patterns of the plurality of image frames; and at least one processor programmed to generate an enhanced image of a given scene having a higher bit depth based on an output of the trained machine learning model.

2. The system of claim 1 , wherein the image sensor comprises a plurality of single-photon avalanche diodes.

3. The system of claim 1 , wherein the plurality of image frames comprises a series of binary frames.

4. The system of claim 1 , wherein a pixel of the image sensor bit depth is represented using 4 bits or less.

5. The system of claim 1 , wherein pixels of the enhanced image of the given scene are represented using at least 8 bits.

6. The system of claim 1 , wherein the trained machine learning model has multiple input channels, each corresponding to a color channel.

7. the plurality of image frames include a first noise content, and the enhanced image of the predetermined scene includes a second noise content that is less than the first noise content. The system of claim 1 .

8. 8. The system of claim 7, wherein the plurality of image frames comprises a moving image of the scene, the first noise content comprises motion artifacts, and the plurality of enhanced images are generated to comprise an enhanced moving image of the scene with the motion artifacts removed based on output of a trained machine learning model.

9. 1. A method for generating a digital image having a bit depth higher than a bit depth of an image sensor detected by an image sensor, comprising: receiving, from a communication system, a series of image frames output from the image sensor for a predetermined scene, the image frames including data at the bit depth of the image sensor for a plurality of pixels; providing data from a plurality of image frames of the series of image frames based on a bit depth of the image sensor to a trained machine learning model that processes spatiotemporal relationships between pixels across the plurality of image frames and predicts pixel intensity values ​​based on a trained representation of the plurality of image frames; generating an enhanced image of the predetermined scene having a higher bit depth based on an output of the trained machine learning model; and A method for providing

10. the image sensor includes a plurality of single-photon avalanche diodes; 10. The method of claim 9.

11. the series of image frames comprises a series of binary frames; 10. The method of claim 9.

12. a bit depth pixel of the image sensor is represented using 4 bits or less; 10. The method of claim 9.

13. 10. The method of claim 9, wherein pixels of the enhanced image of the given scene are represented using at least 8 bits.

14. The method of claim 9 , wherein the trained machine learning model includes multiple input channels, each input channel corresponding to a color channel.

15. 15. The method of claim 14, wherein the trained machine learning model comprises a topology structured to include a feature analysis block trained to process spatiotemporal relationships between pixels across input image frames, and an output block trained to generate estimated pixel intensity values ​​having a bit depth higher than a bit depth of the image sensor based on learned patterns of the plurality of image frames.

16. The trained machine learning model is a three-dimensional (3D) convolutional layer forming at least part of the feature analysis block; a two-dimensional (2D) convolutional long short-term memory (LSTM) layer configured to receive the output of the three-dimensional convolutional layer; a concatenation layer configured to generate a tensor comprising a concatenation of the output of the 2D convolutional LSTM layer and low bit-depth image information; a two-dimensional convolutional layer forming at least a portion of an output block and configured to generate an output based on the tensors generated by the concatenation layer; 16. The method of claim 15, comprising:

17. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for generating a digital image, comprising: receiving from a communication system a series of image frames output from an image sensor for a predetermined scene, the image frames including image sensor bit depth data for a plurality of pixels and a first noise content; providing data from a plurality of initial image frames of the series of image frames based on a bit depth of the image sensor to a trained machine learning model that identifies spatial and temporal patterns of pixels across the plurality of initial image frames and learns statistical prior distributions of the initial image frames to predict pixel intensity values; generating a plurality of enhanced image frames of the scene comprising a bit depth greater than a bit depth of the image sensor and a noise content less than a first noise content based on an output of the trained machine learning model; A non-transitory computer-readable medium containing computer-executable instructions, including:

18. the image sensor includes a plurality of single-photon avalanche diodes; 20. The non-transitory computer-readable medium of claim 17.

19. the series of initial image frames comprises a series of binary frames; 20. The non-transitory computer-readable medium of claim 17.

20. a pixel of the image sensor bit depth is represented using 4 bits or less, and a pixel of the enhanced image frame is represented using at least 8 bits; 20. The non-transitory computer-readable medium of claim 17.

Images