Camera system including monochrome and color cameras with global shutter sensors - Patents.com

Abstract

The camera system includes a monochrome camera having a global shutter for capturing a first image of a scene, and a color camera disposed remotely from the monochrome camera and having a global shutter for capturing a second image of the scene, the second image being registered to the first image and color information of the second image being provided to the first image to obtain a third image representative of the scene.

Description

[Technical field]

[0001] The present disclosure relates generally to camera systems and, more particularly, to a camera system that includes a monochrome camera with a global shutter sensor and a color camera with a global shutter sensor, whose images are used in combination to produce images with minimal visual artifacts. [Background technology]

[0002] Rolling shutter red-green-blue (RGB) cameras are sometimes used to collect images of an environment. Under certain environmental conditions (e.g., under lower light conditions), the integration time is limited in length to reduce motion blur, which can produce noisy and underexposed images. On the other hand, the integration time of a rolling shutter RGB camera must not be too short, otherwise banding artifacts from artificial lighting (e.g., flickering due to alternating current and / or dimming control that varies the duty cycle) can occur. Thus, the integration time of a rolling shutter RGB camera may be narrow, over which operation of the RGB camera is feasible. Summary of the Invention [Means for solving the problem]

[0003] Aspects and advantages of embodiments of the present disclosure are set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of exemplary embodiments.

[0004] In an exemplary embodiment, a camera system includes a monochrome camera having a global shutter and configured to capture a first image of a scene, and a color camera disposed remotely from the monochrome camera, having a global shutter and configured to capture a second image of the scene, The camera system further includes one or more processors configured to register the second image to the first image and provide color information of the second image to the first image to obtain a third image representative of the scene.

[0005] In some implementations, the monochrome camera and the color camera are synchronized to capture the first image and the second image substantially simultaneously.

[0006] In some implementations, the first and second images are captured when peak lighting conditions occur within an environment in which the monochrome and color cameras are disposed, respectively.

[0007] In some implementations, the monochrome camera and the color camera are arranged to face the same direction and are arranged less than a threshold distance from each other, for example, the threshold distance is 10 centimeters or less.

[0008] In some implementations, the one or more processors are configured to provide color information of the second image to the first image, for a portion of the second image that is aligned with a corresponding portion of the first image, by transferring color information of the portion of the second image to the corresponding portion of the first image.

[0009] In some implementations, the one or more processors are configured to identify non-aligned portions of the first image where portions of the second image are not aligned with corresponding portions of the first image, and apply the combined color information to the non-aligned portions of the first image.

[0010] In some implementations, the color camera is a red-green-blue (RGB) camera.

[0011] In some implementations, the color camera has a lower resolution than the monochrome camera.

[0012] In some implementations, the monochrome camera has a larger size than the color camera.

[0013] In some implementations, the first image includes a luma component and the third image includes the luma component of the first image and a chroma component that is based on the second image.

[0014] In some implementations, the camera system includes a first machine learning resource for aligning the second image to the first image and a second machine learning resource for synthesizing color information for non-aligned portions of the first image where portions of the second image are not aligned to corresponding portions of the first image, and applying the synthesized color information to the non-aligned portions of the first image.

[0015] In an exemplary embodiment, a computer-implemented method for a camera system is provided that includes capturing a first image of a scene with a monochrome camera having a global shutter, capturing a second image of the scene with a color camera disposed away from the monochrome camera and having a global shutter, registering the second image with the first image, and providing color information of the second image to the first image to obtain a third image representative of the scene.

[0016] In some implementations, the first image includes a single channel having a luma component, and the third image includes at least three channels including a first channel having a luma component and multiple channels including chroma components based on the second image.

[0017] In some implementations, the monochrome camera and the color camera are synchronized to capture the first image and the second image substantially simultaneously, with the capturing of the first image and the second image, respectively, occurring at the peak of the illumination cycle.

[0018] In some implementations, the color camera and the monochrome camera have substantially the same field of view.

[0019] In some implementations, the difference between the time that the color camera captures the second image and the time that the monochrome camera captures the first image is less than the integration time of the monochrome camera.

[0020] In some implementations, providing color information of the second image to the first image includes, for a portion of the second image that is aligned with a corresponding portion of the first image, transferring color information of the portion of the second image to the corresponding portion of the first image.

[0021] In some implementations, the method includes identifying a non-aligned portion of the first image, where a portion of the second image is not aligned with a corresponding portion of the first image, synthesizing color information for the non-aligned portion of the first image with a machine learning resource, and applying the synthesized color information to the non-aligned portion of the first image.

[0022] In an exemplary embodiment, a non-transitory computer-readable medium is provided that stores instructions executable by one or more processors of a camera system. The non-transitory computer-readable medium stores instructions executable by one or more processors of the camera system. The instructions include instructions for causing a monochrome camera having a global shutter to capture a first image of a scene, instructions for causing a color camera having a global shutter and spaced less than a threshold distance from the monochrome camera to capture a second image of the scene, where the second image has a lower resolution than the first image, instructions for registering the second image with the first image, and instructions for obtaining a third image representing the scene by providing color information of the second image to the first image.

[0023] The non-transitory computer-readable medium may store further instructions for performing other aspects and operations of the camera systems and computer-implemented methods described herein.

[0024] The above and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description, drawings, and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present disclosure and, together with the following description, serve to explain related principles.

[0025] A detailed discussion of exemplary embodiments directed to one of ordinary skill in the art is set forth herein and refers to the accompanying drawings. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 illustrates an example system including a block diagram of a system including a camera system, a server computing system, and one or more external computing devices, in accordance with one or more examples of the present disclosure. [Figure 2A]FIG. 1 is an illustrative diagram of a camera system according to one or more examples of the present disclosure. [Figure 2B] FIG. 1 is an illustrative diagram of a camera system according to one or more examples of the present disclosure. [Diagram 3] 1 is an illustrative diagram of one or more aspects of a camera system in accordance with one or more examples of the present disclosure. [Figure 4] 1 is an illustrative diagram of one or more aspects of a camera system in accordance with one or more examples of the present disclosure. [Diagram 5] FIG. 1 is a flow diagram of an exemplary non-limiting computer-implemented method according to one or more examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Reference is now made to embodiments of the present disclosure, one or more examples of which are illustrated in the drawings. Each example is provided as an explanation of the present disclosure, and is not intended to limit the disclosure. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the scope or spirit of the disclosure. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield yet another embodiment. It is therefore intended that the present disclosure cover such modifications and variations as come within the scope of the appended claims and their equivalents.

[0028] The terms used herein are used to describe exemplary embodiments and are not intended to limit and / or restrict the present disclosure. The singular forms "a", "an" and "the" are intended to include the plural unless the context clearly dictates otherwise. In this disclosure, terms such as "including", "having", "comprising" and the like are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0029] It will be understood that although terms such as first, second, third, etc. may be used herein to describe various elements, the elements are not limited by these terms. Instead, these terms are used to distinguish one element from another. For example, a first element may be referred to as a second element and a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0030] The term "and / or" includes a combination of multiple associated listed items or any item among multiple associated listed items. For example, the scope of an expression or phrase "A and / or B" includes item "A," item "B," and the combination of items "A and B."

[0031] Additionally, the scope of the phrase or expression "at least one of A or B" is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Similarly, the scope of the phrase or expression "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0032] Examples of the present disclosure are directed to camera systems capable of generating or producing high quality images in a wide variety of environments (e.g., on land), including indoor and outdoor spaces, and in natural and artificial light environments. For example, artificial light that flickers due to alternating current, duty cycles of lights such as light emitting diodes, duty cycles of dimmer circuits, etc., can cause banding in images generated or produced by other methods, such as images generated by a rolling shutter camera.

[0033] According to examples of the present disclosure, a camera system may include a monochrome camera having a global shutter and configured to capture a first image of a scene, and a color camera disposed remotely from the monochrome camera, having a global shutter, and configured to capture a second image of the scene. The camera system may further include one or more processors configured to align the first image with the second image and provide color information of the second image to the first image to obtain a third image representative of the scene.

[0034] For example, a global shutter works by exposing the entire scene from one side of the sensor to the other at the same time. A camera with a global shutter may not have much warping or distortion. In contrast, a rolling shutter works by turning on pixels from one side of the sensor and sweeping them to the other side of the sensor in a scan-like manner. A camera with a global shutter may not produce images with the same high resolution as a camera with a rolling shutter. However, a camera with a rolling shutter may exhibit motion artifacts including spatial distortion and banding.

[0035] According to examples of the present disclosure, the camera system includes a monochrome camera with a global shutter and a color camera (e.g., a multi-chromatic camera such as an RGB camera) with a global shutter. Because the monochrome and color cameras utilize a global shutter, banding artifacts can be avoided, and shorter integration times can reduce motion blur. To increase the sensitivity of the camera system, the monochrome camera utilizes a monochrome sensor. For example, a color filter array (CFA) in a typical RGB camera reduces the camera sensitivity by about 2.5 times. By removing the CFA and using only the monochrome sensor to capture images, the camera speed can be increased (e.g., 2.5 times compared to a typical RGB camera) and the linear resolution of the camera can be increased (e.g., about 1.5 to 2 times in pixels compared to a typical RGB camera). For example, in some implementations, the monochrome sensor can have a resolution of 10 to 20 megapixels. Additionally, the camera sensitivity of the camera can be increased by 2 to 2.5 times.

[0036] In general, pixels from an image captured using a global shutter camera have a lower dynamic range than pixels from an image captured using a rolling shutter camera because the global shutter pixel is more complex and has less volume to collect and hold charge from detected photons. For example, the dynamic range of a global shutter pixel (i.e., the "well volume" that affects the contrast ratio between the maximum and minimum measurable light intensity) may be about one bit less than a similarly sized rolling shutter pixel. According to an example of the present disclosure, to increase the dynamic range of the camera system, a monochrome camera includes a monochrome sensor with increased linear resolution. The camera system can be implemented to capture a wider variety of environments (e.g., indoors and outdoors) under a wider range of lighting conditions.

[0037] Color is provided by using a secondary global shutter color camera. For example, the color camera can be an RGB camera. For example, the color camera has a lower resolution than the monochrome camera. For example, the color camera can have 1 / 2 to 1 / 16 the resolution (e.g., in number of pixels) of the monochrome camera. Because the color camera is configured to have a lower resolution, it does not significantly increase the size, cost, or data rate of the camera system. For example, the monochrome and color cameras can include an f-theta (fisheye) lens with a wide (e.g., greater than 120 degrees) field of view.

[0038] In some implementations, the monochrome camera and the color camera can have the same field of view. The monochrome camera and the color camera can be configured to face the same direction to capture the same scene. Static parallax between the monochrome camera and the color camera can be minimized by positioning the monochrome camera and the color camera within a few centimeters of each other (e.g., 3 to 10 centimeters, such as less than 5 centimeters).

[0039] In some implementations, a computational process (e.g., a machine learning algorithm implemented by the machine learning resource) can align a low-resolution color (e.g., RGB) image to a high-resolution monochrome image. For example, the alignment process can include converting the color image to a monochrome image and calculating an alignment and confidence field between the monochrome image from the monochrome camera and the converted monochrome image. The alignment process can be performed by the camera system or can be performed remotely (e.g., by a server computing system). For example, the alignment process can be performed using known alignment methods. The alignment process can be performed in real time or as part of a post-processing procedure.

[0040] In some implementations, a computational process (e.g., a machine learning algorithm implemented by the machine learning resource) can transfer colors from the low-resolution color (RGB) image to the high-resolution monochrome image for the successfully registered portions of the high-resolution monochrome image. For example, colors can be transferred (e.g., interpolated) from the portions of the low-resolution color (RGB) image that correspond to the successfully registered portions of the high-resolution monochrome image. For example, the transfer or merging operation can include creating a lower (e.g., half-resolution) chroma (e.g., CbCr) channel for the high-resolution monochrome image (having a Y or luma component).

[0041] In some implementations, a computation process (e.g., a machine learning algorithm implemented by a machine learning resource) can synthesize (generate) colors for portions of the high-resolution monochrome image that have failed to be aligned (e.g., due to occlusion or insufficient signal-to-noise ratio). The alignment can be considered to have failed when the confidence level for the alignment is below a threshold level. For example, colors for portions of the monochrome image that have not been successfully aligned can be synthesized (e.g., generated) by referencing colors of portions of the high-resolution monochrome image adjacent to the unaligned portions, or colors can be synthesized using a machine learning resource that synthesizes colors using a neural network trained to synthesize colors. In some implementations, the luma component of the monochrome image remains unchanged. That is, only the chromatic channel is synthesized. For example, the color channel can be synthesized at half the resolution of the color image.

[0042] The camera system can be configured to generate a resultant image (e.g., a third image) based on the monochrome image and the color image (e.g., based on the luma component of the monochrome image and the chromatic component of the color image), according to color information transferred to parts of the high-resolution monochrome image that are successfully aligned, and according to color information synthesized when alignment is not successful.

[0043] In some implementations, a computational process (e.g., a machine learning algorithm implemented by a machine learning resource such as a trained neural network) can optimize the entire process end-to-end, such that the input to the network is two source images (i.e., a monochrome image and a color image) and the output is a composite high-resolution RGB image. In this implementation, there are no separate alignment and color transfer operations, because the neural network performs the process end-to-end, with alignment performed implicitly inside.

[0044] Because the monochrome and color cameras are both global shutter cameras, they can be strongly synchronized in time to avoid motion parallax (e.g., they can each capture images of the scene simultaneously), and with the fields of view aligned, they can only fail to align if there is static parallax, which can be minimized by positioning the monochrome and color cameras within a few centimeters of each other, as discussed above. Additionally, the color (e.g., RGB) camera can have a slightly wider field of view than the monochrome camera to ensure that more border pixels in the monochrome image have associated pixels in the color (e.g., RGB) image.

[0045] The camera system disclosed herein can be implemented in any electronic device capable of housing two or more cameras, for example, the camera system can be implemented in a smartphone, a security system, a surveillance system, in a vehicle, etc.

[0046] A color camera can capture images in any color space. Images generated based on monochrome and color images can be represented by any color space. For example, color spaces can include RGB (red, green, and blue additive primaries), YCbCr (luma component, blue-difference chroma component, red-difference chroma component), HSV (hue, saturation, value), YUV (luminance, blue projection, red projection), L*a*b* (lightness value, green-red opponent channel, and yellow-blue opponent channel), etc.

[0047] Exemplary aspects of the present disclosure provide several technical effects, advantages, and / or improvements in camera system technology. For example, according to the examples disclosed herein, camera systems have expanded operation and capabilities because the cameras will operate in more and more diverse environments. Thus, there is no need to have separate indoor and outdoor cameras. For example, pose and 3D reconstruction algorithms are more accurate and of higher quality when the underlying images are captured using global shutter cameras. The camera systems disclosed herein can be used to map a greater variety of environments under a wider range of lighting conditions, resulting in images being collected in more environments and with a greater operational range of images. Additionally, camera systems can be made smaller, lighter, and more portable (e.g., by utilizing smaller color cameras rather than monochrome cameras).

[0048]

[0023] Referring now to the drawings, Figure 1 illustrates a block diagram of a system including a camera system, a server computing system, and one or more external computing devices, according to one or more examples of the present disclosure. In Figure 1, an exemplary system 1000 includes a camera system 100, a server computing system 300, and one or more external computing devices 400, which are interconnected via a network 200. Any communication interface (such as a network interface card) suitable for communicating over the network 200 may be utilized as appropriate or desired for the camera system 100, the server computing system 300, and the one or more external computing devices 400.

[0049] The camera system 100 may include, for example, any electronic device capable of housing two or more cameras. For example, the camera system may be implemented in a smartphone, a security system, a surveillance system, in a vehicle, etc. The server computing system 300 may include a server or a combination of servers (e.g., web servers, application servers, etc.) that communicate with each other, for example, in a distributed manner. The one or more external computing devices 400 may include a personal computer, a smartphone, a laptop, a tablet computer, etc.

[0050] According to some implementations of the present disclosure, the camera system 100 can communicate with the server computing system 300 to generate a third image based on a first image captured by the monochrome camera 182 and a second image captured by the color camera 184. For example, the server computing system 300 can be configured to align the first and second images and / or combine color information to generate the third image.

[0051] According to some implementations of the present disclosure, the camera system 100 can communicate with a server computing system 300 to execute an imaging application 330 that can be used to activate or control the monochrome camera 182 and / or the color camera 184.

[0052] According to some implementations of the present disclosure, the camera system 100 can communicate with one or more external computing devices 400 to execute imaging applications for activating or controlling the monochrome camera 182 and / or the color camera 184, for aligning a first image with a second image, and / or for synthesizing color information to generate a third image. By way of example, the one or more external computing devices 400 can include a home appliance, such as a home security system, or a vehicle, or a smartphone.

[0053] For example, the network 200 may include any type of communication network, such as a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), etc. For example, wireless communication between elements of the examples described herein may be implemented via wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), Bluetooth low energy (BLE), near field communication (NFC), radio frequency (RF) signals, etc. For example, wired communication between elements of the examples described herein may be implemented via paired cables, coaxial cables, fiber optic cables, Ethernet cables, etc. Communications over the network may use a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0054] The camera system 100 may include one or more processors 110, one or more memory devices 120, an imaging application 130, machine learning resources 140, input devices 150, output devices 160, a display device 170, and one or more sensors 180. Each of the components of the camera system 100 may be operatively connected to one another via a system bus. For example, the system bus may be any of several types of bus structures that may be further interconnected to a memory bus (with or without a memory controller), a peripheral bus, and / or a local bus using any of a variety of commercially available bus architectures.

[0055] The server computing system 300 can include one or more processors 310, one or more memory devices 320, an imaging application 330, and one or more machine learning resources 340. Each of the features of the server computing system 300 can be operatively connected to one another via a system bus. For example, the system bus can be any of several types of bus structures that can be further interconnected to a memory bus (with or without a memory controller), a peripheral bus, and / or a local bus using any of a variety of commercially available bus architectures.

[0056] The one or more external computing devices 400 may include similar structural features as the server computing system 300 (e.g., one or more processors 310, one or more memory devices 320, an imaging application 330, and one or more machine learning resources 340). Thus, a description of these features in the context of the one or more external computing devices will not be repeated again for the sake of brevity.

[0057] For example, the one or more processors 110, 310 may be any suitable processing device that may be included in the camera system 100 or the server computing system 300. For example, the one or more processors 110, 310 may include one or more of a processor, processor core, controller and arithmetic logic unit, central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), image processor, microcomputer, field programmable array, programmable logic unit, application specific integrated circuit (ASIC), microprocessor, microcontroller, etc., and combinations thereof, including any other device capable of responding to and executing instructions in a prescribed manner. The one or more processors 110, 310 may be a single processor or may be multiple processors operatively connected, for example, in parallel.

[0058] The one or more memory devices 120, 320 may include one or more non-transitory computer-readable storage media, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and volatile memory devices such as flash memory, USB drives, random access memory (RAM), internal or external hard disk drives (HDD), floppy disks, Blu-ray disks, or optical media such as CD ROM disks or DVDs, and combinations thereof. However, examples of the one or more memory devices 120, 320 are not limited to the above description, and as will be appreciated by those skilled in the art, the one or more memory devices 120, 320 may be realized by a variety of other devices and structures.

[0059] For example, the one or more memory devices 120 may store instructions that, when executed by the one or more processors 110, perform one or more operations of the imaging application 130 to cause a monochrome camera 182 with a global shutter to capture a first image of a scene, a color camera 184 with a global shutter and spaced less than a threshold distance from the monochrome camera 182 to capture a second image of the scene, where the second image has a lower resolution than the first image, aligning the second image with the first image, and obtaining a third image representing the scene by providing color information of the second image to the first image, as described according to examples of the present disclosure.

[0060] For example, the one or more memory devices 320 may store instructions that, when executed by the one or more processors 310, perform one or more operations of the imaging application 330 to cause a monochrome camera 182 with a global shutter to capture a first image of a scene, a color camera 184 with a global shutter and spaced less than a threshold distance from the monochrome camera 182 to capture a second image of the scene, where the second image has a lower resolution than the first image, aligning the second image with the first image, and obtaining a third image representing the scene by providing color information of the second image to the first image, as described according to examples of the present disclosure.

[0061] The one or more memory devices 120 may also include data 122 and instructions 124 that may be retrieved, manipulated, created, or stored by the one or more processors 110. In some examples, such data may be accessed and used as input to perform one or more operations of an imaging application 130, such as causing a monochrome camera 182 with a global shutter to capture a first image of a scene, causing a color camera 184 with a global shutter and spaced less than a threshold distance from the monochrome camera 182 to capture a second image of the scene, where the second image has a lower resolution than the first image, aligning the second image with the first image, and providing color information of the second image to the first image to obtain a third image representative of the scene, as described according to examples of the present disclosure.

[0062] The one or more memory devices 320 may also include data 322 and instructions 324 that may be retrieved, manipulated, created, or stored by the one or more processors 310. In some examples, such data may be accessed and used as input to perform one or more operations of an imaging application 330, such as causing a monochrome camera 182 with a global shutter to capture a first image of a scene, causing a color camera 184 with a global shutter and spaced less than a threshold distance from the monochrome camera 182 to capture a second image of the scene, where the second image has a lower resolution than the first image, aligning the second image with the first image, and providing color information of the second image to the first image to obtain a third image representative of the scene, as described according to examples of the present disclosure.

[0063] Imaging application 130 and imaging application 330 may include any application used to capture one or more images. For example, the captured images may be used in connection with security applications, surveillance applications, navigation applications, geomapping applications, etc. For example, images may be captured using one or more of monochrome camera 182 and color camera 184. As described according to examples of the present disclosure herein, image information obtained from images captured by monochrome camera 182 and color camera 184 may be used to generate another image.

[0064] In some implementations, the camera system 100 may include one or more machine learning resources 140. For example, one or more memory devices 120 may store or include one or more models including an image position matcher 142 and a color combiner 144. For example, the image position matcher 142 and the color combiner 144 may be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) and other types of machine-learned models including nonlinear and / or linear models. The neural networks may include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine-learned models may leverage attention mechanisms such as self-attention. For example, some exemplary machine-learned models may include a multi-headed self-attention model (e.g., a transformer model).

[0065] In some implementations, the inputs to the image position matcher 142 and the color combiner 144 may include image data from images captured by, for example, the monochrome camera 182 and the color camera 184. The image position matcher 142 and the color combiner 144 may process this image data to generate an output. As an example, the image position matcher 142 may process image data from images captured by the monochrome camera 182 and the color camera 184 to generate an image registration output (e.g., a registration or mapping of image data between the images captured by the monochrome camera 182 and the color camera 184) that determines whether a particular portion of the image captured by the color camera 184 is registered with a particular portion of the image captured by the monochrome camera 182. As another example, the color combiner 144 may process image data to generate an output. As an example, the color combiner 144 may process image data corresponding to portions of the image captured by the monochrome camera 182 that are determined by the image position matcher 142 to not be registered with portions of the image captured by the color camera 184. The color combiner 144 may process image data corresponding to misaligned portions of the images captured by the monochrome camera 182 by combining color information for the misaligned portions.

[0066] In some implementations, the server computing system 300 includes one or more machine learning resources 340, including an image position matcher 342 and a color composer 344. For example, processing of image data from images captured by the monochrome camera 182 and the color camera 184 can be performed remotely by the server computing system 300, such as by the image position matcher 342 and / or the color composer 344. The features and operations of the image position matcher 342 and the color composer 344 may correspond to the same features and operations of the image position matcher 142 and the color composer 144, respectively, described herein. Thus, a description of the features and operations of the image position matcher 342 and the color composer 344 in the context of the server computing system 300 will not be repeated again for the sake of brevity.

[0067] The camera system 100 may include an input device 150 configured to receive input from a user, such as one or more of a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, a button, a switch, an electronic pen or stylus pen, a gesture recognition sensor (e.g., for recognizing a user's gestures including movements of body parts), an input sound device or a voice recognition sensor (e.g., a microphone for receiving voice commands), a trackball, a remote controller, a portable (e.g., cellular or smart) phone, etc. The input device 150 may also be embodied by, for example, a touch-sensitive display device having touch screen capabilities. The input device 150 may be used by a user of the camera system 100 to provide input for executing the imaging application 130. The input device 150 may also be used by a user of the camera system 100 to provide input for causing the monochrome camera 182 and / or the color camera 184 to capture an image of a scene. For example, the input may be a voice input, a touch input, a gesture input, a click via a mouse or a remote controller, etc.

[0068] The camera system 100 includes an output device 160 configured to provide output to a user, and may include, for example, one or more of an audio device (e.g., one or more speakers), a haptic device (e.g., a vibration device 166) for providing haptic feedback to the user, a light source (e.g., one or more light sources, such as an LED, that provide visual feedback to the user), etc.

[0069] The camera system 100 includes a display device 170 that presents user-viewable information, for example, on a user interface (e.g., a graphical user interface). For example, the display device 170 can be a non-touch-sensitive display. The display device 170 can include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED), a flexible display, a 3D display, a plasma display panel (PDP), a cathode ray tube (CRT) display, and the like. However, the disclosure is not limited to these examples and can include other types of display devices. For example, the display device 170 can be configured to provide a graphical user interface to allow a user to select various options for the imaging application 130 (e.g., options to allow a user to define conditions for controlling the monochrome camera 182 and / or the color camera 184 to capture an image of a scene).

[0070] Camera system 100 includes one or more sensors 180. One or more sensors 180 may include a monochrome camera 182 and a color camera 184. For example, monochrome camera 182 and color camera 184 may be positioned adjacent to one another in a vertical or horizontal direction. With reference to Figures 2A-2B, an exemplary diagram of a camera system according to one or more examples of the present disclosure is shown.

[0071] For example, in FIG. 2A, the camera system 2100 shows a monochrome camera 182 disposed above a color camera 184, where the monochrome camera 182 is spaced apart from the color camera 184 by a distance d1. The distance d1 can correspond to approximately 3 centimeters to 10 centimeters, for example, less than 5 centimeters. The monochrome camera 182 and the color camera 184 can be aligned along an axis A1 and can face the same direction. In some implementations, the monochrome camera 182 can be larger in size than the color camera 184. In some implementations, the monochrome camera 182 can be disposed below the color camera 184.

[0072] 2B, the camera system 2200 shows the monochrome camera 182 disposed to the left of the color camera 184, where the monochrome camera 182 is spaced a distance d1 from the color camera 184. The distance d1 can correspond to approximately 3 to 10 centimeters, e.g., less than 5 centimeters. The monochrome camera 182 and the color camera 184 can be aligned along an axis A2 and can face the same direction. In some implementations, the monochrome camera 182 can be disposed to the right of the color camera 184.

[0073] The camera system 100 may include other types of sensors. For example, the one or more sensors 180 may include other sensors, such as an inertial measurement unit including one or more accelerometers and / or one or more gyroscopes. The one or more accelerometers may be used to capture motion information about the camera system 100. The one or more gyroscopes may also be used in addition to or instead of capturing motion information about the camera system 100. The one or more sensors 180 may also include other sensors, such as magnetometers, GPS sensors, etc.

[0074]

[0023] Referring to Figure 3, one or more aspects of a camera system according to one or more examples of the present disclosure are shown. In Figure 3, camera system 3000 includes one or more processors 110, monochrome camera 182, color camera 184, a first image 3100, a second image 3200, and a third image 3300.

[0075] In an exemplary embodiment, monochrome camera 182 is configured to capture a first image 3100 of a scene, and color camera 184 is configured to capture a second image 3200 of the scene. One or more processors 110 are configured to receive the first image 3100 and the second image 3200, and to perform various operations (such as aligning the second image 3200 to the first image 3100 and providing color information of the second image 3200 to the first image 3100, which will be described in more detail below) to generate a third image 3300 based on the first image 3100 and the second image 3200.

[0076] As discussed above, the monochrome camera 182 and the color camera 184 may each include a global shutter and may be synchronized with each other to capture images of the scene at the same time or substantially the same time (e.g., within 1 millisecond of each other). For example, the monochrome camera 182 and the color camera 184 may be synchronized to capture the first image 3100 and the second image 3200 when peak lighting conditions occur in the surrounding environment. For example, the first image 3100 and the second image 3200 may each be captured at the peak of a lighting cycle in the environment. For example, the peak lighting conditions or the peak of the lighting cycle may be determined by the camera system 100 observing or monitoring the environment and knowing the timing of the lighting cycle (e.g., by determining the duty cycle of a light such as a light emitting diode or a duty cycle of a dimmer circuit, or by determining the period of an AC power supply, etc.). For example, the difference between the time the color camera 184 captures the second image 3200 and the time the monochrome camera 182 captures the first image 3100 may be less than a threshold time. For example, the threshold time may correspond to the integration time of monochrome camera 182. The integration time corresponds to the time interval during which monochrome camera 182 captures (collects) and holds an electrical charge.

[0077] The color camera 184 can be configured to capture multi-color images in any color space. For example, the color camera 184 can include a color filter array. For example, the color filter array can be a Bayer color filter array (e.g., with an RGGB filter pattern) for arranging RGB color filters on a photosensor in a square grid (e.g., 2×2). The color camera 184 can also be configured to capture multi-color images in other color spaces, including YCbCr (luma, blue difference, red difference), HSV (hue, saturation, value), YUV (luminance, blue projection, red projection), L*a*b* (value, green-red opponent channel, and yellow-blue opponent channel), and the like.

[0078] As discussed above, the monochrome camera 182 may be physically larger in size and have a higher resolution than the color camera 184. For example, the resolution of the color camera 184 may be ½ to 1 / 16 of the resolution of the monochrome camera 182. One or more of the monochrome camera 182 and the color camera 184 may include an f-theta (fisheye) lens with a wide (e.g., greater than 120 degrees) field of view. The monochrome camera 182 and the color camera 184 may face the same direction and may be aligned with each other to reduce or avoid static parallax. Static parallax between the monochrome camera 182 and the color camera 184 may be further minimized by positioning the monochrome camera 182 and the color camera 184 less than a threshold distance from each other. For example, the monochrome camera 182 and the color camera 184 may be positioned within a few centimeters of each other (e.g., 3 to 10 centimeters, such as less than 5 centimeters).

[0079] Referring to FIG. 4, one or more aspects of a camera system according to one or more examples of the present disclosure are illustrated. In FIG. 4, the camera system 4000 includes a first image 3100, a second image 3200, an image position matcher 142, a color composer 144, and a third image 3300, as well as an operation 4100 of transferring color information. For example, one or more processors 110 may be configured to utilize or activate one or more machine learning resources 140, including the image position matcher 142 and the color composer 144, to generate the third image 3300. Although not illustrated in FIG. 4, in some implementations, one or more processors 110 may be configured to utilize or activate one or more machine learning resources 340, including the image position matcher 342 and the color composer 344, from the server computing system 300 to generate the third image 3300. For example, the image position matcher 342 and the color combiner 344 may perform operations similar to the image position matcher 142 and the color combiner 144 on the first image 3100 and the second image 3300, respectively.

[0080] For example, a computational process (e.g., a machine learning algorithm implemented by the image position matcher 142) may align the second image 3200 (which is a relatively low resolution, color image, such as an RGB image) to the first image 3100 (which is a relatively high resolution, monochrome image). For example, the image position matcher 142 may be configured to perform the alignment process by converting the second image 3200 to a monochrome image and calculating the alignment and confidence field between the first image 3100 (which is a monochrome image captured by the monochrome camera 182) and the converted monochrome image. For example, the image position matcher 142 may implement a neural network to find corresponding pixels between the first image 3100 and the second image 3200 and assign (i.e., map) pixels between the first image 3100 and the second image 3200. As described above, the alignment process may be performed by the camera system 100 or may be performed remotely (e.g., by the server computing system 300). For example, the image alignment 142 can be configured to perform the alignment process using known alignment methods, e.g., the alignment process can be performed in real time or as part of a post-processing procedure.

[0081] The image alignment 142 may be configured to determine that some portions of the second image 3200 are aligned with corresponding portions of the first image 3100. For these aligned portions, one or more of the processors 110 and / or the color combiner 144 may be configured to transfer color information from the second image 3200 (i.e., a low-resolution color (RGB) image) for the successfully aligned portions of the first image to the first image (i.e., a high-resolution monochrome image). For example, color information may be transferred (e.g., interpolated) from the portions of the second image 3200 (i.e., a low-resolution color (RGB) image) that correspond to the successfully aligned portions of the first image 3100 (i.e., a high-resolution monochrome image). For example, the transfer or merging operation may include creating a lower (e.g., half-resolution) chroma (e.g., CbCr) channel for the first image 3100 (i.e., a high-resolution monochrome image) having a Y or luma component. The Y or luma component of the first image 3100 may retain its resolution and may remain unchanged.

[0082] The image alignment 142 may be configured to determine that some portions of the second image 3200 are not aligned with corresponding portions of the first image 3100. For the non-aligned portions of the first image 3100, one or more processors 110 and / or the color synthesizer 144 may be configured to synthesize (generate) color information in case of unsuccessful alignment. In some implementations, a computational process (e.g., a machine learning algorithm implemented by the color synthesizer 144) may synthesize (generate) color information for portions of the first image 3100 (i.e., a high-resolution monochrome image) that have failed to be aligned (e.g., due to occlusion or insufficient signal-to-noise ratio). Alignment may be considered to have failed when a confidence level for the alignment performed by the image alignment 142 is below a threshold level (e.g., if a pixel of the first image 3100 does not map to a pixel of the second image 3200 or if the uncertainty level for the mapping is greater than a threshold level). For example, colors for the unsuccessfully aligned portions of the first image 3100 can be synthesized (e.g., generated) by referring to training examples for the neural network or by referring to colors of portions of the first image 3100 adjacent to the unaligned portions. For example, training data for the neural network implemented by the color synthesizer 144 can be obtained by (1) having the same RGB camera capture a first test image and a second test image of the scene, (2) synthesizing a grayscale image from the first test image to simulate the first image 3100 and reducing the size of the second test image to simulate the second image 3200, and (3) running a machine learning algorithm to synthesize color information for the unaligned portions between the first test image and the second test image, and comparing the synthesized color information with the original first test image (which serves as ground truth), such that the machine learning algorithm accurately outputs the synthesized color information and / or the confidence level of the output synthesized color information is greater than a threshold level.In some implementations, the luma component of the first image 3100 remains unchanged and retains its resolution, i.e., only the color channels are synthesized, and the resolution of the color channels can be halved, i.e., the color channels can be synthesized at half the resolution of the second image 3200.

[0083] The camera system 4000 can be configured to generate a resultant image (e.g., a third image 3300) based on the first image 3100 and the second image 3200 (e.g., based on the luma component of the first image 3100 and the color components of the second image 3200) in accordance with color information transferred in operation 4100 to the successfully aligned portions of the first image 3100, and in accordance with color information synthesized by the color synthesizer 144 for the non-aligned portions of the first image 3100 where alignment is not successful.

[0084] In some implementations, the camera system 4000 can be configured with a neural network trained to optimize the entire process shown in Figure 4 such that the inputs to the network are the first image 3100 and the second image 3200 (i.e., the monochrome and color images) and the output is the third image 3300 (i.e., the combined high-resolution RGB image). In this implementation, the alignment and color transfer operations are not separate operations because the neural network performs the process end-to-end with alignment performed implicitly internally.

[0085] Referring to Figure 5, a flow diagram of an exemplary non-limiting computer-implemented method according to one or more examples of the present disclosure is shown. The flow diagram in Figure 5 illustrates a method 5000 for generating an image through images captured by a monochrome camera 182 with a global shutter (i.e., a global shutter sensor) and a color camera 184 with a global shutter (i.e., a global shutter sensor).

[0086] At 5100, the method includes capturing a first image of a scene with a monochrome camera having a global shutter. For example, monochrome camera 182 can include a global shutter and can be configured to capture an image of the scene. For example, monochrome camera 182 can be mounted on a vehicle and capture images of an environment (e.g., at night or during the day and under various lighting conditions, including artificial lighting conditions). As another example, monochrome camera 182 can be implemented in a security system and capture images of an environment (e.g., at night or during the day and under various lighting conditions, including artificial lighting conditions).

[0087] At 5200, the method includes capturing a second image of the scene by a color camera having a global shutter. For example, color camera 184 can include a global shutter and can be configured to capture an image of the scene. For example, color camera 184 can be mounted on a vehicle and capture images of the environment (e.g., at night or during the day and under various lighting conditions, including artificial lighting conditions). As another example, color camera 184 can be implemented in a security system and capture images of the environment (e.g., at night or during the day and under various lighting conditions, including artificial lighting conditions). Color camera 184 can be disposed away from monochrome camera 182, but color camera 184 can be disposed adjacent to monochrome camera 182 and face the same direction. For example, monochrome camera 182 and color camera 184 can be positioned within a few centimeters of each other (e.g., 3 to 10 centimeters, such as less than 5 centimeters) to reduce or prevent static parallax between images captured by monochrome camera 182 and color camera 184. For example, the monochrome camera 182 and the color camera 184 can be mutually aligned along at least one axis. For example, the monochrome camera 182 and the color camera 184 can be mutually aligned along a vertical axis (e.g., A1 in FIG. 2A) or a horizontal axis (e.g., A2 in FIG. 2B). In addition, the monochrome camera 182 and the color camera 184 can be mutually aligned along a second axis, such as along a depth axis perpendicular to the vertical and horizontal axes.

[0088] At 5300, the method includes aligning the second image to the first image. For example, the image alignment 142 may implement a computational process. For example, the image alignment 142 may be configured to align the second image 3200 (which is a relatively low resolution, color image, such as an RGB image) to the first image 3100 (which is a relatively high resolution, monochrome image) in the manner discussed with respect to FIG.

[0089] In 5400, the method includes obtaining a third image representing the scene by providing color information of the second image to the first image. For example, as discussed with respect to FIG. 4, the image alignment 142 can be configured to determine that some portions of the second image 3200 are aligned with corresponding portions of the first image 3100. For these aligned portions, the one or more processors 110 and / or the color compositor 144 can be configured to transfer color information from the second image 3200 (i.e., a low-resolution color (RGB) image) to the first image (i.e., a high-resolution monochrome image) for the successfully aligned portions of the first image. For example, as discussed with respect to FIG. 4, the image alignment 142 can be configured to determine that some portions of the second image 3200 are not aligned with corresponding portions of the first image 3100. For non-aligned portions of the first image 3100, one or more processors 110 and / or color combiner 144 may be configured to synthesize (generate) color information for the unaligned portions of the first image 3100. The third image 3300 may represent a scene and may correspond to the first image 3100 with a Y or luma component (having the same resolution and unchanged) from the first image 3100 and two or more chroma components or channels obtained through the second image 3200 according to the alignment process and / or color synthesis process described herein. The two or more chroma components or channels may also have a lower resolution than the resolution of the second image 3200.

[0090] Aspects of the exemplary embodiments described above can be recorded in a non-transitory computer-readable medium that includes program instructions for implementing various computer-implemented operations. The medium can also include data files, data structures, and the like, alone or in combination with the program instructions. Examples of non-transitory computer-readable media include magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD ROM disks, Blu-ray disks, and DVDs; magneto-optical media, such as optical disks; and other hardware devices specially configured to store and execute program instructions, such as semiconductor memory, read-only memory (ROM), random access memory (RAM), flash memory, USB memory, and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher-level code that can be executed by a computer using an interpreter. The program instructions can be executed by one or more processors. The hardware devices described herein can also be configured to function as one or more software modules to perform the operations of the embodiments described above, and vice versa. Additionally, the non-transitory computer-readable storage medium may be distributed among computer systems connected over a network, and the computer-readable code or computer-readable program instructions may be stored and executed in a decentralized manner. Additionally, the non-transitory computer-readable storage medium may be embodied within at least one application specific integrated circuit (ASIC) or field programmable gate array (FPGA).

[0091] Each block in the flowchart diagrams may represent a unit, module, segment, or portion of code that includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may be performed out of order. For example, depending on the functionality involved, two blocks shown in succession may in fact be performed substantially in parallel (concurrently), or the blocks may sometimes be performed in the reverse order.

[0092] Although the present disclosure has been described above with respect to various exemplary embodiments, each example is provided as an explanation of the present disclosure and not as a limitation of the present disclosure. Those skilled in the art will be able to easily create alternatives, variations, and equivalents of such embodiments upon gaining an understanding of the foregoing. Thus, the present disclosure is not intended to exclude modifications, variations, and / or additions to the disclosed subject matter that would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to produce yet another embodiment. Thus, it is intended that the present disclosure encompass such alternatives, variations, and equivalents. [Explanation of symbols]

[0093] 100 Camera System 110 Processor 120 Memory Devices 122 Data 124 Command 130 Imaging Applications 140 Machine Learning Resources 142 Image position aligner 144 Color synthesizer 150 Input Devices 160 Output Devices 166 Vibration Device 170 Display Devices 180 Sensors 182 Monochrome Camera 184 Color Camera 200 Network 300 Server Computing System 310 Processor 320 Memory Devices 322 Data 324 Command 330 Imaging Applications 340 Machine Learning Resources 342 Image position aligner 344 Color Synthesizer 400 External Computing Devices 1000 Systems 2100 Camera System 2200 Camera System 3000 Camera System 3100 First Image 3200 Second Image 3300 3rd image, 2nd image 4000 Camera System 4100 operation 5000 ways A1 axis A2 axis d1 Distance

Claims

1. a monochrome camera having a global shutter and configured to capture a first image of the scene; a color camera disposed remotely from the monochrome camera, the color camera having a global shutter and configured to capture a second image of the scene; registering the second image to the first image; and providing color information of the second image to the first image to obtain a third image representative of the scene; one or more processors configured to A camera system comprising:

2. The camera system of claim 1 , wherein the monochrome camera and the color camera are synchronized to capture the first image and the second image substantially simultaneously.

3. The camera system of claim 2 , wherein the first image and the second image are captured when peak lighting conditions occur within an environment in which the monochrome camera and the color camera are disposed, respectively.

4. The camera system of claim 1 , wherein the monochrome camera and the color camera are arranged to face the same direction and are arranged less than a threshold distance from each other.

5. The camera system of claim 4 , wherein the threshold distance is less than or equal to 10 centimeters.

6. 2. The camera system of claim 1, wherein the one or more processors are configured to provide color information of the second image to the first image for a portion of the second image that is aligned with a corresponding portion of the first image by transferring color information of the portion of the second image to the corresponding portion of the first image.

7. 7. The camera system of claim 6, wherein the one or more processors are configured to identify non-aligned portions of the first image in which portions of the second image are not aligned with corresponding portions of the first image, and to apply synthesized color information to the non-aligned portions of the first image.

8. The camera system of claim 1 , wherein the color camera is a red-green-blue (RGB) camera.

9. The camera system of claim 1 , wherein the color camera has a lower resolution than the monochrome camera.

10. The camera system of claim 1 , wherein the monochrome camera has a larger size than the color camera.

11. the first image includes a luma component; the third image includes the luma component of the first image and a chroma component based on the second image; 2. The camera system of claim 1.

12. a first machine learning resource for aligning the second image to the first image; and a second machine learning resource for synthesizing color information for a non-aligned portion of the first image, where a portion of the second image is not aligned to a corresponding portion of the first image, and applying the synthesized color information to the non-aligned portion of the first image. The camera system of claim 1 , further comprising:

13. capturing a first image of a scene with a monochrome camera having a global shutter; capturing a second image of the scene with a color camera disposed remotely from the monochrome camera and having a global shutter; registering the second image to the first image; obtaining a third image representative of the scene by providing color information of the second image to the first image; 4. A computer-implemented method comprising:

14. the first image includes a single channel having a luma component; the third image includes at least three channels, including a first channel having the luma component and a plurality of channels including chroma components based on the second image; The method of claim 13.

15. the monochrome camera and the color camera are synchronized to capture the first image and the second image substantially simultaneously; wherein the steps of capturing the first image and the second image each occur at a peak of an illumination cycle. The method of claim 13.

16. The method of claim 15 , wherein the color camera and the monochrome camera have substantially the same field of view.

17. 14. The method of claim 13, wherein a difference between a time when the color camera captures the second image and a time when the monochrome camera captures the first image is less than an integration time of the monochrome camera.

18. 14. The method of claim 13, wherein providing the color information of the second image to the first image includes, for a portion of the second image that is aligned with a corresponding portion of the first image, transferring color information of the portion of the second image to the corresponding portion of the first image.

19. identifying non-registered portions of the first image, where portions of the second image are not registered with corresponding portions of the first image; synthesizing, by a machine learning resource, color information for the non-aligned portion of the first image; applying the combined color information to the non-registered portion of the first image; 20. The method of claim 18, further comprising:

20. A non-transitory computer-readable medium storing instructions executable by one or more processors of a camera system, the instructions comprising: instructions to cause a monochrome camera with a global shutter to capture a first image of a scene; instructions for causing a color camera having a global shutter and spaced less than a threshold distance from the monochrome camera to capture a second image of the scene, the second image having a lower resolution than the first image; instructions for registering the second image to the first image; obtaining a third image representing the scene by providing color information of the second image to the first image. A non-transitory computer readable medium comprising:

Images