Resolution enhancement in space-frequency space

Abstract

A camera system includes a lamp configured to emit light, a lamp driver configured to energize the lamp, an image sensor array configured to acquire image data, a lenslet array, and an image engine. The lenslet array includes a plurality of lenslets disposed laterally over the image sensor array and configured to focus light reflected from the object onto the image sensor array. The image engine is configured to receive image data from the image sensor array, parse the image data into a plurality of component images, and return an enhanced image based on the plurality of component images, the enhanced image having an enhanced spatial resolution relative to any component image.

Description

Background Art

[0001] Over the past 20 years, digital imaging technology has evolved at a remarkable rate. High-resolution, high-speed, high-sensitivity digital image sensors are now available at low cost. Digital image sensors are used, for example, in color cameras, as well as in a variety of other electronic devices targeting the consumer market. Importantly, digital image sensing is not limited to color camera implementations, but can be used for eye tracking, infrared machine vision, depth vision, and night vision, among others. Summary of the invention

[0002] One aspect of the present disclosure relates to a camera system, which includes a lamp configured to emit light, a lamp driver configured to energize the lamp, an image sensor array configured to acquire image data, a lenslet array, and an image engine. The lenslet array includes a plurality of lenslets, the plurality of lenslets being arranged laterally above the image sensor array and configured to focus light reflected from an object onto the image sensor array. The image engine is configured to (a) receive image data from the image sensor array, (b) parse the image data into a plurality of component images, and (c) return an enhanced image based on the plurality of component images, the enhanced image having an enhanced spatial resolution relative to any component image.

[0003] Another aspect of the present disclosure relates to a method comprising: energizing a light of a camera system that emits light onto an object; acquiring image data that records light reflected from the object and focused by an array of small lenses arranged laterally above an image sensor array; parsing the image data into multiple component images; and returning an enhanced image based on the multiple component images, the enhanced image having enhanced spatial resolution relative to any of the component images.

[0004] This summary is provided to introduce a selection of concepts that are further described in the detailed description in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 Aspects of an example electronic device are shown.

[0006] Figure 2A and Figure 2B Aspects of an example camera system are shown from different perspectives.

[0007] Figure 3 Aspects of example machine vision methods that may be enacted by a camera system or an electronic device in which a camera system is installed are shown.

[0008] Figure 4 Aspects of an example up-conversion method for merging component images into an enhanced image are shown.

[0009] Figure 5 An example overlay of Fourier magnitude maps based on component images acquired simultaneously under single point illumination is illustrated.

[0010] Figure 6 yes Figure 4 Schematic diagram of various aspects of the method.

[0011] Figure 7 Aspects of an example computer system are shown. DETAILED DESCRIPTION

[0012] Camera systems are used in many different kinds of electronic devices. Two contributors to the cost and size of a camera system are (a) the physical aperture size, and (b) the number of sensor elements of the image sensor array. Larger physical apertures require larger objective lenses, which are expensive to manufacture, and providing additional sensor elements naturally increases the cost and size of the image sensor array. However, the physical aperture size and number of sensor elements also control the spatial resolution of the camera system - the number of pixels that the image sensor array returns (with an acceptable signal-to-noise ratio) per image capture.

[0013] The present disclosure provides a solution for increasing the spatial resolution of a camera system with a limited physical aperture size and a limited number of sensor elements. The camera system disclosed herein acquires multiple component images that capture light reflected from the same object, but are shifted in angular space. The shift is provided in part by an array of lenslets positioned above an image sensor array rather than a single objective lens. Due to the offset positioning of the individual lenslets, the lenslets receive light reflected from the same object trajectory at different angles, and the difference is recorded in component images acquired simultaneously on different quadrants of the image sensor array. When acquiring a sequence of component images, additional shifts in angular space are achieved by dynamically changing the illumination angle of the object. To this end, the camera system may include a series of LED lights arranged in parallel with the image sensor array; the lights may be energized one at a time or in any order synchronized with the component image capture.

[0014] When acquiring multiple component images, the camera system's image engine transforms each component image into the spatial frequency domain and fuses them into a combined image with a wider spatial frequency content than any individual component image. Finally, the image engine inversely transforms the combined image back to the spatial domain to obtain an image with enhanced spatial resolution.

[0015] The solution presented herein can be applied to a variety of devices with limited physical aperture size and / or limited number of sensor elements. Specific applications include rear camera systems on handheld and wearable devices that image the user's eyes to support iris recognition and eye tracking.

[0016] Turning now to the accompanying drawings, Figure 1 Aspects of an example electronic device 10 are shown. The electronic device includes a camera system 12 configured to image an object 14. In the illustrated example, the device 10 is a near-eye display device, and the object 14 includes a human eye. More generally, electronic devices consistent with the present disclosure include smartphones, tablet computers, laptop computers, gaming systems, digital cameras, virtual and augmented reality systems, and various other devices. In the device 10, the camera system may be configured for functions such as gaze tracking and / or iris recognition; in other devices, the camera system may be configured for other purposes.

[0017] The camera system 12 may be configured for color imaging, monochrome imaging, hyperspectral (ultraviolet, infrared, or near infrared) imaging, and / or depth imaging. Contemplated depth imaging variants include structured light imaging, time of flight (ToF) imaging, stereoscopic imaging, and light field imaging. In some examples, the camera system 12 may be configured to acquire a time-resolved sequence (i.e., video) of images of an object in rapid succession.

[0018] Figure 2A and Figure 2B Aspects of an example camera system 12 are shown from different perspectives. The camera system includes an image sensor array 16 configured to acquire image data. The image sensor array includes a rectangular matrix of sensor elements 18, each sensor element responding to an incident photon flux. In some examples, the image sensor array is a complementary metal oxide silicon (CMOS) array. In a non-limiting example, the image sensor array includes 2560 horizontal rows and 1920 vertical columns of sensor elements. In some examples, each sensor element may include a CMOS photodiode. The camera system 12 includes a row driver 20 and a readout circuit 22 electrically coupled to the image sensor array. The row driver 20 is configured to select a row of the image sensor array 16 to be read in a current readout operation. The readout circuit 22 is configured to output an electronic signal that varies depending on the incident photon flux received by the sensor element of the selected row. To this end, the readout circuit can be configured to amplify and digitize the charge sensed along each column of the image sensor array. The readout circuit can include analog and digital component portions - for example, multiple transconductance amplifier analog-to-digital converters. In some examples, the readout circuitry may include application specific integrated circuit (ASIC) componentry.

[0019] In the illustrated example, the image sensor array 16 is operatively coupled to an image engine 24 via a readout circuit 22. The image engine may be an onboard or networked controller for the camera system 12 or an electronic device in which the camera system is mounted. In some examples, the image engine may include a microprocessor and associated computer memory. In some examples, the image engine may include a system on a chip (SoC), etc.

[0020] The camera system 12 includes a lenslet array 26 having a plurality of lenslets 28 arranged laterally above the image sensor array 16 and configured to focus light reflected from the object 14 onto the image sensor array. Each lenslet is a lens of appropriate size and configuration. For ease of illustration, the lenslet array is arranged laterally above the image sensor array 16 and configured to focus light reflected from the object 14 onto the image sensor array. Figure 2B , is represented as an opaque layer; in fact, the lenslet array is substantially transparent at least for the wavelengths imaged by the camera system. The lenslet array typically includes a rectangular matrix of replicated lenslets. In some examples, the rectangular matrix is ​​a two-by-two matrix. Each lenslet 28 of the lenslet array is configured to focus light reflected from the object onto a corresponding area 30 (e.g., a quadrant) of the image sensor array. Assuming that the object is point-illuminated (see below), the light reflected from a given trajectory of the object is received by each lenslet at a slightly different angle and is therefore focused onto non-corresponding sensor elements of the corresponding area. Therefore, the replicated sub-images captured simultaneously on several areas 30 are shifted relative to each other.

[0021] The camera system 12 includes a series of lamps 32 and a lamp driver 34, each lamp 32 being configured to emit light when energized (i.e., electrically biased), and the lamp driver 34 being configured to energize the lamp. In some examples, each lamp includes one or more light emitting diodes (LEDs). The emission spectrum of the lamp is not particularly limited. For color imaging applications, the lamps may include blue, green, and red LEDs or combinations thereof. For depth imaging and / or near-eye applications, the lamps may include infrared (IR) or near-infrared (NIR) LEDs. In some examples, the image sensor array 16 may include a wavelength filter configured to match the emission spectrum of the lamp. Typically, each lamp has a Lambertian emission profile, which is acceptable for implementations herein. If desired, appropriate focusing or beam shaping optics may be coupled to any, some, or all of the lamps to change the emission profile.

[0022] In the illustrated example, the series of lights includes eight lights distributed around the image sensor array 16 in a plane parallel to the image sensor array. In other examples, the lights can be distributed on different planes, which may not necessarily be parallel to the image sensor array. In some examples, each lamp can be oriented so that the main ray of its emission profile is perpendicular to the plane in which the lamp is located. In other examples, any, some or all of the lights can be oriented so that the main ray is tilted relative to the plane to further change the angular range of light emitted from the (multiple) lamps. Other camera systems also include additional lights, fewer lights, lights that are arranged differently, and so on. In some examples, the lamp driver 34 is configured to sequentially energize a series of lights so that the illumination angle of the object changes as a function of time during image data acquisition. For example, one lamp can be energized at a time.

[0023] Due to its unique position, each lamp 32 is configured to illuminate a given trajectory of the object 14 from a different illumination angle, causing the light to be reflected onto the lenslet array 26 at different reflection angles and thus be focused onto different sensor elements. Although even if only one objective lens is placed above the image sensor array, shifts in the responding sensor elements will be observed for different illumination points, the arrangement of M lenslets converts each such shift into M mutually non-parallel shifts, each of which is useful for expanding the effective aperture of the camera system. Among other advantages, the lenslet array reduces the number of individually driven lamps required for a given increase in effective aperture size or image resolution.

[0024] In some contexts, the term "resolution" is used to quantify only the number of pixels of a digital image. However, in the present disclosure, "resolution" refers to the level of granularity of the information content of a digital image, regardless of the number of pixels used to represent the image. For example, the resolution of an image acquired by a camera system may be limited by the size of the physical aperture of the camera system and / or the number of sensor elements of the image sensor array. As used herein, upsampling an image so that the same information spans more pixels does not increase or enhance "resolution". In this context, an appropriate quantifier of image resolution is the width of the spatial frequency range representing the information content of the image - that is, the frequency at which the signal-to-noise ratio exceeds an appropriate threshold. Therefore, increasing or enhancing the resolution of an image does not necessarily require (but does not prohibit) a corresponding increase in pixel count. For example, a 500×500 pixel image before resolution enhancement may include an information signal in noise that drops above a certain spatial frequency. The corresponding image after resolution enhancement may still be a 500×500 pixel image, but one in which the information signal is extended to a higher spatial frequency before dropping to the noise.

[0025] To provide digital processing for the solutions herein, the image engine 24 of the camera 12 is configured to (a) receive image data from the image sensor array, (b) parse the image data into multiple component images, and (c) return an enhanced image based on the multiple component images, the enhanced image having an enhanced spatial resolution relative to any of the component images. The term "image data" herein refers to any digital image acquired on the image sensor array 16, which may include (due to the lenslet array 26) a tile of multiple component images of the same object. The term "image data" also refers to any sequence of digital images of the same object, which may also include multiple component images.

[0026] In some examples, the multiple component images resolved by the image engine 24 include images acquired simultaneously over different regions 30 of the image sensor array 16. In general, the number of component images acquired simultaneously is equal to M, the number of lenslets 28 in the lenslet array 26. In the illustrated example, where the lenslet array includes four lenslets, each lenslet being arranged over a respective region 30 of the image sensor array, illumination from a given lamp 32 provides four component images.

[0027] In the example where the lamps 32 are powered on sequentially, the image data received by the image engine includes, for each region 30, a sequence of component images, where each component image is acquired when a different lamp or combination of lamps is powered on. Thus, in a scenario where each of the N lamps is powered on one at a time, a total of M×N component images may be resolved—M component images are acquired simultaneously for each lamp.

[0028] Figure 3 Aspects of an example machine vision method 36 that may be enacted by the camera system 12 or a device in which the camera system is installed are shown.

[0029] At 38A of method 36, a light driver of the camera system energizes the light, which emits light onto the subject. In examples where a series of lights are provided, the lights may be energized in any desired order—e.g., one at a time. More specifically, the light driver may cycle through a schedule of intervals, energizing only one or more lights associated with the interval during each interval, and then advancing to the next interval.

[0030] At 38B, the image sensor array acquires image data that records light reflected from the object and focused via a lenslet array that includes a plurality of lenslets arranged laterally above the image sensor array. In an example where the lamp driver cycles through an interval schedule, as described above, an image may be acquired during each interval, thereby producing a sequence of acquired images, wherein each image in the sequence corresponds to a lamp or combination of lamps being powered on. At 38C, the image engine receives image data from the image sensor array.

[0031] At 38D, the image engine resolves the image data into multiple component images. Component image resolution is achieved in two stages. The first stage applies to examples where the lamp driver cycles through an interval schedule and acquires one image during each interval. The first stage includes assigning each image of the sequence to a corresponding lamp or combination of lamps. The second stage includes spatially splitting each image into multiple component images based on the lenslet geometry of the lenslet array.

[0032] At 38E, the image engine transforms each of the plurality of component images into the space-frequency domain. Each image acquired by the image sensor array is a function of horizontal and vertical space coordinates, i.e., sensor element indices X and Y. Any function of the space coordinates may alternatively be represented as a function of the space-frequency coordinates, e.g., 2πC / X and 2πR / Y, where C and R are the number of columns and rows of the image sensor array. The terms "transformed into the space-frequency domain" and "transformed into the space-frequency space" herein refer to the action of calculating the alternative representation. Typically, the space-frequency representation is a Fourier representation, but other representations including sine, cosine and / or wavelet basis functions may also be used. For a representation including a real part and an imaginary part, such as a Fourier representation, the term "amplitude" refers to the modulus of the real part and the imaginary part. In some examples, a "fast Fourier transform" may be used to transform one or more component images into the space-frequency domain. In other examples, a Fresnel transform may be used.

[0033] At 38G, the image engine combines the plurality of component images in a spatial-frequency space to obtain a combined image having a wider spatial frequency than any of the component images. In some examples, combining the plurality of component images includes accumulating spatial frequency content from each component image into the combined image using a phase estimate derived from the enhanced image. Figure 4 Describe this aspect.

[0034] At 38H, the image engine inverse transforms the combined image to obtain an enhanced image having enhanced spatial resolution relative to any component images. In other words, the combined image, originally represented as a function of space-frequency coordinates, is transformed into an equivalent function of horizontal and vertical space coordinates (e.g., X and Y). To this end, an inverse Fourier transform, an inverse sine, cosine, or wavelet transform, or an inverse Fresnel transform may be applied. In some examples, the effective aperture size for the enhanced image exceeds the physical aperture size of the lenslet array. At 38I, the image engine returns the enhanced image. At optional step 38J, iris recognition is performed based on the enhanced image. Iris recognition can be performed by a dedicated iris recognition engine of the camera system or of the device in which the camera system is installed.

[0035] Figure 4 Aspects of a more specific up-conversion method 40 are shown for combining component images into an enhanced image, ie, an image having enhanced spatial resolution relative to any of the component images.

[0036] At 42A of method 40 , the image engine computes a Fourier magnitude map and a Fourier phase map for each component image of the object. Figure 5 An example overlay of Fourier magnitude maps based on component images is illustrated. In the illustrated example, the component images are acquired simultaneously under single point illumination. Thus, the four overlapping circles correspond to Fourier magnitude maps derived from the four quadrants (regions 30A to 30D) of the image sensor array 16. However, as described above, the same principles apply to the sequentially acquired component images. Figure 6 Accompanying schematic diagrams of some aspects of method 40 are provided.

[0037] At 42B, the image engine attenuates (e.g., to zero) all Fourier amplitude values ​​outside the spatial frequency response function defined by the bandwidth of the physical aperture of the camera system and the spatial-frequency offset shift based on the illumination angle. Attenuation removes Fourier transform artifacts and residues that are outside the system bandwidth. These frequency constraints are iteratively applied during phase recovery and are placed in the correct position in the higher resolution frequency space (see below).

[0038] At 42C, the image engine initializes the phase retrieval algorithm. In some examples, the initialization includes: creating an initial combined image by upsampling one of the component images at 42D; and creating initial Fourier magnitude and Fourier phase maps for the initial combined image at 42E and 42F. These three "high resolution" data sets provide a working space in which the lower resolution content from the component images will be iteratively merged.

[0039] At 42G, iterative phase retrieval begins and the image engine applies a frequency constraint to the first component image. At 42H, the image engine updates the combined image by formulating an inverse Fourier transform of the updated spectrum applying the latest constraint and using the current Fourier phase map of the combined image. As each new spatial frequency loop is added to the higher resolution FFT space, the constraint is imposed in the overlapping regions of the transformed component images. Figure 6 One of the six overlapping regions 44 in the higher resolution FFT space is highlighted. For each of the overlapping component images, the corresponding phases in the overlapping region should be the same. Thus, as the content from each additional transformed component image is added to the high resolution image, the higher resolution phases are guided to converge. Steps 42G and 42H are now repeated one at a time using the frequency constraints for the remaining component images.

[0040] At 42H, the image engine determines whether the Fourier phase map of the combined image has stabilized to the desired state. If so, the iteration stops and the image engine returns the current combined image as the enhanced image.

[0041] No aspect of the above drawings or descriptions should be construed in a limiting sense, as many variations, extensions, and omissions are contemplated. For example, while the illustrated camera system includes a series of lights that can be powered one at a time, the lights can be powered in any suitable combination. In other examples, the camera system can include only one light so that the illumination angle of the object is constant.

[0042] The methods herein may be bound to a computer system of one or more computing devices. Such methods and processes may be implemented as applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0043] Figure 7 A schematic representation of a computer system 50 configured to provide some or all of the computer system functions disclosed herein is provided. Computer system 50 includes a logic system 52 and a computer memory system 54. Computer system 50 may optionally include a display system 56, an input system 58, a network system 60, and / or other systems not shown in the drawings.

[0044] The logic system 52 includes one or more physical devices configured to execute instructions. For example, the logic system may be configured to execute instructions as part of at least one operating system (OS), application, service, and / or other program construction. The logic system may include at least one hardware processor (e.g., a microprocessor, a central processing unit, a central processing unit (CPU), and / or a graphics processing unit (GPU)) configured to execute software instructions. Additionally or alternatively, the logic system may include at least one hardware or firmware device configured to execute hardware or firmware instructions. The processor of the logic system may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The individual components of the logic system may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing.

[0045] The computer memory system 54 includes at least one physical device configured to temporarily and / or permanently store computer system information, such as data and instructions executable by the logic system 52. When the computer memory system includes two or more devices, the devices may be collocated or remotely located. The computer memory system 54 may include at least one volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location read addressable, file read addressable, and / or content read addressable computer memory device. The computer memory system 54 may include at least one removable and / or built-in computer memory device. As the logic system executes instructions, the state of the computer memory system 54 may be transformed—for example, to store different data.

[0046] Aspects of logic system 52 and computer memory system 54 are integrated together into one or more hardware-logic components. For example, any such hardware-logic component may include at least one program or application specific integrated circuit (PASIC / ASIC), program or application specific standard product (PSSP / ASSP), system on chip (SOC), or complex programmable logic device (CPLD).

[0047] The logic system 52 and the computer memory system 54 can collaborate to illustrate one or more logic machines or engines. As used herein, the terms "machine" and "engine" each collectively refer to a combination of any other components that collaborate hardware, firmware, software, instructions, and / or provide computer system functions. In other words, machines and engines are never abstract concepts and always have a tangible form. A machine or engine can be illustrated by a single computing device, or a machine or engine can include two or more subcomponents illustrated by two or more different computing devices. In some implementations, a machine or engine includes a local component (e.g., a software application executed by a computer system processor) that collaborates with a remote component (e.g., a cloud computing service provided by a network of one or more server computer systems). The software and / or other instructions that give a specific machine or engine its functionality can be optionally stored on one or more computer memory devices as one or more unexecuted modules.

[0048] The machines and engines (as used throughout the above description) may be implemented using any suitable combination of machine learning (ML) and artificial intelligence (AI) techniques. Non-limiting examples of techniques that may be incorporated into the implementation of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., spatial convolutional networks for processing images and / or videos, and / or any other suitable convolutional neural network configured to convolve and aggregate features across one or more temporal and / or spatial dimensions), recursive neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), and / or graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases).

[0049] When included, the display system 56 can be used to present a visual representation of the data stored by the computer memory system 54. In some examples, the visual representation can take the form of a graphical user interface (GUI). The display system can include one or more display devices utilizing virtually any type of technology. In some implementations, the display system can include one or more virtual, augmented, or mixed reality displays.

[0050] When included, the input system 58 may include or interface with one or more input devices. The input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, or a touch screen.

[0051] When included, the network system 60 can be configured to communicatively couple the computer system 50 with one or more other computer systems. The network system can include wired and / or wireless communication devices compatible with one or more different communication protocols. The network system can be configured for communication via a personal, local area network, and / or wide area network.

[0052] In summary, one aspect of the present disclosure relates to a camera system, which includes a lamp configured to emit light, a lamp driver configured to energize the lamp, an image sensor array configured to acquire image data, a lenslet array, and an image engine. The lenslet array includes a plurality of lenslets that are arranged laterally above the image sensor array and configured to focus light reflected from an object onto the image sensor array. The image engine is configured to receive image data from the image sensor array, parse the image data into a plurality of component images, and return an enhanced image based on the plurality of component images, the enhanced image having an enhanced spatial resolution relative to any component image. As described above herein, the lenslet array provides a technical benefit of enabling multiple component images of different reflective angle contents to be captured simultaneously. Thus, fewer lamps and less time can be used to fill the reflective angle space than if only one objective lens is used. This feature reduces the cost and complexity of the camera system by reducing the number of lamps, and improves performance by reducing the total time to acquire an enhanced resolution image. The image engine provides a technical benefit of forming an image with a higher resolution than an image that may normally have a fixed physical aperture size and the number of sensors of the image sensor array.

[0053] In some implementations, each lenslet of the lenslet array is configured to focus light onto a different area of ​​the image sensor array, and the multiple component images include images acquired simultaneously on different areas of the image sensor array. In some implementations, the lamp is one of a series of lamps, the lamp driver is configured to sequentially power on the series of lamps, and the image data includes a corresponding sequence of component images acquired when different lamps or combinations of lamps are powered on. This feature provides additional technical benefits of expanding the number of component images beyond the number of lenslets of the lenslet array and expanding the reflection angle space spanned by the component images. This further improves the spatial resolution of the enhanced image. In some implementations, each lamp in the series of lamps is configured to illuminate the trajectory of the object from a different illumination angle, so that the light is reflected onto the lenslet array at different reflection angles. In some implementations, the series of lamps includes N lamps, the lenslet array includes M lenslets, and the multiple component images include M×N component images. In some implementations, the image engine is further configured to: transform each of the plurality of component images into a spatial-frequency domain; combine the plurality of component images in the spatial-frequency space to obtain a combined image having a wider spatial frequency than any of the component images; and inversely transform the combined image to obtain an enhanced image. These features provide a technical benefit of formulating a data transformation process (resolution enhancement) using efficient applied mathematics. In particular, the combination of the transformed component images in the spatial-frequency domain is more efficient than any hypothetical strategy formulated in pixel space. The improvement in processing efficiency makes the entire method a practical method - limiting the use of computer resources, etc. In some implementations, combining the plurality of component images includes accumulating spatial-frequency content from each component image into the combined image using a phase estimate derived from the enhanced image. In some implementations, the effective aperture size for the enhanced image exceeds the physical aperture size of the lenslet array. This feature provides an additional technical benefit of limiting the size and manufacturing cost of a camera system configured for a given spatial resolution. For the reasons described above in this article, the size of the physical aperture of the camera system drives both size and cost. In some implementations, the lenslet array includes a rectangular matrix of replicated lenslets. In some implementations, the rectangular matrix is ​​a two-by-two matrix. This feature provides an additional technical benefit of multiplying the number of component images obtainable from a single illumination point by four. In some implementations, the series of lights includes eight lights distributed around the image sensor array in a plane parallel to the image sensor array. This feature provides an additional technical benefit of multiplying the number of component images obtainable from a single image capture by eight. In some implementations, the series of lights is a series of light emitting diodes (LEDs), and the image sensor array is a complementary metal oxide silicon (CMOS) array. In some implementations, the camera system is arranged in a near-eye display device, and the object includes a human eye. In some implementations, the image engine is also configured to formulate iris recognition based on the enhanced image.This feature provides the additional technical benefit of using lower native resolution camera hardware resources (such as may be suitable for gaze tracking) to achieve resolution-demanding iris recognition tasks.

[0054] Another aspect of the present disclosure relates to a method comprising: energizing a light of a camera system, the light emitting light onto an object; acquiring image data, the image data recording light reflected from the object and focused by a lenslet array, the lenslet array comprising a plurality of lenslets arranged laterally above an image sensor array; parsing the image data into a plurality of component images, and returning an enhanced image based on the plurality of component images, the enhanced image having an enhanced spatial resolution relative to any of the component images. The lenslet array provides a technical benefit of enabling multiple component images of different reflective angle content to be captured simultaneously. Thus, fewer lights and less time can be used to fill the reflective angle space than if only one objective lens were used. This feature reduces the cost and complexity of the camera system by reducing the number of lights, and improves performance by reducing the total time to acquire an enhanced resolution image. The image engine provides a technical benefit of forming an image with a higher resolution than would normally be possible with a fixed physical aperture size and number of sensors of an image sensor array.

[0055] In some implementations, the method further includes transforming each of the plurality of component images into a spatial-frequency domain; combining the plurality of component images in the spatial-frequency space to obtain a combined image having a spatial frequency wider than any of the component images; and inversely transforming the combined image to obtain an enhanced image. These features provide a technical benefit of formulating a data transformation process (resolution enhancement) using efficient applied mathematics. In particular, the combination of the transformed component images in the spatial-frequency domain is more efficient than any hypothetical strategy formulated in pixel space. The improvement in processing efficiency makes the entire method a practical method - limiting the use of computer resources, etc. In some implementations, combining the plurality of component images includes accumulating spatial-frequency content from each component image into the combined image using a phase estimate derived from the enhanced image. In some implementations, the effective aperture size for the enhanced image exceeds the physical aperture size of the lenslet array. This feature provides an additional technical benefit of limiting the size and manufacturing cost of a camera system configured for a given spatial resolution. For the reasons described above in this article, the size of the physical aperture of the camera system drives both size and cost.

[0056] Another aspect of the present disclosure relates to a camera system, the camera system comprising a series of lamps configured to emit light, a lamp driver configured to energize the series of lamps, an image sensor array configured to acquire image data, and a lenslet array. The lenslet array comprises a plurality of lenslets arranged laterally above the image sensor array and configured to focus light reflected from an object onto the image sensor array. Image data from the image sensor array is provided to an image engine, the image engine being configured to resolve the image data into a plurality of component images, transform each of the plurality of component images into a spatial-frequency domain, combine the plurality of component images in the spatial-frequency space to obtain a combined image having a wider spatial frequency than any component image in the component images, inversely transform the combined image to obtain an enhanced image having an enhanced spatial resolution relative to any component image in the component images, and return the enhanced image. The lenslet array provides a technical benefit of enabling multiple component images of different reflective angle contents to be captured simultaneously. Thus, fewer lamps and less time can be used to fill the reflective angle space than if only one objective lens were used. This feature reduces the cost and complexity of the camera system by reducing the number of lamps, and improves performance by reducing the overall time to acquire an enhanced resolution image. The series of lamps provides the additional technical benefit of expanding the number of component images beyond the number of lenslets of the lenslet array and expanding the reflective angle space spanned by the component images. This further improves the spatial resolution of the enhanced image.

[0057] In some implementations, the lamp driver is configured to sequentially power on the series of lamps, each lamp in the series of lamps is configured to illuminate the trajectory of the object from a different illumination angle so that the light is reflected onto the lenslet array at different reflection angles, and the image data includes a corresponding sequence of images acquired when different lamps or combinations of lamps are powered on.

[0058] The present disclosure is presented by way of example and with respect to the accompanying drawings. Substantially identical components, process steps, and other elements may be identified in a coordinated manner and described with minimal repetition in one or more of the accompanying drawings. However, it should be noted that the coordinated identified elements may also be different to some extent. It should also be noted that the accompanying drawings are schematic and are not generally drawn to scale. Instead, the various drawing scales, aspect ratios, and numbers of components shown in the accompanying drawings may be intentionally distorted to make certain features or relationships easier to see.

[0059] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as limiting, because many variations are possible. The specific routine or method described herein can represent one or more processing strategies in any number of processing strategies. Thus, the various actions illustrated and / or described can be performed in the illustrated and / or described order, in other orders, in parallel or omitted. Equally, the order of the above process can be changed.

[0060] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A camera system comprising: a lamp configured to emit light; a lamp driver configured to energize the lamp; an image sensor array configured to acquire image data; a lenslet array comprising a plurality of lenslets arranged laterally above the image sensor array and configured to focus the light reflected from an object onto the image sensor array; as well as An image engine, wherein the image engine is configured to: receiving said image data from said image sensor array, parsing the image data into a plurality of component images, and An enhanced image is returned based on the plurality of component images, the enhanced image having an enhanced spatial resolution relative to any of the component images.

2. The camera system of claim 1 , wherein each lenslet in the lenslet array is configured to focus the light onto a different area of ​​the image sensor array, and wherein the multiple component images include images acquired simultaneously on different areas of the image sensor array.

3. The camera system of claim 1 , wherein the lamp is one of a series of lamps, wherein the lamp driver is configured to sequentially energize the series of lamps, and wherein the image data comprises a corresponding sequence of component images acquired when different lamps or combinations of lamps are energized.

4. The camera system of claim 3, wherein each light in the series of lights is configured to illuminate the trajectory of the object from a different illumination angle so that the light is reflected onto the lenslet array at different reflection angles. 5 . The camera system of claim 3 , wherein the array of lights comprises N lights, the lenslet array comprises M lenslets, and the plurality of component images comprises M×N component images.

6. The camera system of claim 1 , wherein the image engine is further configured to: transforming each component image of the plurality of component images into a space-frequency domain; combining the plurality of component images in a space-frequency space to obtain a combined image having a wider spatial frequency than any of the component images; and The combined image is inversely transformed to obtain the enhanced image.

7. The camera system of claim 6, wherein combining the plurality of component images comprises accumulating spatial-frequency content from each component image into the combined image using a phase estimate derived from the enhanced image.

8. The camera system of claim 1, wherein an effective aperture size for the enhanced image exceeds a physical aperture size of the lenslet array.

9. The camera system of claim 1, wherein the lenslet array comprises a rectangular matrix of replicated lenslets.

10. The camera system of claim 1, wherein the rectangular matrix is ​​a two-by-two matrix.

11. The camera system of claim 1 , wherein the series of lights comprises eight lights distributed around the image sensor array in a plane parallel to the image sensor array.

12. The camera system of claim 1, wherein the array of lights is an array of light emitting diodes (LEDs), and wherein the image sensor array is a complementary metal oxide silicon (CMOS) array.

13. The camera system of claim 1, wherein the camera system is disposed in a near-eye display device, and wherein the object comprises a human eye.

14. The camera system of claim 1, wherein the image engine is further configured to formulate iris recognition based on the enhanced image.

15. A method comprising: energizing a light of the camera system, the light emitting light onto the subject; acquiring image data recording the light reflected from the object and focused by a lenslet array comprising a plurality of lenslets arranged laterally above an image sensor array; parsing the image data into a plurality of component images, and An enhanced image is returned based on the plurality of component images, the enhanced image having an enhanced spatial resolution relative to any of the component images.

Images