Automatic sensor pixel placement method optimized for multiple camera tasks

Abstract

A method for optimizing alignment of X different types of pixels on a sensor array for performance of N different camera tasks, where X and N are integers greater than 1, includes obtaining an output of each of the X types of pixels on the sensor array for an input image, evaluating a quality of the output for each of the N different camera tasks, and obtaining an optimal pixel alignment pattern by adjusting potential pixel alignment patterns for performance of the N different camera tasks according to a user-determined balance of priorities among the N camera tasks.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. patent application Ser. No. 17 / 952,077, filed June 16, 2022, entitled "AUTOMATIC SENSOR SPARSE PLACEMENT METHOD OPTIMIZED FOR MULTIPLE CAMERA TASKS" (SYP347918US02), and U.S. Provisional Patent Application No. 63 / 268 / 386, filed February 23, 2022, entitled "METHOD FOR PLACING SENSOR PIXELS" (SYP347918US01), both of which are hereby incorporated by reference for all purposes as if fully set forth herein. [Background technology]

[0002] Modern camera image sensors include different types of pixels. Some types specialize in how they address aspects of the captured image quality, such as accurate color representation (using pixels of a color filter array) or high dynamic range (using a combination of highly sensitive pixels and a high threshold saturation limit). Other types are designed to address aspects that do not determine the quality of the image captured by the sensor array, but are relevant for downstream camera tasks, including image processing such as autofocus, object recognition, object removal or depth determination. In these cases, phase-sensitive pixels of various subtypes are often relevant.

[0003] The pattern in which different types of pixels are arranged (registered) across the image sensor surface plays a key role in determining the appearance of the captured image, some of which are directly related to the quality of the image captured by the image sensor, while others are related to the downstream camera tasks that obtain the captured image as input for analysis. However, a particular pattern that is advantageous for the performance of one camera task of interest to a user may be detrimental to another camera task of interest to the user as well, and so it may be desirable to find an "optimized" pattern that balances the camera user's priorities across two or more tasks.

[0004] Manual design of an optimized pixel alignment pattern that results in good performance for one or more camera tasks that are considered high priority, at the expense of reduced performance for one or more camera tasks of lower priority, is a time-consuming and expensive process for camera manufacturers.

[0005] Several approaches have been proposed to automate pixel alignment design with the sole objective of accurate color representation in the image output by the camera. In this case, the task is to maximize the fidelity of colors captured by the image sensor array, which is determined in large part by three or more subtypes of color filter array (CFA) pixels on the sensor array. All pixels are of the same basic type, responding to the luminance or intensity of incident light, and their outputs are simply combined to result in an image that shows the spatial variation of light emitted from the surface of the object being imaged. The pixel subtypes differ from each other only in the spectral transmittance of the filters fabricated on or above the top layer of pixels of a given subtype, which selectively block light outside the corresponding spectral band. In an RGB image sensor system, for example, three subtypes of CFA pixels are created by red, blue and green filters respectively, and a typical alignment or arrangement of the filtered pixels is a 2x2 Bayer block pattern, repeated over the entire sensor array, with the green filtered pixels placed on the diagonal and the red and blue filtered pixels occupying the other two spaces. It has been found that improved color performance can be achieved by a more complex alignment, in one example involving 4x2 pixel blocks of four different color filtered subtypes, although many other combinations can (and are) anticipated.

[0006] Another example of an automated approach to pixel alignment is the design process for a pattern of shutter functions that are superimposed on the pixel array of an image sensor, in this case with the goal of maximizing the dynamic range of the captured image. The pixels in the array are of one basic type that captures gray levels of intensity (because there are no color filters in this case), and the "alignment pattern" is not actually a pattern of different pixel types or subtypes, but a spatial distribution on the sensor surface of a set of circuit parameters that determine exposure times that vary continuously across the array. The spatial distribution of exposure times on the pixel array primarily determines the achievable dynamic range in the images captured by those pixels.

[0007] To date, approaches have addressed the problem of optimizing the alignment of pixels on an image sensor for a single camera task where only one type of pixel is relevant and these pixels determine an aspect of the image quality. No approach appears to address the problem of simultaneously optimizing the performance of multiple camera tasks, taking into account their relative priorities. In these situations, a mixture of different pixel types may be relevant, where one type may not be relevant to the perceived quality of the captured image, which may be of great practical importance to the end user; phase-sensitive pixels are one such example, necessary for such tasks as camera autofocus, 3D reconstruction, etc.

[0008] Thus, there is a need for a system and method that automates the design of pixel alignment for an image sensor having at least two different types of pixels in order to optimize overall camera performance in a manner that achieves a desired balance between the performance of two or more different tasks. Summary of the Invention

[0009] The present invention includes methods and systems for optimization of the alignment of X different types of pixels on an image sensor array for performance of N different camera tasks, where X and N are integers greater than one.

[0010] In one embodiment, the method includes obtaining an output of each type of pixel in a sensor for an input image, evaluating the quality of the output for each of the N camera tasks, and obtaining an optimal pixel alignment pattern by adjusting potential pixel alignment patterns with respect to performance of the N camera tasks according to a balance of user-determined priorities among the N camera tasks.

[0011] In another embodiment, a method includes simulating pixel responses to the training images, where for each training image, X sensor images are output, each sensor image corresponding arbitrarily to only one of the X different types located at each possible location of the image sensor array; generating a pixel alignment pattern according to which the X different types of pixels are distributed on the image sensor array as a 2D tessellation in the trainable pixel alignment; subsampling each of the X sensor images with the pixel alignment pattern and outputting X corresponding subsampled images; and simultaneously training a pixel alignment layer and a stack of N neural networks, each neural network uniquely corresponding to only one of the N different camera tasks, receiving and processing one or more of the X subsampled sensor images to make a discrete and distinct selection of the X different types of pixels, such that the pixel alignment pattern is optimized for performance of the N camera tasks subject to a user determined balance of priorities among the N camera tasks.

[0012] In another embodiment, the apparatus includes one or more processors and logic encoded in one or more non-transitory media for execution by the one or more processors, the logic comprising, when executed, the steps of: simulating pixel responses to the training images, where for each training image, X sensor images are output, where each sensor image uniquely corresponds to only one of X different types of pixels located at each possible location of the image sensor array; generating a pixel alignment pattern in a trainable pixel alignment layer, where the X different types of pixels are distributed over the locations of the image sensor array as a 2D tessellation; and generating a pixel alignment pattern for each of the X corresponding pixels. and simultaneously training a pixel alignment layer and a stack of N neural networks, each neural network uniquely corresponding to only one of the N different camera tasks, such that the pixel alignment pattern is optimized with respect to performance of the N camera tasks according to a user-determined balance of priorities among the N camera tasks, receiving and processing the X subsampled sensor images to make a discrete and distinct selection of the X different types of pixels.

[0013] In yet another embodiment, a system includes a pixel response simulator, a trainable pixel alignment layer, a subsampler, and a stack of N trainable neural networks, each neural network uniquely corresponding to only one of the N camera tasks, the pixel response simulator configured to operate on training images and for each training image send X sensor images to the subsampler, each sensor image uniquely corresponding to pixels of only one of X different types located at each possible location on the image sensor array, the trainable pixel alignment layer configured to map the X different types of pixels to the locations of the image sensor array as a 2D tessellation. In one embodiment, the pixel alignment pattern is distributed over the X sensor images to generate and send to the subsampler, the subsampler being configured to subsample each of the X sensor images with the pixel alignment pattern and send one or more subsampled output images to one or more neural networks for processing therein, the N neural networks and the pixel alignment layer being trained simultaneously using inner and outer feedback loops, respectively, to make a discrete and distinct selection of the X different types of pixels such that the pixel alignment pattern is optimized with respect to the performance of the N camera tasks according to a user determined balance of priorities among the N camera tasks.

[0014] A further understanding of the nature and advantages of specific embodiments disclosed herein may be realized by reference to the remaining portions of the specification and the attached drawings. [Brief description of the drawings]

[0015] [Figure 1] 1 is a flowchart illustrating method steps according to some embodiments of the present invention. [Diagram 2] 1 is a block diagram forming a system according to some embodiments of the present invention. [Diagram 3]3 is a portion of a system according to the embodiment of FIG. 2. [Figure 4] This is how the combined performance value can be calculated in terms of task losses in some of the embodiments of FIG. [Diagram 5] 1 is a block diagram forming a system according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Described herein are embodiments of systems and methods that optimize the alignment of pixels of two or more different types on an image sensor array for performance of two or more camera tasks, taking into account the priority level of the tasks.

[0017] 1 is a flow chart illustrating steps of a method 100 according to some embodiments of the present invention. In step 101, the output of each type of pixel on an image sensor array is obtained given an input image. In step 102, the quality of the output for each of the N camera tasks is evaluated for potential pixel alignment patterns on the sensor array. In step 103, an optimal pixel alignment pattern is obtained by adjusting the potential pixel alignment patterns for the performance of the N camera tasks according to a user-determined balance of priorities among the N camera tasks.

[0018] In some embodiments of the method 100, obtaining the optimal pixel alignment pattern includes evaluating all possible combinations of pixel alignments on the sensor array. In other embodiments, it includes using one or more rule-based algorithms. In some embodiments of the method 100, obtaining the optimal pixel alignment pattern includes using machine learning. The machine learning may utilize trainable pixel alignment layers, in some cases involving successive relaxation techniques. Alternatively, the machine learning may utilize reinforcement learning techniques.

[0019] In some embodiments of method 100, the sensor array includes a tessellation of identical pixel blocks, each block including a first arrangement of X different types of pixels, and obtaining an optimal pixel alignment pattern includes obtaining an optimal arrangement of the X different types of pixels in a block. In one subset of these embodiments, obtaining an optimal arrangement of pixels in each block includes using one or more rule-based algorithms. In yet another subset of these embodiments, obtaining an optimal arrangement of pixels in each block includes using machine learning, which may utilize a trainable pixel alignment layer, and in some cases involving or instead of successive relaxation techniques, the machine learning may utilize reinforcement learning techniques.

[0020] In another embodiment of method 100, the sensor array includes a tessellation of first and second blocks of pixels, each of the first blocks including a first arrangement of X different types of pixels and each of the second blocks including a second arrangement of the X different types of pixels different from the first arrangement, and obtaining the optimal pixel alignment pattern includes obtaining an optimal arrangement of the X different types of pixels in the first blocks and the second blocks. Subsets of embodiments can be readily envisioned for the above various options regarding different approaches to obtaining the optimal pixel alignment pattern (evaluating all possible combinations, using a rule-based algorithm, etc.).

[0021] In some embodiments of method 100, each pixel type is characterized by one or more associated adjustable parameters, such as phase angle, filter wavelength range, etc. In some embodiments of method 100, step 101, in which the output of each type of pixel on the sensor is obtained, involves simulating the response of that type of pixel. A description of how this may be done is provided below with reference to system 200 illustrated in FIG. 2. In other embodiments, obtaining those outputs involves measuring the response of each type of pixel, rather than simulating it. This may be done by physically capturing an image using a camera and directly recording the pixel response.

[0022] 2 illustrates a system 200 according to some embodiments of the present invention that essentially performs method 100 using a particular combination of selections for implementing steps 101, 102, and 103. Other systems may be expected to use other selections without departing from the spirit and scope of the present invention.

[0023] The pixel response simulator 202 receives training images 201 from a training image database (not shown), generates a number of X sensor images 203, one for each type of pixel whose response is to be simulated, and sends the images to a subsampler 206. A trainable pixel alignment layer module 204 generates a pixel alignment pattern 205 for the mix of pixel types and sends the pattern to the subsampler 206. The subsampler 206 operates on each one of the received sensor images 203 in turn, extracting data for its corresponding pixel type at each location indicated for that pixel type in the pattern 205, and then tessellates a small portion of the resulting simulated sensor image on the sensor array. In this manner, the subsampler 206 generates X subsampled images 207, one image for each of the X types of pixels.

[0024] A stack 208 of N trainable networks receives the subsampled images 207, with each individual network dedicated to simulating only one camera processing task receiving one or more of the subsampled images appropriate for the operation of that particular task, so that, for example, network 208-1 dedicated to simulating camera processing task #1 receives one or more subsampled images appropriate for the operation of task #1, network 208-2 for task #1 receives either the subsampled image or images associated with the network for task #2, etc. As described below with respect to Figures 3 and 4, there may be some overlap between the subsampled images sent to different networks, depending on the nature of the corresponding camera tasks, but this is not necessary.

[0025] In some embodiments, each of the N neural networks may receive all of the X subsampled images and process some of them using information that is not directly related to the particular camera task of that particular network, but that provides relevant side information. Each network in stack 208 provides an output value that reflects how well it is currently performing its corresponding task, with 208-1 providing a performance value 209-1 for its performance of task #1, 208-2 providing 109-2, and network 208-N providing 209-N for the Nth task. The individual performance values ​​are input to module 210, which calculates a combined performance value (another word for "combined performance value" in optimization literature is "objective function") 212 that reflects how well all N tasks can be performed for the current pixel alignment pattern 205 and the current version of each network's internal configuration. In some scenarios, these parameters may be provided to module 210 as separate inputs (not shown).

[0026] The performance value 212 of the combination is fed back from the output of the module 210 via simultaneous inner and outer feedback loops. The inner loop includes path A between the module 210 and the stack 208, allowing each of the networks 208-1 to self-learn via 208-N to optimize the performance value 212. In some embodiments, this means minimizing the corresponding combined loss value. The outer loop includes path B between the module 210 and the pixel alignment layer 204, allowing the layer 204 to learn and optimize the performance value 212 by optimizing the pixel alignment pattern 205.

[0027] Training of the system 200 continues according to the process described above for each of a series of training images available from the database.

[0028] Modules 204 and 208 are tunable, allowing them to be optimized using clues from combined performance value 212 .

[0029] One possible way to implement this optimization is by using stochastic gradient descent, The performance value is increased by computing the gradient of the performance value with respect to the tunable parameters (the neural network weights) and applying updates to the tunable parameters in the direction of the gradient. In the stochastic gradient descent framework, all modules between the tunable modules and the performance value are required to be differentiable.

[0030] Given these requirements, it can be appreciated that a trainable alignment layer, such as layer 204 in Figure 2, must be defined as a differentiable function. The gradient of the performance value must somehow be computed for different pixel selections. However, because the pixel selections are discrete, the gradient is not defined.

[0031] To address this, layer 104 is implemented using a method that approximates the selection of discrete pixels in a differentiable manner.

[0032] The methods used in some embodiments of the present invention fall under the umbrella of "continuous relaxation", a technique used to address a general category of problems that require derivatives and / or gradients to be calculated from non-differentiable discrete or classifier functions by reconstructing them into approximations using continuous variables.

[0033] While continuous relaxation is one approach to approximating a non-differentiable function with a differentiable function, there are other methods that can be considered for other embodiments.

[0034] By implementing 204 with a differentiable approximation, the present invention allows sensor designers to optimize sensors where discrete and categorical design choices can be made, in contrast to prior art techniques that limit the optimization space to only continuous ones.

[0035] Figure 3 illustrates a portion of a system of the type shown in Figure 2 for one example embodiment where there are three different pixel types to place on the sensor and two different camera tasks of interest. In this embodiment, a pixel response simulator 302 receives a training image 301 and outputs three sensor images 303A, 303B, and 303C, where 303A shows what the sensor output would be if the sensor array only had "A" type pixels at every pixel location on the sensor array, and 303B and 303C do the same for "B" and "C" type pixels, respectively.

[0036] The pixel alignment layer 304 generates a pattern 305 in the form of 4x4 blocks, which is repeated as a 2D tessellation on the surface of the image sensor array (not shown). Each block has five each of "A" and "C" type pixels, and six of "B" type pixels, the latter positioned diagonally, although it should be understood that this is only one "candidate" for an optimized arrangement.

[0037] Subsampler 306 operates on sensor images 303A, 303B and 303C to extract data from pixels of types A, B and C in each block indicated for that pixel type in pattern 305, and then tessellates the resulting fractional portions of the simulated sensor image on the sensor array. The resulting output is three subsampled images 307A, 307B and 308C.

[0038] In this exemplary system with two camera tasks of interest, stack 308 consists of two corresponding trainable neural networks 308-1 and 308-2 delivering outputs 309-1 and 309-2. In the case shown, only subsampled image 307A is fed into network 308-1 as input, since the performance of the camera task of network 308-1 is only affected by how type A pixels perform, and not by type B or C. The task of network 308-1 can be, for example, an object recognition task, completely determined by type A pixels. At the same time, the performance of the camera task corresponding to network 308-2 can be affected to an extent by how type B and type C pixels perform, and therefore network 308-2 can be required to be provided with both images 307B and 308C as input. The task of network 308-2 can be, for example, an "image quality" task providing images with a high dynamic range, which can require a mix of high sensitivity pixels and a high saturation threshold.

[0039] In some systems similar to that shown in FIG. 3, all two types of pixels may be relevant for the performance of both objective tasks. In other embodiments not shown, at least one pixel type may be relevant for both camera tasks, etc. As the number of variables increases, more pixel types and / or more camera tasks are naturally modeled. An important feature of all embodiments of the current invention is that the X pixel types are distinct from one another, and the N camera tasks are distinct from one another. In some scenes, at least one task achieves an image quality goal determined by one or more types of pixels in one subcategory of simple image capture pixels, while another task achieves a goal that depends on distinct types of pixels, such as phase detection pixels. Producing an image with a high dynamic range is one example of an image quality task, while processing the captured image to identify objects of a particular shape in the scene is an example of a very challenging type of camera or image processing task. Other tasks may include generating a depth map, or creating a semantic understanding of the scene.

[0040] Returning to Figure 3, the output of the two networks are performance values ​​309-1 and 309-2, respectively, which are then fed into a module (as shown) corresponding to module 210 of system 200 which operates on those values ​​to generate a combined performance value reflecting how well the set of tasks performed for the three pixel types of interest and for the current version of pixel alignment pattern 305 for the current internal configuration of neural networks 308-1 and 308-2. The combined performance value is fed back to the neural network and pixel alignment layer 304 of stack 308 to generate improved network parameters and pixel alignment patterns by processes well known in the art of machine learning for optimization of system parameters.

[0041] 4 shows how performance values ​​are calculated and processed in some embodiments of the invention where the performance values ​​are expressed as loss values ​​that are inversely related to how well a task is performed. In the scene shown, there are N neural networks in a stack 408, the first network outputs a performance value 408-1 in the form of a task loss L1, one by one up to the last Nth network, which outputs the task loss L N Next, in module 410, the kth term in the function, where k varies from 1 to N, is f (α k , L k, A combined performance value is calculated as a function of each performance value and a corresponding weighting parameter α for each task, such as: k ×L k , . In other embodiments, the function may be, for example, (L k, ) 2 The weighting parameters indicating the relative priorities of the N tasks can be either pre-defined inputs embedded in the system or inputs as adjustable parameters set by the user to train the system on performance.

[0042] Another option is that the performance values ​​immediately expected for the training system of the present invention are restated as loss values ​​inversely related to how well the task is performed, but the combined performance value is calculated as a function of each performance value and a combination of two or more task specific weighting parameters. In the two parameter case, for example, the kth term of the function is e.g. [(α k ) 2 * L k ] + [β k * (L k ) 2] Many variables can be predicted, some of which may be relevant for more than two weighting parameters.

[0043] Although the embodiments described above and shown in the figures all relate to the use of a stack of separate neural networks, the invention is not necessarily limited to this particular architecture. Figure 5 illustrates some other embodiments in which there is one large "super" neural network 508 where all X sensor images are input and output N values ​​corresponding to each task. The other elements of the system 500 operate as in the system 200 shown in Figure 2, as follows.

[0044] The pixel response simulator 502 receives training images 501 from a training image database (not shown), generates a number of X sensor images 503, one for each type of pixel whose response is to be simulated, and sends the images to a subsampler 506. The trainable pixel alignment layer module 404 generates a pixel alignment pattern 505 for the mix of pixel types and sends the pattern to the subsampler 506. The subsampler 506 operates on each one of the received sensor images 503 in turn, extracting the data for its corresponding pixel type at each location shown for that pixel type in the pattern 505, and then tessellates a functional portion that is the result of the simulated sensor image on the sensor array. In this manner, the subsampler 506 generates X subsampled images 507, one image for each of the X types of pixels.

[0045] However, the trainable network 508 receives all images 597 and provides N output values, one for each of the N camera tasks of interest, indicating how well the system currently performs its corresponding task. The individual performance values ​​are input to a module 510 which calculates a combined performance value 512 reflecting how well all of the N tasks are performing for the current version of the pixel alignment pattern 505 and the current internal configuration of the super-network 508. The calculation involves weighting the individual performance values ​​by a pre-determined task priority parameter.

[0046] Embodiments of the present invention provide a significant advantage over the prior art in this area in providing a system and method for automatic optimization of different underlying types of pixel alignment for a camera intended to perform more than one task, taking into account user-determined priorities between those tasks. In some embodiments, this is achieved by performing end-to-end training of a system that includes a stack of task-specific neural networks and a trainable pixel alignment layer, with the user setting one or more parameters that balance competing aspects of the camera's performance.

[0047] Although descriptions are given with respect to specific embodiments herein, these specific embodiments are merely illustrative and not limiting.

[0048] Any suitable programming language may be used to implement the routines of a particular embodiment, such as C, C++, Java, assembly line, etc. Different programming techniques may be used, such as procedural and object-oriented. The routines may be executed on a single processing device or multiple processors. Although steps, operations or computational uses may be given in a particular order, this order may be changed in different particular embodiments. In some particular embodiments, multiple steps shown in this specification as being sequential may be performed simultaneously.

[0049] Certain embodiments may be implemented in a computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device. Certain embodiments may be implemented in the form of control logic in software or hardware, or a combination of both. The control logic, when executed by one or more processors, may be operable to perform what is described in certain embodiments.

[0050] Particular embodiments may be implemented using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum, or nano-optical systems, components, and mechanisms available using a programmed general purpose digital computer. In general, the functionality of particular embodiments may be achieved by any means known in the art. Distributed, networked systems, components, and / or circuits may be used. Communication or transfer of data may be by wire, wireless, or any other means.

[0051] It will be appreciated that one or more of the elements depicted in the drawings / figures may also be implemented in a more discrete or integrated manner, or in some cases removed or rendered inoperative, as may be useful depending on the particular application. It is also within the intent and scope to implement a program or code stored on a machine readable medium and enabling a computer to perform any of the methods described above.

[0052] A "processor" includes any suitable hardware and / or software system, mechanism or component that processes data, signals or other information. A processor may include a system with a general purpose central processing unit, a microprocessor, dedicated circuitry for implementing a function, or other system. Processing need not be limited to a geographic location or have time limitations. For example, a processor may perform its functions in "real time", "offline", "batch mode", etc. Portions of processing may be performed at different times and in different locations by different (or the same) processing systems. Examples of processing systems may include servers, clients, end user devices, routers, switches, networked storage devices, etc. A computer may be any processor in communication with a memory. Memory may be a random access memory (RAM), a magnetic or optical disk, or any suitable processor-readable storage medium, other non-transitory medium suitable for storing instructions for execution by a processor.

[0053] As used in this description and throughout the claims which follow, "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Furthermore, as used in this description and throughout the claims which follow, the meaning of "in" includes "in" and "on," unless the context clearly dictates otherwise.

[0054] Thus, although specific embodiments have been described herein, it will be recognized that latitude in modification, various changes, and substitutions are contemplated in the foregoing disclosure, and that in some cases, some features of a specific embodiment may be utilized without the corresponding use of other features without departing from the scope and intent set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit.

Claims

1. 1. A method for optimization of the alignment of X different types of pixels on a sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1, the method comprising: obtaining an output for each of the X types of pixels on the sensor array for an input image; evaluating a quality of the output for each of the N different camera tasks; and obtaining an optimal pixel alignment pattern by adjusting potential pixel alignment patterns with respect to performance of the N different camera tasks according to a user-determined balance of priorities among the N different camera tasks.

2. The method of claim 1 , wherein obtaining the optimal pixel alignment pattern comprises evaluating all possible combinations of pixel alignments on the sensor array.

3. The method of claim 1 , wherein obtaining the optimal pixel alignment pattern comprises using one or more rule-based algorithms.

4. The method of claim 1 , wherein obtaining the optimal pixel alignment pattern comprises using machine learning.

5. The method of claim 4 , wherein using machine learning comprises utilizing a trainable pixel alignment layer.

6. The method of claim 5 , wherein utilizing an alignment layer of trainable pixels comprises using a successive relaxation technique.

7. The method of claim 4 , wherein the machine learning comprises utilizing reinforcement learning techniques.

8. the sensor array being a tessellation of identical blocks of pixels, each block including an arrangement of pixels of the X different types; Obtaining an optimal pixel alignment pattern includes obtaining an optimal arrangement of the X different types of pixels in a block. The method of claim 1.

9. The method of claim 8 , wherein obtaining the optimal arrangement of pixels in each block comprises evaluating all possible combinations of alignments of pixels in the block.

10. The method of claim 8 , wherein the optimal pixel alignment pattern comprises using one or more rule-based algorithms.

11. The method of claim 8 , wherein obtaining the optimal pixel alignment pattern comprises using machine learning.

12. The method of claim 8 , wherein using machine learning includes utilizing a trainable pixel alignment layer.

13. The method of claim 12 , wherein utilizing an alignment layer of trainable pixels comprises using a successive relaxation technique.

14. The method of claim 8 , wherein the machine learning comprises utilizing reinforcement learning techniques.

15. the sensor array is a tessellation of first and second blocks of pixels, each of the first blocks including a first arrangement of the X different types of pixels and each of the second blocks including a second arrangement of the X different types of pixels different from the first arrangement; obtaining the optimal pixel alignment pattern includes obtaining an optimal arrangement pattern of pixels in the first block and the second block; The method of claim 1.

16. The method of claim 1 , wherein each pixel type is characterized by one or more associated adjustable patterns.

17. The method of claim 1 , wherein obtaining an output for each type of pixel on the sensor array comprises simulating a response of that type of pixel.

18. The method of claim 1 , wherein obtaining an output for each type of pixel on the sensor array comprises measuring a response of that type of pixel.

19. 1. A method of optimizing the alignment of X different types of pixels on an image sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1, the method comprising: simulating pixel responses to training images such that for each training image X sensor images are output, each sensor image uniquely corresponding to one and only one of the X different types located at each possible location on the image sensor array; generating a pixel alignment pattern by distributing the X different types of pixels over the locations of the image sensor array as a 2D tessellation in a trainable pixel alignment layer; subsampling each of the X sensor images according to the pixel alignment pattern to output X corresponding subsampled sensor images; training the pixel alignment layer and a stack of N neural networks simultaneously, each neural network uniquely corresponding to only one of the N different camera tasks, receiving and processing one or more of the X subsampled sensor images, and making a discrete and distinct selection of the X different types of pixels such that the pixel alignment pattern for the N different camera tasks is optimized according to a user-determined balance of priorities among the N different camera tasks; A method comprising:

20. 20. The method of claim 19, wherein simultaneously training the pixel alignment layer and N neural networks comprises parallel use of an inner feedback loop that feeds back an output of a combined performance value derived from the N outputs from the stack to each of the N neural networks, and an outer feedback loop that feeds back the combined performance value to the trainable pixel alignment layer.

21. 21. The method of claim 20, wherein the combined performance value output from the stack comprises a total loss value including a combination of performance value outputs from each of the N neural networks, each performance value output being a function of a loss value for the corresponding neural network and at least one user-provided weighting parameter for the camera task corresponding to that neural network.

22. 22. The method of claim 21, wherein the function of a loss value and at least one user-provided weighting parameter for one of the neural networks is a linear function of that loss value.

23. 20. The method of claim 19, wherein at least one of the N tasks depends primarily on image quality captured by pixels in the sensor array, and at least another of the N tasks does not depend primarily on image quality captured by pixels in the sensor array.

24. 1. A system for optimization of alignment of X different types of pixels on an image sensor array for a camera capable of performing N different camera tasks, where X and N are integers greater than 1, the system comprising: A pixel response simulator, a trainable pixel alignment layer; A subsampler, a stack of N trainable neural networks, each neural network uniquely corresponding to one of the N different camera tasks; Including, said pixel response simulator operating on training images; configured to send, for each training image, X sensor images to the subsampler, each sensor image uniquely corresponding to one and only one type of pixel located at each possible location on the image sensor array; the trainable alignment layer generates and sends to a subsampler an alignment pattern of pixels where the X different types of pixels are distributed over the locations of the image sensor array as a 2D tessellation; the subsampler is configured to subsample each of the X sensor images according to the pixel alignment pattern and to send one or more subsampled output images to one or more of the neural networks for processing therein; The N neural networks and the pixel alignment layer are trained simultaneously using an inner feedback loop and an outer feedback loop, respectively, to perform individual category selection of the X different types of pixels such that the pixel alignment pattern is optimized with respect to performance of the N camera tasks according to a balance of priorities among the N camera tasks determined by a user.

Images