Image sensor

Abstract

An image sensor is provided. An imaging device can encode light passing through an imaging optical lens arranged in a multi-lens array (MLA), transmit the light to a sensing element, and the sensing element can restore an image based on the sensed information.

Description

[0001] This application claims priority to Korean Patent Application Nos. 10-2020-0055588, filed on May 11, 2020, and 10-2020-0143664, filed on October 30, 2020, with the Korean Intellectual Property Office, the entire disclosures of which are incorporated herein by reference for all purposes. Technical Field

[0002] A method and apparatus consistent with the disclosed example embodiments relate to sensing an image. Background Art

[0003] Due to the development of optical technology and image processing technology, image capture devices are being used in a wide range of fields (e.g., multimedia content, security, or identification). For example, an image capture device may be installed in a mobile device, a camera, a vehicle, or a computer to capture an image, identify an object, obtain data for controlling the device, etc. The volume of an image capture device may be determined based on, for example, the size of a lens, the focal length of the lens, and the size of a sensor. If the size of the lens is reduced, the focal length of the lens may be reduced. In order to reduce the volume of the image capture device, a multi-lens including a compact and small lens may be used. Summary of the Invention

[0004] One or more example embodiments may at least solve the above problems and / or disadvantages and other disadvantages not described above. Further, the example embodiments need not overcome the above disadvantages, and the example embodiments may not overcome any of the above problems.

[0005] According to an aspect of an example embodiment, there is provided an image sensor including: a mask array including a plurality of mask elements configured to block light on a first portion of the plurality of directions and allow light on a second portion of the plurality of directions to pass through the plurality of mask elements among light that passes through an imaging optical lens and is incident on the mask array in a plurality of directions; and a sensing array including a plurality of sensing elements configured to sense light that passes through the imaging optical lens and the mask array.

[0006] The image sensor may further include: a color filter disposed above the sensing array and configured to filter a portion of a wavelength band of light incident on each of the plurality of sensing elements, wherein the mask array is disposed between the color filter and the sensing array.

[0007] The image sensor may further include: a condenser lens array disposed above the sensing array, wherein the mask array is disposed between the condenser lens array and the sensing array.

[0008] The mask array and the plurality of sensing elements may be spaced apart from each other by 1 micrometer (μm) or less.

[0009] The mask array and the plurality of sensing elements may be in contact with each other.

[0010] A first region of the mask array corresponding to a sensing element among the plurality of sensing elements may include: an aperture region that occupies an area corresponding to an aperture ratio with respect to the total area of the first region; and a mask region that occupies the remaining area of the first region, and the plurality of mask elements are disposed in the mask region.

[0011] The aperture ratio may be between about 40% and about 60%.

[0012] In each partial region of the mask array, the area occupied by the aperture may be greater than or equal to the area corresponding to the set aperture ratio.

[0013] The mask array may be divided into a plurality of group regions corresponding to a plurality of groups of sensing elements, and each group region among the plurality of group regions in the mask array may be configured to cover a group of sensing elements, and the group of sensing elements includes a plurality of sensing elements grouped to represent a single pixel.

[0014] The mask pattern of the group region may be repeated in the mask array.

[0015] All of the group regions among the plurality of group regions in the mask array may have the same mask pattern.

[0016] The number of mask elements included in the mask pattern repeated in the mask array may be greater than or equal to the number of imaging optical lenses.

[0017] The plurality of mask elements may have two or more transmission levels.

[0018] Each mask element among the plurality of mask elements may be divided into a plurality of regions, and the transmittance of each mask element among the plurality of mask elements may be determined based on the ratio of the open region to the closed region among the plurality of regions.

[0019] The image sensor may further include: a processor configured to: recover an image based on sensing information sensed by the plurality of sensing elements.

[0020] The processor may further be configured to: generate frequency information by transforming the sensing information into the frequency domain, generate deblurred frequency information by dividing the frequency information by the frequency transformation result of the blur kernel, and recover a high-resolution image by inverse-transforming the deblurred frequency information into the time domain, and the blur kernel corresponds to the mask pattern of the mask array.

[0021] The image sensor may include a plurality of mask patterns, and each of the plurality of mask patterns may be configured to cover a group of sensing elements, the group of sensing elements including two or more sensing elements in a sensing array.

[0022] According to an aspect of an example embodiment, there is provided a camera device including: an imaging lens array including an imaging optical lens configured to transmit light received from outside the camera device; a sensing array including a plurality of sensing elements configured to sense light passing through the imaging lens array; and a mask array including a plurality of mask elements, the mask array being disposed between the imaging lens array and the sensing array.

[0023] The mask array may be disposed at one of a position inside the sensing array on the plurality of sensing elements and a position in contact with the sensing array.

[0024] A first region of the mask array corresponding to a sensing element among the plurality of sensing elements may include: an aperture region occupying an area corresponding to an aperture ratio with respect to a total area of the first region; and a mask region occupying the remaining area in the first region, the plurality of mask elements being disposed in the mask region.

[0025] In each partial region of the mask array, an area occupied by the aperture may be greater than or equal to an area corresponding to a set aperture ratio.

[0026] The mask pattern of the group region may be repeated in the mask array.

[0027] The camera device may further include: a processor configured to: generate frequency information by transforming sensing information sensed by the plurality of sensing elements into a frequency domain, generate deblurred frequency information by dividing the frequency information by a frequency transformation result of a blur kernel, and restore a high-resolution image by inverse-transforming the deblurred frequency information into a time domain, the blur kernel corresponding to the mask pattern of the mask array. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or other aspects will become more apparent by describing specific example embodiments in conjunction with the drawings, in which:

[0029] Figure 1A and Figure 1B shows an example of an imaging device according to an example embodiment;

[0030] Figure 2 is a diagram showing an example in which a sensing element receives light through a lens element according to an example embodiment;

[0031] Figure 3Is a diagram showing the relationship between the number of sensing elements and the number of lens elements according to an exemplary embodiment;

[0032] Figure 4 Shows the reduction in focal length based on the structure of a multi-lens array (MLA) in an imaging device according to an exemplary embodiment;

[0033] Figure 5 Shows a blur kernel based on the structure of an MLA in an imaging device according to an exemplary embodiment;

[0034] Figure 6 Shows the blur kernel of an imaging device including a mask array according to an exemplary embodiment;

[0035] Figure 7 Is a cross-sectional view of an imaging device in which a mask array is provided according to an exemplary embodiment;

[0036] Figure 8 Shows the design of the mask pattern of the mask array according to an exemplary embodiment;

[0037] Figure 9 And Figure 10 Shows an example of a mask array according to an exemplary embodiment;

[0038] Figure 11 Shows the arrangement of the mask patterns of each sensing element group in an image sensor according to an exemplary embodiment;

[0039] Figure 12A And Figure 12B Shows an example of the arrangement of the mask array according to an exemplary embodiment;

[0040] Figure 13 Is a block diagram showing the configuration of an imaging device according to an exemplary embodiment;

[0041] Figure 14 Is a block diagram showing the configuration of an electronic terminal according to an exemplary embodiment; and

[0042] Figure 15 And Figure 16 Is a diagram showing an example of a device in which an image sensor is to be implemented according to an exemplary embodiment. Detailed Description

[0043] Hereinafter, some exemplary embodiments will be described in detail with reference to the accompanying drawings. However, various changes and modifications can be made to the exemplary embodiments. Here, the exemplary embodiments are not to be construed as limited to the disclosure, and should be understood to include all changes, equivalents, and substitutions within the scope of the disclosed concept and technology.

[0044] The terms used herein are for the purpose of describing particular example embodiments only and are not intended to be limiting of the example embodiments. As used herein, unless the context clearly dictates otherwise, the singular forms are intended to also include the plural forms. It will also be understood that the terms "comprising / including" and their variants, when used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It will also be understood that terms, such as those defined in a general dictionary, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0046] When describing example embodiments with reference to the accompanying drawings, like reference numerals denote like constituent elements and repeated descriptions thereof will be omitted. In the description of example embodiments, when a detailed description of a related known structure or function is considered to result in an unclear interpretation of the disclosure, such description will be omitted. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative dimensions, proportions, and depictions of elements in the drawings may be exaggerated.

[0047] Figure 1A and Figure 1B shows the structure of an imaging device according to an example embodiment. Figure 1A is a perspective view of the imaging device, Figure 1B is a cross-sectional view of the imaging device.

[0048] The imaging device 100 may include a lens array 110 and an image sensor 120. The lens array 110 may include lens elements, and the image sensor 120 may include optical sensing elements. The lens elements may be arranged along the plane of the lens array 110, and the optical sensing elements may be arranged along the plane of a sensing array 121 in the image sensor 120. The plane of the lens array 110 may be placed parallel to the plane of the sensing array 121. The lens array 110 may be a multi-lens array (MLA) for imaging and may also be referred to as an "imaging lens array".

[0049] The term "optical sensing element" (hereinafter referred to as "sensing element") used herein may be an element that senses optical information based on light incident on the element and may output a value indicating the intensity of the incident light. The optical sensing elements may include, for example, complementary metal oxide semiconductor (CMOS), charge-coupled device (CCD), and / or photodiode.

[0050] The term "picture element" as a pixel can be basic unit information constituting an image, and can indicate optical information obtained by sensing light reflected from a physical position on an object corresponding to a pixel position using a sensing element. The pixel position can be the position of a pixel in an image, and can conform to a pixel coordinate system, and the physical position can conform to a world coordinate system.

[0051] For reference, a pixel constituting a color image can have a plurality of color values for a single pixel position (e.g., a red value, a green value, and a blue value in an RGB color system). A unit pixel of a display in the display field can include sub-pixels for a plurality of colors (e.g., a red sub-pixel, a green sub-pixel, and a blue sub-pixel in an RGB color system) to represent the color value of a single pixel position. Different from the display field, in the image sensor field, generally, a pixel is not divided into sub-pixels for each color, but refers to a sensing element that senses one color value (e.g., a photodiode having a front end provided with a color filter). In addition, in the image sensor field, the term "pixel" can refer to both a single sensing element and the value sensed by the sensing element. However, for the clarity of the description in the exemplary embodiments, the term "pixel" is used herein to represent the basic unit information constituting an image, and the term "sensing element" refers to a hardware element that outputs the pixel value of a pixel in response to light received from an object, so the meanings of a pixel and a sensing element can be distinguished.

[0052] In the following description, an example in which each pixel is represented by a single sensing element is mainly described. However, the exemplary embodiments are not limited thereto. For example, a single pixel can be represented by a plurality of sensing elements. A plurality of sensing elements grouped to represent a single pixel can be referred to as a "group of sensing elements". Although the amount of light that can be sensed by a single sensing element is limited, the sensitivity can be enhanced by representing a single pixel using the values sensed by a plurality of sensing elements. An example of sensing a single pixel value by a group of sensing elements including four sensing elements will be described below with reference to Figure 11 Describe an example of sensing a single pixel value by a group of sensing elements including four sensing elements.

[0053] The image sensor 120 can include a sensing array 121, a filter 122, and a condenser lens array 123. However, this is only an example. Instead of the filter 122, a single condenser microlens 123a of the condenser lens array 123 can have an optical characteristic of transmitting a predetermined wavelength band and blocking the remaining wavelength bands other than the predetermined wavelength band. In this case, the filter 122 can be omitted.

[0054] The condenser lens array 123 may include a plurality of condenser microlenses 123a configured to focus the light passing through the lens array 110 onto the sensing array 121. For example, the condenser lens array 123 may include the same number of condenser microlenses 123a as the number of sensing elements included in the sensing array 121. The plurality of condenser microlenses 123a may be arranged between the imaging optical lens and the sensing array 121 to focus and transmit the light passing through the imaging optical lens to the sensing element 121a corresponding to each condenser microlens 123a. For example, as Figure 1B shown, the condenser microlens 123a may be disposed above each sensing element 121a of the sensing array 121 to focus the light onto the sensing element 121a located below the condenser microlens 123a. In addition, as Figure 1B shown, the color filter 122a may be disposed between the condenser microlens 123a and the sensing element 121a.

[0055] The optical filter 122 may be a filter having an optical characteristic of transmitting a predetermined wavelength band and blocking the remaining wavelength bands. For example, the optical filter 122 may be implemented as a color filter array (CFA) including a plurality of color filters arranged along the filter plane. Each color filter 122a may be a filter that allows light in a wavelength band corresponding to a predetermined color to pass through and blocks light in the remaining wavelength bands. The color filter 122a may include, for example, a red-pass filter, a green-pass filter, and a blue-pass filter. The red-pass filter may allow light in a wavelength band corresponding to red to pass through and may block light in the remaining wavelength bands. The green-pass filter may allow light in a wavelength band corresponding to green to pass through and may block light in the remaining wavelength bands. The blue-pass filter may allow light in a wavelength band corresponding to blue to pass through and may block light in the remaining wavelength bands. In the color filter array, the color filters that individually transmit color light may be arranged in a Bayer pattern or other pattern along the filter plane. The optical filter 122 may also be an infrared cut-off filter that blocks infrared rays while allowing visible light to pass through.

[0056] The quality of the image captured and restored by the image sensor 120 can be determined based on the number of sensing elements included in the sensing array 121 and the amount of light incident on the sensing element 121a. For example, the resolution of the image can be determined based on the number of sensing elements included in the sensing array 121, and the sensitivity of the image can be determined based on the amount of light incident on the sensing element 121a. The amount of light incident on the sensing element 121a can be determined based on the size of the sensing element 121a. When the size of the sensing element 121a increases, the amount of incident light can increase, and the dynamic range of the sensing array 121 can increase. Therefore, when the number of sensing elements included in the sensing array 121 increases, the resolution of the image captured by the image sensor 120 can increase. As the size of the sensing element 121a increases, the image sensor 120 can operate more effectively in capturing high-sensitivity images in low-light environments.

[0057] A single lens element 111 of the lens array 110 can cover a predetermined sensing area 129 corresponding to the lens size of the single lens element 111 in the sensing array 121. The sensing area 129 covered (or substantially covered) by the lens element 111 in the sensing array 121 can be determined based on the lens size of the lens element 111. The sensing area 129 can indicate an area in the sensing array 121 where light rays of a predetermined field of view (FOV) reach after passing through the corresponding lens element 111. The size of the sensing area 129 can be represented by the diagonal length or the distance from the center of the sensing area 129 to the outermost point. In other words, the light passing through a single lens element 111 can be incident on the sensing elements included in the sensing area 129.

[0058] Each of the sensing elements in the sensing array 121 can generate sensing information based on the light rays passing through the lenses of the lens array 110. For example, the sensing element 121a can sense the intensity value of the light received through the lens element 111 as sensing information. The imaging device 100 can determine the intensity information corresponding to the original signal related to the points included in the field of view of the imaging device 100 based on the sensing information output by the sensing array 121, and can restore the captured image based on the determined intensity information.

[0059] In addition, the sensing element 121a can generate a color intensity value of the corresponding color as sensing information by sensing the light passing through the color filter 122a. Each of the multiple sensing elements included in the sensing array 121 can be set to sense a color different from the color sensed by the adjacent sensing element disposed adjacent to it in space.

[0060] When the diversity of the sensing information is sufficiently ensured and a full-rank relationship is formed between the sensing information and the original signal information corresponding to the points included in the field of view of the imaging device 100, a captured image corresponding to the maximum resolution of the sensing array 121 can be obtained. The diversity of the sensing information can be ensured based on the parameters of the imaging device 100 (such as the number of lenses included in the lens array 110 and the number of sensing elements included in the sensing array 121).

[0061] In the structure of the MLA for imaging, the imaging optical lens and the sensing array 121 can be arranged based on a fractional alignment structure. For example, the fractional alignment structure can represent a structure in which the sensing area 129 covered by a single lens element 111 includes a non-integer number of sensing elements.

[0062] When the lens elements included in the lens array 110 have the same lens size, the number of lens elements included in the lens array 110 and the number of sensing elements included in the sensing array 121 can be in a relatively prime relationship. The ratio P / L between the number L of lens elements corresponding to one axis of the lens array 110 and the number P of sensing elements corresponding to one axis of the sensing array 121 can be determined as a real number. Each of the lens elements can cover the same number of sensing elements as the pixel offset corresponding to P / L. For example, the Figure 1A sensing area 129 that can be covered (or substantially covered) by a single lens element 111 can include "2.3" (= 7 / 3) sensing elements along the vertical axis and "3.67" (= 11 / 3) sensing elements along the horizontal axis. In addition, the lens element 111 can cover a plurality of non-integer condenser microlenses 123a. Therefore, in the image sensor 120, the number of condenser microlenses 123a can be the same as the number of sensing elements of the sensing array 121. In addition, the number of lens elements (e.g., imaging optical lenses) of the lens array 110 can be less than the number of condenser microlenses 123a.

[0063] Through the fractional alignment structure as described above, the optical central axis (OCA) of each lens element 111 in the imaging device 100 can be arranged slightly differently with respect to the sensing array 121. In other words, the lens element 111 can be set to be eccentric with respect to the sensing element 121a. Therefore, each lens element 111 of the lens array 110 can receive different light field (LF) information. This will be further described with reference to Figure 2 the LF information received by the fractional alignment structure.

[0064] Figure 2 is a diagram showing an example in which a sensing element receives light rays through a lens element according to an exemplary embodiment.

[0065] The LF may refer to a field that indicates the direction and intensity of light rays radiated from an arbitrary target point and reflected from an arbitrary point on an object. The LF information may be information obtained by combining a plurality of LFs. Since the direction of the chief ray of each lens element also varies, the sensing area may receive different LF information. Accordingly, the imaging device may obtain a greater amount of optically sensed information.

[0066] As Figure 2 shown, the sensing array 220 may receive and detect light rays corresponding to respective points 230 (e.g., X1 to X10). Multiple light rays emitted from each of the respective points 230 may form an LF. The light rays emitted from the first point X1 may form a first LF and may be incident on the first sensing element S1, the fourth sensing element S4, and the seventh sensing element S7. The light rays emitted from the remaining points X2 to X10 may also form corresponding LFs. The respective points 230 may be points on a predetermined object (e.g., an object). The light rays emitted from the respective points 230 may be light rays such as sunlight reflected from the object. As a cross-sectional view showing an example of the imaging device, for ease of description, Figure 2 a lens array 210 including three lens elements along one axis and a sensing array 220 including ten sensing elements S1 to S10 are shown. However, the exemplary embodiment is not limited thereto.

[0067] The sensing elements S1 to S10 may sense light rays that pass through a plurality of lens elements and overlap each other. The sensing element S1 may generate overlapping sensing information (e.g., intensity values) of the light rays emitted from the points X1 to X3. Similarly, the sensing elements S2 to S10 may also generate overlapping sensing information of the light rays emitted from the respective points 230. The image sensor may recover the overlapping sensing information.

[0068] By Figure 2 the sensing elements S1 to S10 shown, the sensing information generated may be modeled as original signal information (e.g., intensity values) corresponding to the light rays incident from each of the points 230 according to Equation 1 shown below.

[0069] [Equation 1]

[0070] S = T·X

[0071] In Equation 1, S represents a matrix indicating sensing information (e.g., detected intensity values) sensed by each sensing element. X represents a matrix indicating signal values (e.g., color intensity values of incident light rays) corresponding to the light rays incident on the sensing elements S1 to S10 from each point. T represents a transformation matrix and may indicate the relationship between the sensing information sensed by the sensing elements S1 to S10 and the signal information corresponding to the incident light. In Figure 2In the structure shown, the light rays corresponding to the respective points X1 to X10, the lens elements, and the sensing elements S1 to S10 can be modeled as shown in Equation 2 below. In Equation 2, the respective points X1 to X10 can be modeled as being at the focal point at infinity from the image sensor. The distance between the image sensor and each of the respective points X1 to X10 can be greater than a threshold distance.

[0072] [Equation 2]

[0073]

[0074] In Equation 2, for ease of description, the light ray signal information (e.g., light intensity value) corresponding to the respective points X1 to X10 is represented by X1 to X10. Additionally, the sensing information (e.g., sensing intensity value) sensed by the sensing elements S1 to S10 is represented by S1 to S10. The relationship (e.g., the above transformation matrix) between the sensing information corresponding to the sensing elements S1 to S10 included in the sensing array 220 and the original signals corresponding to the light rays incident from the respective points X1 to X10 can be determined based on the arrangement of the lens elements and the sensing elements, the number of lens elements included in the lens array 210, and / or the number of sensing elements S1 to S10 included in the sensing array 220.

[0075] Equation 2 corresponds to the case where the respective points X1 to X10 are focal points at infinity from the image sensor. When the respective points X1 to X10 are at focal points at a finite distance from the image sensor, the original signals received by each sensing element can vary according to the distance between the object and the image sensor and / or the geometric structure of the image sensor.

[0076] As described above, the imaging device can acquire a plurality of low-resolution input images based on various acquired sensing information, and can restore an output image having a resolution higher than that of the low-resolution input images according to the low-resolution input images. The following will refer to Figure 3 Describe a method of generating a single image by rearranging a plurality of low-resolution input images.

[0077] Figure 3 is a diagram showing the relationship between the number of sensing elements and the number of lens elements according to an exemplary embodiment.

[0078] As described above, the imaging optical lens and the sensing array can be arranged in a fractional alignment structure. Figure 3 Shows an example where the ratio P / L of the number P of sensing elements to the number L of lens elements is 10 / 3.

[0079] Based on the above geometric structures of the lens array and the sensing array, the sensing elements covered by each lens element can receive LF information different from that sensed by the sensing elements covered by another lens element. For example, in Figure 2 's structure, the first sensing element S1 can receive LF information including a combination of a first LF of a first point X1, a second LF of a second point X2, and a third LF of a third point X3. On the other hand, in Figure 2 's structure, the second sensing element S2 adjacent to the first sensing element S1 can receive LF information including a combination of a fourth LF, a fifth LF, and a sixth LF. As described above, each sensing element can receive LF information different from that sensed by another sensing element.

[0080] To restore an image with high resolution, the imaging device and / or the image sensor can rearrange the image pixel positions of the pixels indicating the same point or adjacent points on the object in the plurality of captured low-resolution images based on the correlation between the LF information.

[0081] For example, if the image is a color image, the color image can have color values based on a color system as pixel values. However, due to physical limitations, it may be difficult for the image sensor to simultaneously sense three colors at a single point. Generally, a color filter that can only allow one color to pass is disposed at the front end of the sensing element. Therefore, the color that can be sensed at the position of each of the sensing elements can be different from the color sensed by the adjacent sensing elements. Thus, with respect to the first sensing element at a predetermined position (e.g., a sensing element having a front end where a blue-pass filter is disposed), the imaging device and / or the image sensor can interpolate the color value (e.g., a red value) not sensed by the first sensing element by using the color value sensed by the second sensing element adjacent to the first sensing element (e.g., a sensing element having a front end where a red-pass filter is disposed). The imaging device and / or the image sensor can obtain three color channel images by performing interpolation for each color channel. However, the interpolation of the above color values is merely an example, and other methods can also be performed according to the design.

[0082] The imaging device and / or the image sensor may rearrange the pixels of each color channel, which will be described below. For example, in the RGB color system, the imaging device and / or the image sensor may restore the high-resolution red channel image by rearranging the pixels of the low-resolution red channel image. Similarly, the imaging device and / or the image sensor may restore the high-resolution blue channel image and the high-resolution green channel image. Accordingly, the imaging device and / or the image sensor may obtain a high-resolution color image. However, the exemplary embodiments are not limited thereto. For example, the imaging device and / or the image sensor may obtain a low-resolution color image by combining the three color channel images obtained by interpolation as described above, and may restore the high-resolution color image by rearranging the pixels of the low-resolution color image.

[0083] The imaging device and / or the image sensor may construct the pixel information of the high-resolution image by rearranging the pixel positions of the pixels corresponding to the sensing elements that receive similar LF information to be adjacent to each other. As described above, each sensing element may receive LF information in which a plurality of LFs overlap. When the number of the same LFs included in the plurality of information sensed by two sensing elements increases, the correlation between the plurality of information may increase. The rearrangement of the pixel positions of the pixels may be performed based on the depth at which the corresponding pixels are captured. In one example, the depth at which the pixel is captured may be set to any depth value estimated by stereo image matching or measured by a depth sensor. In another example, the pixel positions may also be rearranged by a neural network designed to rearrange the pixel positions based on the depth at which the object is captured even when the depth at which the pixel is captured is not measured and / or estimated. The above rearrangement of the pixel positions may also be referred to as "pixel shuffle". For example, a neural network designed to output a single high-resolution output image in response to the input of a plurality of low-resolution input images may be used to rearrange the pixel positions. The neural network may be trained based on a training data set obtained by capturing an object at various depths.

[0084] The image sensor may assume that the points on the object from which light is reflected are at an infinite focal point and are farther from the image sensor than a threshold distance, and may determine the LF information to be sensed in each sensing element. The image sensor may rearrange the pixel positions of the pixels having the output values output by the sensing elements such that the pixel positions may be adjacent to each other, the sensing element receiving the LFs emitted from the points spatially adjacent to each other on the object.

[0085] For reference, the respective points X1 to X10 are shown in Figure 2 in the order of being spatially adjacent to each other at an infinite focal length. The first point X1 may be adjacent to the second point X2. The second point X2 may be adjacent to the first point X1 and the third point X3.

[0086] In Figure 3 the sensing elements 311 that have not been rearranged yet, both the LF information sensed in the first sensing element S1 and the LF information sensed in the eighth sensing element S8 may include LFs corresponding to the second point X2 and the third point X3. Accordingly, the first sensing element S1 and the eighth sensing element S8 may receive similar LF information. Equation 3 represents the result obtained by rearranging the pixels corresponding to the similar LF information adjacent to each other according to Equation 2 above.

[0087] [Equation 3]

[0088]

[0089] The sensing elements 312 rearranged according to Equation 3 may be as shown in Figure 3 the figure. The first sensing element S1 may be covered by the first lens, and the eighth sensing element S8 may be covered by the third lens. In addition, the fifth sensing element S5 may be covered by the second lens. Since the sensing information sensed in each sensing element corresponds to the pixels constituting the image, the image sensor and / or the imaging device may rearrange the pixels such that the sensing information corresponding to the light passing through different lenses may be adjacent. The reconstructed image 325 may be an image in which the pixel positions of the pixels having the sensed values obtained by the sensing elements are arranged adjacent to each other, the sensing elements receiving similar LF information in the low-resolution images 321, 322, 323, and 324 captured by the respective lenses.

[0090] Figure 4 The reduction in the focal length based on the structure of the MLA in the imaging device according to an exemplary embodiment is shown.

[0091] The volume of the imaging device may be determined by the focal length of the lens element. This is because the image sensor needs to be spaced apart from the lens element by a distance corresponding to the focal length of the lens element to collect the light refracted by the lens element. The focal length of the lens element may be determined by the FOV of the imaging device and the size of the lens element. If the FOV is fixed, the focal length may increase in proportion to the size of the lens element. In order to capture an image within a predetermined FOV range, the size of the lens element may need to increase as the size of the sensing array increases.

[0092] As described above, in order to increase the sensitivity of an image while maintaining the FOV and the resolution of the image, the volume of the image sensor can be increased. In order to increase the sensitivity of the image while maintaining the resolution of the image, it may be necessary to increase the size of each sensing element while maintaining the number of sensing elements included in the sensing array, and thus the size of the sensing array increases. In order to maintain the FOV, the size of the lens element and the focal length of the lens element increase as the size of the sensing array increases, and thus the volume of the image sensor increases.

[0093] When the size of each lens element included in the lens array decreases (i.e., when the number of lenses included in the same area on the lens array increases), the focal length of each lens element can decrease. Accordingly, a thin camera of an imaging device having a reduced thickness can be realized. As Figure 4 shown, the focal length f' of each in the MLA 420 can be smaller than the focal length f of the single lens 410. For example, if the MLA 420 including "2×2" lenses is used instead of the single lens 410, the focal length f' can be equal to f / 2.

[0094] However, the incident area S' of a single lens of the MLA 420 on which light is incident can be smaller than the incident area S of the single lens 410 on which light is incident. For example, the incident area S' can be equal to S / 4. In addition, the incident solid angle Ω corresponding to the angular range of light rays incident on a single sensing element (e.g., the sensing element S2) increases due to the decrease in the focal length. For example, the incident solid angle Ω' corresponding to a single lens of the MLA 420 can be equal to 4Ω. A blur kernel based on the increase in the incident solid angle will be described below with reference to Figure 5 Describe the blur kernel based on the increase in the incident solid angle.

[0095] Figure 5 Shows a blur kernel according to an exemplary embodiment based on the structure of the MLA in an imaging device.

[0096] When the lens is designed at the same FOV in the structure of the MLA 510, as referred to above Figure 4As described, if the number of lenses for the same area increases, the focal length can be decreased and the incident solid angle increases. When the incident solid angle increases, the number of signals overlapping in the sensing element increases. When signals overlap, the information sensed by the sensing element may be blurred. For example, when the FOV is 100 degrees and the image sensor 520 includes "100" sensing elements in an imaging device with a single lens, the FOV of each of the sensing elements can be 1 degree. In one example, if the number of lenses on the same area is "2", then "50" sensing elements can be covered by each lens of the MLA 510. Thus, the FOV of each of the sensing elements can be 2 degrees. In another example, if the number of lenses on the same area is "50", then "2" sensing elements can be covered by each lens of the MLA 510, and the FOV of each of the sensing elements can be 50 degrees. As the FOV that can be sensed by a single sensing element increases, a larger amount of LF information overlaps, and thus the level of blurring of the sensed information increases.

[0097] The blur kernel can be a kernel obtained by modeling the blur caused by the overlap of LF information in a single sensing element, and can also be referred to as a "blur model" or a "blur filter". to The LF information from to can be converged by the lens aperture S by the MLA 510, and the converged LF information can be sensed by the i-th sensing element.

[0098] [Equation 4]

[0099]

[0100] [Equation 5]

[0101]

[0102] In Equation 4, x a represents the intensity of the light rays converging in the direction a. In Equation 5, x b represents the intensity of the light rays converging in the direction b. In Equation 4 and Equation 5, S represents the incident area, and A represents the variable of the area.

[0103] The i-th sensing element can sense the intensity value obtained by accumulating all LF information within the cumulative FOV. The intensity value obtained by accumulating all LF information within the FOV of the i-th sensing element can be discretely approximated, which can be expressed as shown in Equation 6 below.

[0104] [Equation 6]

[0105]

[0106] In Equation 6, s[i] represents the intensity value sensed by the i-th sensing element, and θ represents the variable of the angle. In an imaging device configured with an imaging optical system including the MLA 510, the blur kernel of the image can be modeled such that the LF information within the FOV of each sensing element can overlap with the same size (e.g., "1"). As shown in Equation 7 below, the sensed information s[i] sensed by the i-th sensing element can be modeled as the convolution relationship between the original LF information x Ω [i] and the uniform blur kernel h[i].

[0107] [Equation 7]

[0108] s[i] = (x Ω [i] * h[i]) + t[i]

[0109] In Equation 7, h[i] represents the blur kernel, and t[i] represents the noise component. As shown in Equation 8 below, if the Fourier transform of Equation 7 is performed, the convolution operation can be replaced by multiplication.

[0110] [Equation 8]

[0111] S(f) = X Ω (f) · H(f) + T(f) In Equation 8, s(f) represents the frequency information of the intensity value sensed by the i-th sensing element, X Ω (f) represents the frequency information of the original signal, H(f) represents the frequency response characteristic of the blur kernel, and T(f) represents the frequency information of the noise. As shown in Equation 9 below, the frequency information X Ω (f) of the original signal to be restored can be calculated according to Equation 8.

[0112] [Equation 9]

[0113]

[0114] If the inverse Fourier transform is applied to Equation 9, the deblurred x Ω [i] can be obtained.

[0115] However, since T(f) which is the noise component in Equation 9 is an unknown component, even if T(f) is modeled as a statistical probability distribution, errors caused by T(f) / H(f) may occur. As Figure 5 shown, due to the uniform blur kernel h[i] in the spatial domain, the noise component may be amplified. The frequency response characteristic H(f) of the uniform blur kernel h[i] after frequency conversion may include zero-crossings 590, and impulse components may be caused by the zero-crossings 590 in the reciprocal of the frequency response characteristic H(f). Since the impulse components are multiplied by the noise components, the noise can be greatly amplified in the deblurring process.

[0116] Figure 6 Shows a blur kernel of an imaging device including a mask array according to an exemplary embodiment.

[0117] The blur kernel h'[i] of the imaging device can be designed to suppress the noise component. For example, the blur kernel h'[i] can be designed such that the zero-crossings 690 can be minimized in the frequency response characteristic H'(f) in the frequency domain.

[0118] The mask array can be disposed between the imaging lens array and the sensing element, and can include mask elements that block light guided in a part of the directions. For example, the mask array can block light guided in a part of the directions among the light passing through the MLA, and can selectively allow light guided in other directions to pass through to form the blur kernel h'[i].

[0119] Figure 7 Is a cross-sectional view of an imaging device in which a mask array is provided according to an exemplary embodiment.

[0120] The imaging device can include an imaging lens array 710 and an image sensor 720. The imaging lens array 710 can include imaging optical lenses configured to transmit light received from the outside of the imaging device, and can be disposed as shown in FIG. 1 above. The imaging optical lenses can form an imaging optical system.

[0121] The imaging optical system can be an optical system that performs imaging on the sensing array 721, and the optical characteristics can be determined by, for example, the focal length, size, shape, and structure of the imaging lens array 710 and / or the geometric relationship between the imaging lens array 710 and the sensing array 721. For example, the imaging optical system can further include a blocking portion 711 configured to prevent light passing through a single imaging optical lens from reaching a sensing area covered by another imaging optical lens. In addition, the imaging optical system can further include an aperture (not shown) configured to transmit light to the imaging optical lenses.

[0122] The sensing array 721 may include a plurality of sensing elements configured to sense light received from the outside. Each sensing element may receive light guided in multiple directions, and a light beam incident on the imaging device in a single direction may be focused by the imaging optical lens into a light ray in that direction. The sensing elements belonging to any sensing region may receive light rays respectively corresponding to multiple directions and focused by the imaging optical lens covering the sensing region. In Figure 7 as an example, a first light ray 791 guided in a first direction to the sensing element and a second light ray 792 guided in a second direction through the imaging optical lens are shown.

[0123] As described above, since the imaging optical system includes the imaging lens array 710 having a multi-lens structure, a single sensing element of the sensing array 721 may sense light received in multiple directions and overlapping with each other. In other words, the overlap of the light imaged in the sensing array 721 may be modeled as Figure 6 the blurring kernel h'[i].

[0124] The mask array 724 may include a plurality of mask elements and may be disposed on the sensing array 721. The mask elements may be disposed above the positions where the sensing elements of the sensing array 721 are disposed, and may absorb and block part or all of the light guided to those positions. The mask array 724 may modify the imaging optical system corresponding to Figure 5 the blurring kernel to an optical system corresponding to Figure 6 the blurring kernel h'[i]. The mask array 724 may also be understood as making the light passing through the imaging optical lens transmit to the sensing array 721 by encoding the light, and may be referred to as the "encoding mask array 724". The mask array 724 may be disposed at a position inside the sensing array 721 or at a position in contact with the sensing array 721. Although it is desirable that there is no gap between the mask array 724 and the sensing elements (e.g., the first sensing element 721a to the fifth sensing element 721e) regardless of the position of the mask array 724, due to manufacturing process limitations, the mask array 724 is spaced apart from the sensing elements by about 1 micrometer (μm) or less.

[0125] The plurality of mask elements may be arranged in a pattern that minimizes the zero-crossings of the frequency response characteristics of the blurring kernel. For example, the first light ray 791 passing through the imaging optical lens may be incident on the sensing element through the mask element. The second light ray 792 may be incident on the mask element and may be absorbed. The second light ray 792 may be the light ray that causes a zero-crossing in the frequency response characteristics of the blurring kernel described above with reference to Figure 6 . Therefore, the mask array 724 may block the light in the direction that causes the zero-crossing.

[0126] The imaging device and / or the image sensor 720 may sense the overlapping LF information in a direction selectively filtered by the mask array 724. Accordingly, the imaging device and / or the image sensor 720 including the mask array 724 may recover an image with reduced noise. Although the noise may be caused by a loss in the amount of light caused by the mask array 724, the noise suppression effect of the blur kernel modified by the mask array 724 may be greatly increased. As a result, the quality of the recovered image may be enhanced by the mask pattern of the mask array 724.

[0127] A plurality of mask elements may be formed and / or set according to the mask pattern. The mask pattern may be repeated in units of sensing elements or in units of groups of sensing elements. The first mask pattern provided on the first sensing element 721a, the second mask pattern provided on the second sensing element 721b, the third mask pattern provided on the third sensing element 721c, the fourth mask pattern provided on the fourth sensing element 721d, and the fifth mask pattern provided on the fifth sensing element 721e may be the same.

[0128] Hereinafter, reference will be made to Figure 8 the design of the mask pattern of the mask array 724.

[0129] Figure 8 A method of designing a mask pattern of a mask array according to an exemplary embodiment is shown.

[0130] A plurality of mask elements may be formed in a pattern in which a cost function determined based on the frequency response characteristics of a filter is minimized. The mask pattern provided for each sensing element or for each group of sensing elements in the mask array may be determined as a pattern in which the Euclidean norm of 1 / H(f), which is the reciprocal of the frequency response characteristics of the blur kernel, is minimized.

[0131] Referring to Figure 8 , in operation 810, a target aperture ratio of the mask array may be set. The aperture ratio may be the ratio of the light to be transmitted to the light incident on the pattern area corresponding to the mask pattern, and may represent the ratio of the area of the opening area to the area of the pattern area. The pattern area may be an area corresponding to a sensing element or a group of sensing elements in the mask array. The aperture ratio may be set to a ratio of, for example, 10% to 90% or a ratio of 30% to 70%. Desirably, the aperture ratio may be set to a ratio of 40% to 60%.

[0132] In operation 820, a mask pattern according to a set aperture ratio may be generated. The mask array may be divided into pattern regions corresponding to respective sensing elements, and each of the pattern regions may be divided into a plurality of spaces. A single space may define a unit region in which a mask element may be formed. A space in which a mask element is formed may be referred to as a "closed space", and a space in which a mask element is not formed may be referred to as an "open space". In other words, the closed spaces in the mask array may absorb all or part of the light, and the open spaces may allow the light to pass through. A mask pattern that combines the open spaces and the closed spaces may be generated based on the target aperture ratio set in operation 810. If a pattern region includes "N×N" spaces and the target aperture ratio is 50%, then "N 2 / 2" mask patterns may be generated. The closed spaces may be used to classify each of the mask elements according to the transmittance (or transmission level) of each mask element, which will be described below with reference to Figure 10 .

[0133] For ease of description, an example of generating the entire combination of mask patterns is shown in operation 820. However, the example embodiments are not limited thereto. In one example, predefined patterns may be combined, or mask patterns to be searched may be generated by swapping or switching between patterns within a given combination. For example, if a condition of pattern symmetry is added, then patterns may be searched only with respect to the repeated regions, and the generated patterns may be used symmetrically.

[0134] In operation 830, a cost function value may be calculated for each generated mask pattern. A cost function based on the reciprocal of the frequency response characteristic of the blur kernel described above with reference to Figure 5 and Figure 6 may be used. Since the mask elements are formed based on the mask pattern, the blur kernel of the imaging optical system may be modified, and the frequency response characteristic H'(f) of the modified blur kernel may be calculated. A cost function for the reciprocal 1 / H'(f) of the frequency response characteristic H'(f) of the modified blur kernel may be determined based on the Euclidean norm of the reciprocal of the function representing the frequency response characteristic, the variance of the reciprocal, and / or the reciprocal of the Euclidean norm of the function. For example, a cost function E shown in Equation 10 below may be used.

[0135] [Equation 10]

[0136]

[0137] In Equation 10, represents the Euclidean norm of the reciprocal of the frequency response characteristic H'(f). The number of zero crossings may be minimized by . Represents the reciprocal of the Euclidean norm of the frequency response characteristic H'(f). By the value of the frequency response characteristic H'(f) can be designed to be non-zero. Represents the variance of the reciprocal of the frequency response characteristic H'(f). 1 / H'(f) can be designed such that the variance can be minimized in the frequency domain. The cost function E of Equation 10 can be a weighted average of the respective cost factors, and α, β, and γ represent the weights.

[0138] In operations 840 and 850, the mask pattern in which the cost function value is minimized can be searched for and determined. The cost function value of each in the mask pattern having a combination of closed spaces and open spaces can be calculated under given conditions, and the mask pattern that minimizes the noise amplification through the cost function can be searched for. In operation 850, the mask pattern found as a search result can be determined as the pattern of the mask array. The image sensor can include a mask array in which the found mask pattern is repeated.

[0139] Figure 9 and Figure 10 shows an example of a mask array according to an exemplary embodiment.

[0140] Figure 9 shows an example in which the light transmission state of each individual space of the mask array 924 is classified into a binary state. In other words, the open space 991 can allow all the light incident thereon to pass through, and the closed space 992 can absorb all the light incident thereon.

[0141] The mask array 924 can be divided into a plurality of group regions. The group region 924a can be a region in the mask array 924 that covers a single sensing element or a plurality of sensing elements of the adjacent sensing array 921. Although Figure 9 an example in which the group region 924a of the mask array 924 covers a single sensing element is shown, the group region 924a of the mask array 924 can cover a sensing element group including a plurality of sensing elements grouped to represent a single pixel, which will be described below with reference to Figure 11 The mask pattern 990 of the group region 924a can be repeated. For example, all of the plurality of group regions of the mask array 924 can have the same mask pattern 990.

[0142] The pattern area of the mask array 924 corresponding to a single sensing element may include an aperture area and a mask area. The aperture area may occupy an area corresponding to the aperture ratio with respect to the total area of the corresponding area, and the mask area may occupy the remaining area. For example, the mask pattern 990 may be a pattern designed with a target aperture ratio of 50%. The mask pattern 990 may be divided into a total of "7×7 = 49" spaces, and may include "24" closed spaces 992 and "25" open spaces.

[0143] In addition, the mask pattern 990 may ensure the aperture ratio of the partial area 995 and the aperture ratio with respect to the total area. The area occupied by the aperture portion in the partial area 995 may be greater than or equal to the area corresponding to the set aperture ratio. The partial area 995 of the mask pattern 990 may include a total of "4×4 = 16" spaces, and the "4×4 = 16" spaces include "8" closed spaces 992 and "8" open spaces 991, so the aperture ratio may be 50%. Even when considering a partial area including "4×4 = 16" spaces at another position of the mask pattern 990, the aperture ratio may be 50%. In other words, the mask pattern 990 may be designed such that regions having a target aperture ratio can be evenly distributed.

[0144] In addition, the number of spaces (e.g., the number of mask elements) included in the area corresponding to a single mask pattern 990 may be greater than or equal to the number of imaging optical lenses of the imaging optical system. As described above with reference to Figure 5 As mentioned, the information sensed by the sensing element will be blurred in proportion to the number of lenses in the MLA. In order to restore an image with high resolution by canceling the blurring level proportional to the number of lenses, the pattern area may include spaces corresponding to the number of lenses for providing a deblurring function. When the lens array 910 includes "7×7 = 49" imaging optical lenses, the mask pattern 990 may include at least "7×7 = 49" spaces. The information transmitted through the lens array 910 to the sensing array 921 may be blurred by 1 / 49, while the mask pattern 990 may provide 49 times the corresponding deblurring ability.

[0145] Figure 10 An example is shown in which a single mask element of the mask array has one of two or more transmission levels. The mask element may block and / or absorb a part of the light reaching the mask element, and the transmission level may indicate the level and / or ratio of the transmitted incident light. The mask element may be formed and / or arranged along the plane of the mask array based on the above mask pattern.

[0146] A single mask element may be divided into a plurality of regions, and the transmittance of the single mask element may be determined based on the ratio of the open region to the closed region among the plurality of regions.Figure 10 A mask pattern 1000 designed with five transmission levels is shown. Each mask element can be divided into four regions of the same size. The space 1010 where no mask element is formed can have a first transmission level (e.g., 100% transmission). The first mask element 1021 can have a second transmission level (e.g., 75% transmission) and can include a single enclosed space. The second mask element 1022 can have a third transmission level (e.g., 50% transmission) and can include an enclosed region corresponding to half of the regions (e.g., two regions). The third mask element 1023 can have a fourth transmission level (e.g., 25% transmission) and can include three enclosed regions. The fourth mask element 1024 can have a fifth transmission level (e.g., 0% transmission), and all regions of the fourth mask element 1024 can be enclosed regions. The aperture ratio determined based on the number of open regions and the number of enclosed regions or the area of the open regions and the area of the enclosed regions can be interpreted as the ratio of the amount of light to be transmitted to the amount of light incident on the region.

[0147] Figure 11 The arrangement of mask patterns for each sensing element group in an image sensor according to an exemplary embodiment is shown.

[0148] Figure 9 An example in which a single sensing element is covered by each group region of a mask array is shown, Figure 11 An example in which a sensing element group 1121 including a plurality of sensing elements 1121a is covered by each group region 1141 of a mask array 1140 is shown. In Figure 11 this example, the pixel value of an image pixel can be determined based on the value sensed by the sensing element group 1121 (e.g., “2×2” sensing elements 1121a). The mask array 1140 can include a plurality of mask patterns. As Figure 9 shown, a single mask pattern can be set for each sensing element, while as Figure 11 shown, a single mask pattern 1190 can be set for each sensing element group 1121. All mask patterns of the mask array 1140 can have the same shape.

[0149] Figure 12A and Figure 12B Examples of the arrangement of a mask array according to an exemplary embodiment are shown.

[0150] The mask array can be disposed at various positions where light guided in a predetermined direction in a sensor for a camera can be blocked and / or overlapped. Referring to Figure 12A , the mask array 1240a can be disposed on the color filter 1220 between the condenser lens array 1230 and the sensing array 1210. Referring to Figure 12B, the mask array 1240b may be disposed between the color filter 1220 and the sensing array 1210. As described above, the mask array 1240b and the sensing array 1210 may be spaced apart from each other by 1 μm or less.

[0151] Figure 13 is a block diagram showing the configuration of an imaging device according to an exemplary embodiment.

[0152] The imaging device 1300 may include a lens array 1310 and an image sensor.

[0153] The lens array 1310 may include imaging optical lenses configured to transmit light received from the outside.

[0154] The image sensor may be a sensor that senses light passing through the lens array 1310. The image sensor may include a mask array 1324, a sensing array 1321, and a processor 1330. The mask array 1324 and the sensing array 1321 have been described above with reference to FIGS. 1 to Figure 12B and thus their further description will not be repeated here.

[0155] The processor 1330 may restore an image based on the sensing information sensed by the sensing elements. The processor 1330 of the image sensor may also be referred to as, for example, an image signal processor (ISP). The processor 1330 may generate frequency information by transforming the sensing information into the frequency domain, and may generate deblurred frequency information by dividing the frequency information by the frequency conversion result of the blur kernel corresponding to the mask pattern of the mask array 1324. The processor 1330 may restore a high-resolution image by inverse-transforming the deblurred frequency information into the time domain. In addition to image restoration, the sensing information may also be used, for example, in depth estimation for an object, refocusing, dynamic range imaging, and capturing high-sensitivity images in low-light environments.

[0156] Figure 14 is a block diagram showing the configuration of an electronic terminal according to an exemplary embodiment.

[0157] The electronic terminal 1400 may include an imaging module 1410 and a processor 1420.

[0158] The imaging module 1410 may include a lens array 1411 and an image sensor. The image sensor may include a mask array 1412 and a sensing array 1413. Different from the processor 1330 included in the image sensor as shown in Figure 13 shown, Figure 14 it shows that the processor is placed independently of the image sensor. Since the lens array 1411, the image sensor, and the processor 1420 have been described above, their further description will not be repeated here. Figure 14The processor 1420 can be an application processor (AP).

[0159] Figure 15 and Figure 16 is a diagram showing an example of a device in which an image sensor is to be implemented according to an example embodiment.

[0160] The image sensor and / or imaging device can be applied to various technical fields. A lens array including a plurality of lenses and a sensor including a plurality of sensing elements can be designed to be spaced apart from each other by a relatively short focal length, and the imaging device can be implemented as an ultra-thin camera having a small thickness and a large sensor for high-resolution capture.

[0161] The image sensor and / or imaging device can be mounted on a mobile terminal. The mobile terminal can be a movable terminal not fixed in any position, and can include, for example, a vehicle, an artificial intelligence speaker, and a portable device (such as a smart phone, a tablet personal computer (PC), or a foldable smart phone).

[0162] As Figure 15 shown, the imaging module 1510 can be applied to the front camera or the rear camera of the smart phone 1500. The imaging module 1510 can have a structure in which a large full-frame sensor and an MLA are combined, and can be applied to the camera of the smart phone.

[0163] In addition, the imaging module 1510 can be implemented in a vehicle in a thin structure or a curved structure. As Figure 16 shown, the imaging device 1610 can be implemented as a front camera or a rear camera having a curved shape in the vehicle 1600. Additionally, the imaging device 1610 can also be applied to fields such as a digital single-lens reflex (DSLR) camera, a drone, a closed-circuit television (CCTV), a webcam camera, a panoramic camera, a movie or broadcast video camera, a virtual reality (VR) / augmented reality (AR) camera, a flexible / stretchable camera, a compound eye camera, or a contact lens-type camera. Furthermore, the imaging device 1610 can also be applied to multi-frame super-resolution image restoration, which is used to increase the resolution based on information about a plurality of captured frames.

[0164] The units described herein can be implemented using hardware components and software components. For example, the hardware components can include a microphone, an amplifier, a band-pass filter, an audio-to-digital converter, a non-transitory computer memory, and a processing device. The processing device can be implemented using one or more general-purpose or special-purpose computers (such as, by way of example, a processor, a controller, and an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of responding and executing instructions in a defined manner). The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and create data in response to the execution of the software. For simplicity purposes, the description of the processing device is used as a singular; however, those skilled in the art will understand that the processing device can include multiple processing elements and multiple types of processing elements. For example, the processing device can include multiple processors or a processor and a controller. Additionally, different processing configurations (such as, parallel processors) are possible.

[0165] The software can include a computer program, a piece of code, instructions, or some combination thereof, to independently or jointly direct or configure the processing device to operate as required. The software and data can be permanently or temporarily implemented in any type of machine, component, physical or virtual device, computer storage medium or device, or in a propagated signal wave capable of providing instructions or data to the processing device or being interpreted by the processing device. The software can also be distributed over networked computer systems such that the software is stored and executed in a distributed manner. The software and data can be stored by one or more non-transitory computer-readable recording media.

[0166] The method according to the above exemplary embodiments can be recorded in a non-transitory computer-readable medium, which includes program instructions for implementing various operations executable by a computer. The medium may also include data files, data structures, etc. either alone or in combination with the program instructions. The program instructions recorded on the medium may be program instructions specially designed and constructed for the purposes of the exemplary embodiments, or they may be of the types well-known to those skilled in the art of computer software and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical media (such as compact disc read-only memory (CDROM) and digital versatile disc (DVD)), magneto-optical media (such as optical discs), and hardware devices specially configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory, etc.). Examples of program instructions include both machine code (such as code generated by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. The described hardware devices may be configured to act as one or more software modules to perform the operations of the above exemplary embodiments, and vice versa.

[0167] According to an example embodiment, at least one of the components, elements, modules, and units described herein can be implemented as various numbers of hardware structures, software structures, and / or firmware structures that perform the corresponding functions described above. For example, at least one of these components, elements, and units can use a direct circuit structure (such as a memory, a processor, a logic circuit, a lookup table, etc.) that can perform the corresponding functions under the control of one or more microprocessors or other control devices. In addition, at least one of these components, elements, and units can be specifically implemented as a part of a module, program, or code that includes one or more executable instructions for performing a specified logical function and is executed by one or more microprocessors or other control devices. In addition, at least one of these components, elements, and units can also include a processor (such as a central processing unit (CPU), a microprocessor, etc.) that performs the corresponding function or is implemented by a processor (such as a central processing unit (CPU), a microprocessor, etc.) that performs the corresponding function. Two or more of these components, elements, or units can be combined into a single component, element, or unit that performs all the operations or functions of the combined two or more components, elements, or units. In addition, at least a part of the function of at least one of these components, elements, and units can be performed by other components, elements, or units among these components, elements, or units. In addition, although a bus is not shown in the block diagram, communication between components, elements, or units can be performed through a bus. The functional aspects of the above example embodiments can be implemented in an algorithm executed on one or more processors. In addition, the components, elements, or units represented by blocks or processing operations can employ any number of related technologies for electronic configuration, signal processing, and / or control, data processing, etc.

[0168] Although the present disclosure includes example embodiments, it will be apparent to those of ordinary skill in the art that various changes in form and detail can be made in these example embodiments without departing from the spirit and scope of the claims and their equivalents. The example embodiments described herein will be considered only as descriptive and not for the purpose of limitation. The description of each feature or aspect in each example will be considered applicable to similar features or aspects in other examples. Appropriate results can be achieved if the described techniques are performed in a different order and / or if the components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents.

[0169] Therefore, the scope of the disclosure is not defined by the specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be construed as being included in the disclosure.

Claims

1. An image sensor, the image sensor comprising: A mask array including a plurality of mask elements configured to block light in a first portion of the plurality of directions among light passing through an imaging optical lens and incident on the mask array in a plurality of directions and allow light in a second portion of the plurality of directions to pass through the plurality of mask elements; A sensing array including a plurality of sensing elements configured to sense light passing through the imaging optical lens and the mask array, and A processor configured to restore an image based on sensing information sensed by the plurality of sensing elements, wherein, for each individual imaging optical lens, a low-resolution image is captured, wherein the processor is configured to reconstruct the image by rearranging positions of pixels in the low-resolution image such that sensing information corresponding to light rays passing through different individual imaging optical lenses is adjacent to each other.

2. The image sensor according to claim 1, further comprising: A color filter disposed above the sensing array and configured to filter a portion of a wavelength band of light incident on each of the plurality of sensing elements, wherein the mask array is disposed between the color filter and the sensing array.

3. The image sensor according to claim 1, further comprising: A condenser lens array disposed above the sensing array, wherein the mask array is disposed between the condenser lens array and the sensing array.

4. The image sensor according to claim 1, wherein, The mask array and the plurality of sensing elements are spaced apart by 1 micrometer or less.

5. The image sensor according to claim 1, wherein, The mask array and the plurality of sensing elements are in contact with each other.

6. The image sensor according to claim 1, wherein, A first region of the mask array corresponding to a sensing element among the plurality of sensing elements includes: An aperture region occupying an area corresponding to an aperture ratio with respect to the total area of the first region; and A mask region occupying the remaining area of the first region, and the plurality of mask elements are disposed in the mask region.

7. The image sensor according to claim 6, wherein, The aperture ratio is between 40% and 60%.

8. The image sensor according to claim 1, wherein, In a partial region of the mask array, the area occupied by the aperture is greater than or equal to the area corresponding to the set aperture ratio.

9. The image sensor according to claim 1, wherein, The mask array is divided into a plurality of group regions corresponding to a plurality of groups of sensing elements, and Each group region in the plurality of group regions in the mask array is configured to cover a group of sensing elements, the group of sensing elements including a plurality of sensing elements grouped to represent a single pixel.

10. The image sensor according to claim 9, wherein, The mask pattern of the group region is repeated in the mask array.

11. The image sensor according to claim 9, wherein, All of the plurality of group regions in the mask array have the same mask pattern.

12. The image sensor according to claim 1, wherein, The number of mask elements included in the mask pattern is greater than or equal to the number of imaging optical lenses, and the mask pattern is repeated in the mask array.

13. The image sensor according to claim 1, wherein, The plurality of mask elements have two or more transmission levels.

14. The image sensor according to claim 1, wherein, Each mask element in the plurality of mask elements is divided into a plurality of regions, and The transmittance of each mask element in the plurality of mask elements is determined based on a ratio of an opening region to a closed region among the plurality of regions.

15. The image sensor according to claim 1, wherein, The processor is further configured to: generate frequency information by transforming the sensed information into the frequency domain, generate deblurred frequency information by dividing the frequency information by the frequency transformation result of the blur kernel, and restore a high-resolution image by inverse-transforming the deblurred frequency information into the time domain, where the blur kernel corresponds to the mask pattern of the mask array.

16. The image sensor according to any one of claims 1 to 14, wherein the mask array includes a plurality of mask patterns, and each of the plurality of mask patterns is configured to cover a group of sensing elements, the group of sensing elements including two or more sensing elements in the sensing array.

17. A camera device, the camera device comprising: an imaging lens array including imaging optical lenses configured to transmit light received from outside the camera device; a sensing array including a plurality of sensing elements configured to sense light passing through the imaging lens array; a mask array including a plurality of mask elements, the mask array being disposed between the imaging lens array and the sensing array, wherein the plurality of mask elements are configured to: block light guided in a part of directions among the light passing through the imaging lens array and allow light guided in other directions to pass through the plurality of mask elements; and a processor configured to restore an image based on sensed information sensed by the plurality of sensing elements, wherein, for each individual imaging optical lens, a low-resolution image is captured, wherein the processor is configured to: reconstruct the image by rearranging positions of pixels in the low-resolution image such that the sensed information corresponding to light rays passing through different individual imaging optical lenses is adjacent to each other.

18. The camera device according to claim 17, wherein, The mask array is disposed at one of a position inside the sensing array and a position in contact with the sensing array on the plurality of sensing elements.

19. The camera device according to claim 17, wherein, A first region of the mask array corresponding to a sensing element among the plurality of sensing elements includes: an aperture region occupying an area corresponding to an aperture ratio with respect to the total area of the first region; and a mask region occupying the remaining area in the first region, where the plurality of mask elements are disposed in the mask region.

20. The camera device according to claim 17, wherein, In a partial region of the mask array, the area occupied by the aperture is greater than or equal to the area corresponding to the set aperture ratio.

21. The camera device according to claim 17, wherein, The mask array is divided into a plurality of group regions corresponding to a plurality of groups of sensing elements, and the mask patterns of the group regions are repeated in the mask array.

22. The camera device according to any one of claims 17 to 21, Among them, the processor is further configured to: generate frequency information by transforming the sensed information sensed by the plurality of sensing elements into the frequency domain, generate deblurred frequency information by dividing the frequency information by the frequency transformation result of the blur kernel, and restore a high-resolution image by inverse-transforming the deblurred frequency information into the time domain, where the blur kernel corresponds to the mask pattern of the mask array.

Images