Image sensor and image sensing method

Abstract

An image sensor and an image sensing method are provided. The image sensor can restore a high-resolution image for a high magnification based on sensing information corresponding to different fields of view (FOV) and received through lens elements having the same focal length and different lens sizes.

Description

[0001] This application claims priority to U.S. Provisional Application No. 62 / 741,147, filed in the U.S. Patent and Trademark Office on October 4, 2018, and Korean Patent Application No. 10-2018-0154741, filed in the Korean Intellectual Property Office on December 4, 2018, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] Example embodiments consistent with the present disclosure relate to techniques for sensing images. Background Art

[0003] Due to the development of optical technology and image processing technology, image capture devices are being used in a wide range of fields (e.g., multimedia content, security, and identification). For example, an image capture device may be installed in a mobile device, a camera, a vehicle, or a computer to capture images, identify objects, or obtain data for controlling a device. The size of the image capture device may be determined based on, for example, the size of the lens, the focal length of the lens, or the size of the sensor. For example, the size of the image capture device may depend on the size of the lens or the size of the sensor. As the size of the sensor decreases, the amount of light incident on the sensor may decrease. Therefore, the resolution of the image may be reduced, or it may be difficult to capture an image in a low-light environment. To reduce the size of the image capture device, a multi-lens including a small lens may be used. When the size of the lens is reduced, the focal length of the lens may be reduced. Therefore, the size of the image capture device may be reduced based on the multi-lens. Summary of the invention

[0004] One or more example embodiments may address at least the above problems and / or disadvantages and other disadvantages not described above. Also, an example embodiment is not required to overcome the disadvantages described above, and an example embodiment may not overcome any of the problems described above.

[0005] One or more example embodiments provide an image sensor and an image sensing method.

[0006] According to one aspect of the present disclosure, an image sensor includes a lens array and a plurality of sensing elements, wherein the lens array includes a plurality of lens elements, wherein each of the plurality of lens elements has the same focal length; wherein the plurality of sensing elements are separated from the lens array by the focal length and are configured to sense light passing through the lens array, wherein a lens size of at least one lens element among the plurality of lens elements is different from a lens size of another lens element among the plurality of lens elements.

[0007] The plurality of lens elements may be classified into a plurality of lens groups based on lens size, each lens group among the plurality of lens groups may correspond to a respective lens size, so that for each lens group among the plurality of lens groups, the lens elements included in the lens group have a corresponding lens size; and the respective lens size corresponding to each lens group among the plurality of lens groups may be different from the respective lens sizes of the remaining lens groups among the plurality of lens groups.

[0008] Each lens group among the plurality of lens groups may be configured to cover the same number of sensing elements among the plurality of sensing elements, and the number of lens elements included in each lens group among the plurality of lens groups may be determined based on the number of covered sensing elements.

[0009] For each lens group among the plurality of lens groups, lens elements included in the lens group may be positioned adjacent to each other.

[0010] A lens group among the plurality of lens groups may include a single lens element.

[0011] A first lens element among the plurality of lens elements may be positioned closer to a center position of the lens array than a second lens element among the plurality of lens elements, wherein the second lens element has a lens size greater than a lens size of the first lens element.

[0012] Each lens element of the plurality of lens elements may be randomly positioned relative to the plurality of sensing elements on a plane corresponding to the lens array.

[0013] Each lens element among the plurality of lens elements may be configured to cover a same number of sensing elements among the plurality of sensing elements.

[0014] At least one lens element among the plurality of lens elements may be arranged to cover less than an entire portion of at least one sensing element among the plurality of sensing elements.

[0015] The image sensor may further include a processor configured to restore an image based on sensing information sensed by the plurality of sensing elements so that a resolution of a central area within a field of view (FOV) of the lens array is higher than a resolution of an area adjacent to the central area.

[0016] The image sensor may further include a processor configured to acquire a compound eye vision (CEV) image based on sensing information sensed by the plurality of sensing elements.

[0017] The processor may be further configured to rearrange pixels included in the CEV image based on light field (LF) information sensed by the plurality of sensing elements.

[0018] The processor may be further configured to restore a scene image from the CEV image based on a geometric relationship between the plurality of sensing elements and the plurality of lens elements.

[0019] The processor may be further configured to restore the scene image from the CEV image based on an image restoration model that is fully trained before the CEV image is acquired.

[0020] The image sensor may further include: a processor configured to: select target sensing information from the sensing information sensed by the plurality of sensing elements, wherein the target sensing information corresponds to a scaling factor specified by a user; and the processor may further be configured to: restore a scene image based on the selected target sensing information.

[0021] The processor may be further configured to select information corresponding to a field of view corresponding to a specified zoom factor as the target sensing information.

[0022] Each lens element of the plurality of lens elements may be configured to refract incident light and form a focus of the light exiting the lens element at a point on a sensing array comprising the plurality of sensing elements.

[0023] The multiple lens elements can be classified into multiple lens groups based on the field of view (FOV), each lens group among the multiple lens groups corresponding to a respective FOV, so that the respective FOV corresponding to each lens group among the multiple lens groups is different from the respective FOVs of the remaining lens groups among the multiple lens groups; for each lens group among the multiple lens groups, the sensing elements among the multiple sensing elements included in the sensing area covered by the lens group can be configured to: sense light corresponding to the FOV corresponding to the lens group.

[0024] The image sensor may be included in a mobile terminal.

[0025] According to one aspect of the present disclosure, an image sensing method includes: sensing light passing through multiple lens elements through multiple sensing elements, wherein each of the multiple lens elements has the same focal length; and restoring a scene image through a processor based on the intensity of the light sensed by the multiple sensing elements, wherein a lens size of at least one lens element among the multiple lens elements is different from a lens size of another lens element among the multiple lens elements.

[0026] According to one aspect of the present disclosure, a camera includes: a lens array configured to refract incident light reflected from an imaging subject; and a sensor configured to sense the light refracted by the lens array, wherein the lens array includes a first lens having a first diameter and a second lens having a second diameter different from the first diameter.

[0027] The lens array may include a plurality of first lenses having a first diameter and a plurality of second lenses having a second diameter.

[0028] A first area of ​​the sensor covered by the plurality of first lenses and a second area of ​​the sensor covered by the plurality of second lenses may have the same size.

[0029] The sensor may include a plurality of sensing elements, and each of the plurality of first lenses and the plurality of second lenses may cover the same number of sensing elements among the plurality of sensing elements.

[0030] The number of sensing elements may be a non-integer number.

[0031] Each of the plurality of sensing elements may be configured to sense a first light ray reflected from a first point of the imaging subject and a second light ray from a second point of the imaging subject.

[0032] The plurality of sensing elements may include a first sensing element configured to sense a first light refracted by a first lens having a first diameter and a second light refracted by another lens among the plurality of first lenses having the first diameter.

[0033] The plurality of first lenses may correspond to a first field of view, and the plurality of second lenses may correspond to a second field of view different from the first field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or other aspects will become more apparent through the following description of example embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a diagram illustrating a structure of an image sensor according to example embodiments;

[0036] Figure 2 is a diagram illustrating an example in which a sensing element receives light passing through a lens element according to example embodiments;

[0037] Figure 3 is a diagram illustrating a relationship between the number of sensing elements and the number of lens elements according to example embodiments;

[0038] Figure 4 is a diagram illustrating the number of lens elements and the sizes of lens elements determined based on a focal length and a desired field of view (FOV) in a lens array according to example embodiments;

[0039] Figure 5 is a diagram illustrating an arrangement of lens elements in a lens array according to example embodiments;

[0040] Figure 6 It is shown that the above reference Figure 5 a diagram of the FOV of a scene sensed by the lens element of the arrangement through the sensing element;

[0041] Figures 7 to 10 is a diagram showing an example of an arrangement of lens elements according to example embodiments;

[0042] Fig.11 is a block diagram illustrating a structure of an image sensor according to example embodiments;

[0043] Fig.12 and Fig.13 is a diagram illustrating an example of a device in which an image sensor is implemented according to example embodiments. DETAILED DESCRIPTION

[0044] Reference will now be made in detail to example embodiments, which are illustrated in the drawings, wherein like reference numerals refer to like elements throughout. Example embodiments are described below so as to explain the present disclosure by referring to the drawings.

[0045] The following description of structures or functions only describes example embodiments, and the scope of the example embodiments is not limited to the description provided in this specification. A person of ordinary skill in the art may make various changes and modifications to the example embodiments.

[0046] It should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of the stated features, integers, steps, operations, elements, components, or combinations thereof, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. When a statement such as "at least one of..." follows a list of elements, it modifies the entire list of elements rather than a single element in the list. For example, the statement "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0047] Unless otherwise defined herein, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. Unless otherwise defined herein, terms defined in general dictionaries should be interpreted as having a meaning that matches the contextual meaning in the relevant art and should not be interpreted as an idealized or overly formal meaning.

[0048] Regarding the reference numerals assigned to elements in the drawings, it should be noted that the same reference numerals are designated by the same elements wherever possible even if they are shown in different drawings.

[0049] Figure 1is a diagram illustrating a structure of an image sensor according to example embodiments.

[0050] The quality of an image captured and restored by the image sensor 100 may be determined based on the number of sensing elements included in the sensing array 120 and the amount of light incident on the sensing elements. For example, the resolution of an image may be determined based on the number of sensing elements included in the sensing array 120, and the sensitivity of the image may be determined based on the amount of light incident on the sensing elements. The amount of light incident on the sensing elements may be determined based on the size of the sensing elements. When the size of the sensing elements increases, the amount of light incident on the sensing elements and the dynamic range of the sensing array 120 may increase. On the other hand, the resolution of an image captured by the image sensor 100 may increase as the number of sensing elements included in the sensing array 120 increases. In addition, when the size of the sensing elements increases, the image sensor 100 may improve the quality of high-sensitivity imaging under low illumination.

[0051] The size (volume) of the image sensor 100 may be determined based on the focal length f of the lens element 111. For example, the size of the image sensor 100 may be determined based on the gap between the lens element 111 and the sensing array 120. Because the sensing array 120 needs to be placed at the focal point of the lens element 111 to collect the light 190 refracted by the lens element 111, the sensing array 120 and the lens element 111 included in the image sensor 100 may need to be separated from each other by a distance equal to the focal length f of the lens element 111. The focal length f of the lens element 111 may be determined based on the field of view (FOV) of the image sensor 100 and the size of the lens element 111 (e.g., the radius of the aperture of the lens element 111). For example, when the FOV is fixed, the focal length f may increase in proportion to the size of the lens element 111. In addition, the size of the lens element 111 may be determined based on the size of the sensing array 120. For example, in order to capture an image within a predetermined FOV, the size of the lens element 111 may need to increase as the size of the sensing array 120 increases.

[0052] As described above, in order to increase the sensitivity of the image while maintaining the FOV and the resolution of the image, the size of the image sensor 100 may be increased. For example, in order to increase the sensitivity of the image while maintaining the resolution of the image, it may be necessary to increase the size of each sensing element while maintaining the same number of sensing elements included in the sensing array 120. Therefore, the size of the sensing array 120 may be increased. In this example, in order to maintain the FOV, the size of the lens element 111 may be increased as the size of the sensing array 120 increases, and the focal length f of the lens element 111 may be increased. Therefore, the size of the image sensor 100 may be increased.

[0053] In order to reduce the size of the image sensor 100, a design scheme for reducing the size of the sensing element while maintaining the resolution of the sensing array 120 or a design scheme for reducing the resolution of the sensing array 120 while maintaining the size of the sensing element may be used. In one example, when reducing the size of the sensing element while maintaining the resolution of the sensing array 120, the size of the sensing array 120 and the focal length f of the lens element 111 may be reduced, which may result in a reduction in the size of the image sensor 100. However, in this example, the sensitivity of the image may also be reduced, and the quality of the image may be reduced under low illumination. In another example, when reducing the resolution of the sensing array 120 while maintaining the size of the sensing element, the size of the sensing array 120 and the focal length f of the lens element 111 may be reduced, which may result in a reduction in the size of the image sensor 100. However, in this example, the resolution of the image may also be reduced.

[0054] Reference Figure 1 , the image sensor 100 may include a lens array 110 and a sensing array 120 . The lens array 110 may include lens elements 111 , and the sensing array 120 may include sensing elements 121 .

[0055] When the size of each lens element included in the lens array 110 is reduced, that is, when the number of lens elements per unit area of ​​the lens array 110 is increased, the focal length f of the lens element 111 and the thickness of the image sensor 100 can be reduced. In this example, the image sensor 100 can restore a high-resolution image by rearranging and combining low-resolution images obtained based on light passing through the respective lens elements 111. Therefore, a thinner camera can be realized by increasing the number of lens elements in the lens array 110.

[0056] like Figure 1 As shown in , the lens element 111 of the lens array 110 may cover a predetermined sensing area 125 of the sensing array 120 corresponding to the size of the lens element 111. The sensing area 125 covered by the lens element 111 may be determined based on the size of the lens element 111. The sensing area 125 may represent an area on the sensing array 120 reached by light within a predetermined FOV passing through the lens element 111. The size of the sensing area 125 may be represented as a distance D from a center point of the sensing area 125 to an outermost point of the sensing area 125. Light 190 passing through the lens element 111 may be incident on the sensing elements of the sensing array 120 included in the sensing area 125. The light 190 may include a plurality of light rays. The light 191 may correspond to a flow (i.e., a path) of the photon 101.

[0057] For example, the lens size may correspond to the diameter of the lens.

[0058] Each sensing element in the sensing array 120 may generate sensing information based on incident light 191 passing through the lens element 111 included in the lens array 110. For example, the sensing element 121 may generate sensing information based on the light 191 incident through the lens element 111. The image sensor 100 may determine intensity information corresponding to a raw signal associated with a point included in the FOV of the image sensor 100 based on the sensing information output from the sensing array 120. In addition, the image sensor 100 may restore a captured image based on the determined intensity information.

[0059] The sensing element 121 may also include a color filter for sensing an arbitrary color. In this case, the sensing element 121 may generate a color value corresponding to a predetermined color as sensing information. Each sensing element included in the sensing array 120 may be arranged to sense a color different from the color of an adjacent sensing element spatially adjacent to the sensing element. However, the arrangement of the sensing element 121 and the color filter of the sensing element 121 is not limited thereto.

[0060] When the diversity of the sensing information is sufficiently ensured, and when a full rank relationship is formed between the sensing information and the raw signal information corresponding to the point in the FOV of the image sensor 100, a captured image corresponding to the maximum resolution of the sensing array 120 can be obtained. The diversity of the sensing information may be ensured based on the parameters of the image sensor 100 (e.g., the number of lens elements included in the lens array 110 or the number of sensing elements included in the sensing array 120).

[0061] Although not in Figure 1 , but the image sensor 100 may include a memory and a processor, wherein the memory is configured to store an image restoration model for restoring an image, and the processor is configured to restore the image using the image restoration model stored in the memory.

[0062] The image sensor 100 can restore an image robustly against various noise components that deviate from ideal conditions. In addition, the image sensor 100 can restore an image without being affected by any pattern (such as a Bayer pattern) that may be arranged on a sensing element. An example of an operation of restoring a high-resolution image based on a plurality of low-resolution images captured by the image sensor 100 using a plurality of lenses will be described below. The plurality of low-resolution images used in combination to form a high-resolution image may be collectively referred to as a compound eye vision (CEV) image.

[0063] Figure 2 is a diagram illustrating an example in which a sensing element receives light passing through a lens element according to example embodiments.

[0064] As described above, the sensing array 220 may receive light rays corresponding to the respective points 230 (as shown in X1 to X10). The light rays may be detected by the sensing array 220 via the lens array 210. For example, a plurality of light rays may be emitted from each of the respective points 230. The light rays emitted from the same point may form a light field (LF). The LF emitted from any point on the subject may be a field indicating both the intensity and direction of the light rays reflected from any point on the subject. For example, the light rays emitted from the first point X1 may form a first LF and may be incident on the first sensing element S1, the fourth sensing element S4, and the seventh sensing element S7. The light rays emitted from each of the other points (i.e., point X2 to point X10) may also form a corresponding LF. For example, the respective points 230 may be points on any object that may be used as an imaging subject. For example, the light rays emitted from the respective points 230 may be light rays obtained when sunlight is reflected from the surface of the object. Although as Figure 2 As shown in , for convenience of description, the lens array 210 includes three lens elements, and the sensing array 220 includes ten sensing elements (eg, sensing elements S1 to S10 ), but example embodiments are not limited thereto.

[0065] The sensing elements S1 to S10 can sense light rays that pass through a plurality of lens elements and overlap with each other. Figure 2 In the lens array 210, the focal length between the lens element and the sensing array 220 can be reduced. Therefore, the first sensing element S1 can generate sensing information, for example, as the intensity value of the sum of the intensity values ​​of the light emitted from the point X1 to the point X3 and overlapping each other. Therefore, the information generated by each sensing element includes the original signal information from the plurality of points 230. For example, the image sensor can restore the original signal information from the sensing information using a model as described below.

[0066] As shown in the following Equation 1, sensing information generated by the sensing elements S1 to S10 may be modeled as raw signal information (eg, intensity values) corresponding to each of the rays incident from the respective points 230 .

[0067] [Equation 1]

[0068] S=T·X

[0069] In Equation 1, S represents a matrix indicating sensing information (e.g., detected intensity values) sensed by each sensing element. X represents a matrix indicating original signal values ​​(e.g., color values ​​of incident light) corresponding to light incident on the sensing elements S1 to S10 from each point. T represents a transformation matrix representing the relationship between the sensing information detected by the sensing elements S1 to S10 and the original signal information corresponding to the incident light. As shown in the following Equation 2, Figure 2The light, lens, and sensing elements S1 to S10 corresponding to each point X1 to X10 in the structure of FIG. 2 are modeled. In Equation 2, each point X1 to X10 may be modeled as being disposed at an infinite focus from the image sensor. For example, the distance between the image sensor and each of the points X1 to X10 may be greater than a threshold distance.

[0070] [Equation 2]

[0071]

[0072] For the convenience of description in Equation 2, X1 to X10 represent original signal information (e.g., light intensity values) of light corresponding to each of the individual points X1 to X10. In addition, S1 to S10 represent sensing information (e.g., sensing intensity values) detected by the sensing elements S1 to S10. For example, the relationship between the sensing information (e.g., color information) corresponding to the sensing elements S1 to S10 included in the sensing array 220 and the original signals corresponding to the light incident from the individual points X1 to X10 (e.g., the above-mentioned transformation matrix) can be determined based on the arrangement of the lens elements and the sensing elements, the number of lens elements included in the lens array 210, or the number of sensing elements S1 to S10 included in the sensing array 220. Figure 2 In the example shown in , the transformation matrix T is determined based on the presence of ten sensing elements S1 to S10 and three lens elements.

[0073] An example in which each of the points X1 to X10 is disposed at an infinite focus from the image sensor is described with reference to Equation 2. For example, when each of the points X1 to X10 is disposed at an infinite focus from the image sensor, a raw signal sensed by each sensing element may change based on the distance between the subject and the image sensor and the geometric structure of the image sensor.

[0074] Figure 3 is a diagram illustrating a relationship between the number of sensing elements and the number of lens elements according to example embodiments.

[0075] At least one of the plurality of lens elements may be arranged eccentrically relative to at least one of the plurality of sensing elements. In other words, at least one lens element may cover less than the entire portion of the at least one sensing element. For example, the plurality of lens elements may be arranged eccentrically and may not have a one-to-one relationship with the plurality of sensing elements. Instead of covering an integer number of sensing elements, the lens elements may cover a non-integer number of sensing elements. In one example, the multi-lens array structure may be implemented as a fractional alignment structure. For example, the parameters of the multi-lens array structure may include the number of sensing elements P and the number of lens elements L. The ratio P / L between the number of sensing elements P and the number of lens elements L may be determined as a non-integer rational number. Each lens element may cover the same number of sensing elements (i.e., P / L) as the pixel offset. Figure 3 In FIG. 1 , ten sensing elements and three lens elements are shown. Thus, each lens element covers 10 / 3 or approximately 3.33 sensing elements.

[0076] As described above, the image sensor may have an arrangement of lens elements, wherein the lens elements may have an optical center axis (OCA) slightly different from that of the sensing elements in the sensing array. Therefore, the lens elements included in the lens array may receive different LF information. Since the direction of the main light of each lens element is also changed, the image sensor may optically acquire a larger amount of sensing information. Therefore, the image sensor may restore an image with a higher resolution based on the various sensing information acquired as described above.

[0077] When the lens elements included in the lens array all have the same lens size, the number of lens elements in the lens array and the number of sensing elements included in the sensing array may satisfy Equation 3 as shown below. In this example, the number of lens elements and the number of sensing elements may be mutually prime.

[0078] [Equation 3]

[0079]

[0080] In Equation 3, R represents an integer indicating the resolution of the sensing array for one axis (i.e., the number of sensing elements in a single row of the sensing array), P represents an integer indicating the number of sensing elements in the sensing array for one axis, L represents an integer indicating the number of lens elements in the lens array for one axis (i.e., the number of lens elements in a single row of the lens array), Represents an arbitrary natural number used to satisfy Equation 3. In Equation 3, R, P, and L may be represented on a two-dimensional (2D) plane. For example, the resolution R may be the resolution of the image sensor for the horizontal axis or the vertical axis. The resolution R for the horizontal axis or the vertical axis may correspond to the number P of sensing elements in a single row in a direction corresponding to the horizontal axis or the vertical axis. In this example, L may indicate the number of lens elements based on the corresponding axis. Therefore, the full two-dimensional resolution of the sensing array may be represented by R×R, the total number of sensing elements in the two-dimensional sensing array may be P×P, and the total number of lens elements in the two-dimensional lens array may be L×L.

[0081] For example, Figure 3 A cross-sectional view of a horizontal axis or a vertical axis of an image sensor is shown. For one axis, the number of lens elements L can be 3, and the number of sensing elements P can be 10 to satisfy a coprime relationship. In this example, each lens element can cover approximately 3.33 sensing elements for the one axis.

[0082] exist Figure 3 , the first lens element may cover all of sensing elements S1 to S3 and 1 / 3 of sensing element S4. The second lens element may cover the remaining 2 / 3 of sensing element S4, all of sensing elements S5 and S6, and 2 / 3 of sensing element S7. Similarly, the last lens element may cover the remaining 1 / 3 of sensing element S7 and all of sensing elements S8 to S10. In other words, each lens element may cover a portion of at least one sensing element corresponding to the same parallax as 1 / L (i.e., a portion of the sensing element that is an integer multiple of 1 / L), where L is the number of lenses.

[0083] Based on the geometric structure of the lens array and the sensing array described above, the LF information from the combination of points 230 sensed by the sensing elements covered by one lens element may be different from the LF information from the combination of points 230 sensed by the sensing elements covered by another lens element. The LF information may represent information about the combination of multiple LFs from multiple points 230. For example, the first sensing element S1 may sense a signal having Figure 2 On the other hand, the second sensing element S2 adjacent to the first sensing element S1 may sense the LF information of the combination of the first LF of the first point X1, the second LF of the second point X2, and the third LF of the third point X3. Figure 2 LF information of a combination of a fourth LF at a fourth point X4, a fifth LF at a fifth point X5, and a sixth LF at a sixth point X6 in the structure of . Therefore, one sensing element may sense LF information different from LF information sensed by another sensing element.

[0084] The image sensor may rearrange the positions of pixels of the captured image based on the correlation between the LF information. For example, the image sensor may rearrange the pixels of the captured image (e.g., CEV image) to generate an output image in which pixels of the sensing elements sensing similar LF information among the plurality of sensing elements are adjacent to each other.

[0085] For example, Figure 3 As shown in , the image sensor may rearrange pixels indicating intensity values ​​of signals sensed by a separate sensing element based on similarity between LF information sensed by the sensing element and LF information sensed by another sensing element. For example, it may be determined that as the number of overlapping LFs in LF information sensed by the two sensing elements increases, the similarity between the LF information of the two sensing elements increases.

[0086] When assuming that the point of reflected light is at infinite focus from the image sensor, the image sensor can determine the LF information sensed by each sensing element. For example, the image sensor can determine the point from which the LF sensed by each of the multiple sensing elements is emitted based on the positional relationship between the multiple sensing elements and the light emitted from the point separated from the image sensor by a distance greater than a threshold distance. The point separated from the image sensor by a distance greater than the threshold distance can also be referred to as a "point at infinite focus". The image sensor can rearrange pixels so that pixels representing points in the imaged subject that are spatially adjacent to each other are adjacent to each other in the output image.

[0087] Figure 2 Each point X1 to X10 is Figure 2 . For example, the first point X1 may be adjacent to the second point X2, and the second point X2 may be adjacent to the first point X1 and the third point X3. For example, two adjacent points may be points in the subject that are spatially adjacent to each other.

[0088] The LF information sensed by the first sensing element S1 among the sensing elements 311 that are not rearranged and the LF information sensed by the eighth sensing element S8 may both include LF from both the second point X2 and the third point X3. Therefore, the first sensing element S1 and the eighth sensing element S8 may be determined to sense similar LF information. When pixels corresponding to similar LF information are rearranged to be adjacent to each other, Equation 2 may be expressed as shown in the following Equation 4.

[0089] [Equation 4]

[0090]

[0091] Reference Figure 3, the sensing element 311 is rearranged into the sensing element 312 based on equation 4. In an actual sensing array, the first sensing element S1 may be covered by the first lens, the eighth sensing element S8 may be covered by the third lens, and the fifth sensing element S5 may be covered by the second lens. After rearranging the pixels based on equation 4, the three pixels corresponding to the first sensing element S1, the eighth sensing element S8, and the fifth sensing element S5 are arranged adjacent to each other.

[0092] The image sensor may apply rearrangement and restoration to an image captured by a lens array and a sensing array satisfying the coprime condition and Equation 3 to restore a high-resolution scene image having colors more similar to the original scene.

[0093] Figure 4 is a diagram illustrating the number of lens elements and the sizes of lens elements determined based on a focal length and a desired FOV in a lens array according to example embodiments.

[0094] As described above, the image sensor may include the lens array 410 and the sensing array 420. The lens array 410 may include a plurality of lens elements. The lens size of at least one lens element among the plurality of lens elements may be different from the lens size of another lens element.

[0095] For example, a plurality of lens elements may be classified into a plurality of lens groups based on lens size. A lens group may be a group of lens elements classified by lens size. For example, lens elements included in the same lens group may have the same lens size, and lens elements included in different lens groups may have different lens sizes. In other words, the lens size corresponding to any one lens group may be different from the lens sizes corresponding to all other lens groups. An image sensor may include lens elements having various FOVs. The FOV of a lens element may be expressed as shown in the following Equation 5.

[0096] [Equation 5]

[0097]

[0098] In Equation 5, FOV represents FOV, D represents the size of the sensing area covered by the lens element, and F represents the focal length 419.

[0099] The image sensor may be implemented (i.e., included) in a mobile terminal, and the focal length 419 may be limited to meet the form factor of the mobile terminal. For example, the focal length 419 may be less than the thickness of the mobile terminal. In order to implement the image sensor in the mobile terminal, the lens groups may be designed so that they all have the same focal length 419. In addition, the lens groups may all have different FOVs from each other.

[0100] In order to acquire information of a distant subject, the image sensor may include a lens array 410 including lens groups with various FOVs. For example, when the focal length 419 is fixed for all lens elements as described above, and when the FOV of Equation 5 changes, the size of the sensing area covered by the lens elements (e.g., D of Equation 5) may change. For example, the lens array 410 may be designed so that the number of lens elements included in the lens group increases as the zoom factor of the lens group increases. In this example, based on the design of the lens array 410, the relationship between the FOV and the size of the sensing area enables the constant focal length of the lens group to be maintained. In addition, the lens size of the lens elements included in the lens group may decrease as the zoom factor of the lens group increases. The lens size of each lens element included in the lens group may decrease as the number of lens elements included in the lens group increases. Therefore, the number of lens elements included in any lens group may be determined based on the lens size, wherein the lens size itself may be determined based on the focal length and FOV of the lens group.

[0101] In addition, for lens group i, the number of lens elements shown along one axis of the lens array 410 (ie, the number of lens elements in each row of the lens group) may satisfy Equation 6 as shown below.

[0102] [Equation 6]

[0103]

[0104] In Equation 6, R i represents the resolution of the image sensed by the i-th lens group (i.e., the number of sensing elements in each row of the sensing area corresponding to the lens group), L i represents the number of lens elements corresponding to the i-th lens group along one axis of the lens array 410, represents an arbitrary natural number, and M represents the number of lens groups included in the lens array 410. i , L i , and M may each be an integer greater than or equal to 1. In addition, i may be an integer greater than or equal to 1 and less than or equal to M. For example, the number of i-th lens elements included in the i-th lens group may be L i ×L i .

[0105] Figure 4 4 is a cross-sectional view of a lens array 410 in which the number of lens groups M is 3. For ease of description, a first lens group 411, a second lens group 412, and a third lens group 413 are sequentially shown from the left. 1may represent the number of lens elements corresponding to the first lens group 411 along one axis of the lens array 410, and as Figure 4 As shown in L 1 It can be 3. 2 may represent the number of lens elements corresponding to the second lens group 412 along one axis of the lens array 410, and as Figure 4 As shown in L 2 It can be 9. 3 may represent the number of lens elements corresponding to the third lens group 413 along one axis of the lens array 410, and as Figure 4 As shown in L 3 It can be 27. As mentioned above Figure 3 As mentioned above, due to Figure 4 FIG. 1 also shows a cross-sectional view of the image sensor based on one axis, so L 1 , L 2 and L 3 Each may be the number of lens elements along the one axis. The first lens group 411 may include nine first lens elements obtained by 3×3 (i.e., three rows of lens elements, each row including three lens elements), the second lens group 412 may include 81 second lens elements obtained by 9×9 (i.e., nine rows of lens elements, each row including nine lens elements), and the third lens group 413 may include 729 third lens elements obtained by 27×27 (i.e., 27 rows of lens elements, each row including 27 lens elements).

[0106] The first lens group 411 may transmit light within a first FOV 451 (e.g., 77 degrees) to a first sensing element. The first sensing element may be covered by a first lens element included in the first lens group 411, and may receive light passing through the first lens element. The first sensing element may sense information corresponding to a first zoom factor (e.g., a zoom factor of 1). The second lens group 412 may transmit light within a second FOV 452 (e.g., 30 degrees) to a second sensing element. The second sensing element may be covered by a second lens element included in the second lens group 412, and may receive light passing through the second lens element. The second sensing element may sense information corresponding to a second zoom factor (e.g., a zoom factor of 3). The third lens group 413 may transmit light within a third FOV 453 (e.g., 10 degrees) to a third sensing element. The third sensing element may be covered by a third lens element included in the third lens group 413, and may receive light passing through the third lens element. The third sensing element may sense information corresponding to a third zoom factor (e.g., a zoom factor of 9).

[0107] For example, the image sensor may be implemented as a single sensor, or may be implemented as a plurality of sensors that all include the same sensing element. In this example, the image sensor may be designed so that each lens group may support the same resolution. For example, the sensing area covered by any one lens group may include the same number of sensing elements as the number of sensing elements covered by another lens group. In this example, the number R of lens elements shown along one axis of the lens array 410 may satisfy Equations 7, 8, and 9 shown below.

[0108] [Equation 7]

[0109]

[0110] [Equation 8]

[0111] L M =α i ×L i sti=1,…,M-1

[0112] [Equation 9]

[0113]

[0114] In Equations 7 to 9, R represents a resolution corresponding to one axis of the lens array shared by all lens groups. M L represents the number of lens elements corresponding to the Mth lens group along one axis of the lens array. i represents the number of lens elements corresponding to the i-th lens group along one axis of the lens array. and α i Each represents an arbitrary natural number. The number of lens elements included in each of the plurality of lens groups may be determined based on the resolution shared by the plurality of lens elements. M Substituting the expression of into the expression of R in Equation 7 to derive Equation 9, each lens group may have a resolution shared by all lens groups. In addition, each lens group may include a number of lens elements that is coprime to the number of sensing elements covered by each lens element.

[0115] Therefore, the image sensor may apply rearrangement and restoration to an image captured by the lens array 410 and the sensing array 420 satisfying the coprime condition and Equation 9 to restore a high-resolution scene image having colors more similar to the original scene.

[0116] For example, the image sensor may rearrange an image acquired by a camera using lens elements arranged in a Bayer pattern and a color sensor (e.g., a sensor for sensing images of a red channel, a green channel, and a blue channel in a visible light band), wherein in the color sensor, the sensing elements are arranged in a Bayer pattern in the order of similar LFs as described above. The image sensor may obtain a uniform color pattern to provide color interpolation with higher performance.

[0117] Figure 5 is a diagram illustrating an arrangement of lens elements in a lens array according to example embodiments.

[0118] The lens elements included in the same lens group may be arranged adjacent to each other. Figure 5 , the first lens elements of the first lens group 511 having the first lens size may be arranged adjacent to each other. The lens elements included in each lens group in other lens groups (e.g., the second lens group 512, the third lens group 513, and the fourth lens group 514) may also be arranged adjacent to each other.

[0119] Figure 5 The top view of the lens array 510 is shown to include four lens groups (ie, M=4). 1 =2, the first lens group 511 may include four lens elements arranged in a 2×2 shape, as L 2 =4, the second lens group 512 may include 16 lens elements arranged in a 4×4 shape. Similarly, the third lens group 513 may include 64 lens elements, and the fourth lens group 514 may include 256 lens elements. In one embodiment, the size of the area covered by the first lens group 511, the second lens group 512, the third lens group 513, and the fourth lens group 514 may be equal to each other. Although for convenience Figure 5 , the image sensor is implemented as a single sensor, but example embodiments are not limited thereto. For example, the image sensor may be implemented as four separate image sensors (each image sensor including one of the above lens groups), or as different types of image sensors (e.g., sensors having separate sensing elements that are different in size).

[0120] For example, a processor of an image sensor may acquire a CEV image based on sensing information sensed by a plurality of sensing elements. In the present disclosure, a CEV image may refer to an image acquired by overlappingly capturing the same or similar scene in a manner similar to the operation of an insect's compound eye. For example, an image sensor may acquire a CEV image based on the intensity of light received from a plurality of sensing elements through a plurality of lenses arranged in an array.

[0121] The processor may generate a CEV image, wherein the CEV image includes 2×2 first group images obtained by sensing light received through the first lens group 511, 4×4 second group images obtained by sensing light received through the second lens group 512, 8×8 third group images obtained by sensing light received through the third lens group 513, and 16×16 fourth group images obtained by sensing light received through the fourth lens group 514. As described above with reference to Figure 3 As described above, the processor may rearrange the CEV image based on the LF information sensed by the plurality of sensing elements.

[0122] The following reference Figure 6 Describes the sensed and scene related and through Figure 5 The lens array 510 acquires sensing information.

[0123] Figure 6 It is shown that the above reference Figure 5 Illustration of the FOV of a scene sensed by the lens elements of the arrangement through the sensing element.

[0124] The lens array can receive light from outside the image sensor and can transmit the light to the sensing array. The lens elements in the lens array can have different FOVs based on the lens size. For example, referring to Figure 5 , the first lens element included in the first lens group 511 may have a first FOV, the second lens element included in the second lens group 512 may have a second FOV, the third lens element included in the third lens group 513 may have a third FOV, and the fourth lens element included in the fourth lens group 514 may have a fourth FOV.

[0125] The sensing elements included in the sensing area covered by each lens group may sense light corresponding to the FOV of the corresponding lens group. Figure 6 The sensing information sensed by the sensing element covered by the corresponding lens group is shown for each FOV of the lens group.

[0126] The first sensing element covered by the first lens element may receive light corresponding to the first FOV through the first lens element, and may sense first sensing information 611. The first sensing information 611 may be sensing information associated with the area 610 corresponding to the first FOV. The second sensing element may receive light corresponding to the second FOV through the second lens element, and may sense second sensing information 621. The second sensing information 621 may be sensing information associated with the area 620 corresponding to the second FOV. Figure 6As shown in , area 620 may be smaller than area 610. The third sensing element may receive light corresponding to the third FOV through the third lens element, and may sense third sensing information 631. The third sensing information 631 may be related to area 630 (in Figure 6 In addition, the fourth sensing element may receive light corresponding to the fourth FOV through the fourth lens element, and may sense fourth sensing information 641. The fourth sensing information 641 may be associated with the area 640 (in the area 620) corresponding to the fourth FOV. Figure 6 The sensing information associated with the area smaller than area 630 is shown in FIG.

[0127] like Figure 6 As shown in , since the sensing information is collected only by the first sensing element in the edge portion outside the region 620 of the region 610 corresponding to the entire FOV (e.g., the first FOV), the density of the collected sensing information may be less than the density of the collected sensing information in the central region (e.g., the region 640 corresponding to the fourth FOV). All the sensing information sensed by the first to fourth sensing elements can be collected in the central region, so the density of the sensing information can be higher than that of the edge portion. Therefore, the processor of the image sensor can restore the image based on the sensing information sensed by the plurality of sensing elements, so that the central region within the FOV corresponding to the lens array can have a higher resolution than the region adjacent to the central region.

[0128] In response to input of a zoom factor from a user, the image sensor may generate an enlarged scene image based on the input zoom factor. For example, the image sensor may select target sensing information corresponding to the zoom factor specified by the user from sensing information sensed by a plurality of sensing elements. The image sensor may select information corresponding to the FOV corresponding to the zoom factor from the sensing information as the target sensing information. The image sensor may restore the scene image using the selected target sensing information. For example, when the zoom factor is specified to Figure 6 When a zoom factor corresponding to the second FOV of the second lens group in the image sensor is input, the image sensor may restore the scene image based on the sensing information sensed in the area 620 corresponding to the second FOV. The operation of restoring the scene image may include an operation of rearranging pixels based on the geometric structure of the image sensor, or an operation of estimating an output image from the CEV image using an image restoration model.

[0129] When an enlarged image (e.g., an image whose FOV decreases as the zoom factor increases) is restored, the area corresponding to the FOV provided by each lens element for the enlarged image may overlap. Due to the above overlap of the area, the image sensor can restore an enlarged scene image with a higher resolution by using all information from all lens groups at the same time (e.g., using information from lens elements with various lens sizes).

[0130] exist Figure 6 In the embodiment of the present invention, it is assumed that light reflected or emitted from a point at an infinite focus is received. When light from a finite focus is received, the FOV range of each lens group of the image sensor may be changed.

[0131] Figures 7 to 10 2 is a diagram illustrating an example of an arrangement of lens elements according to example embodiments.

[0132] exist Figure 7 In the lens array 710, one of the plurality of lens groups may include a single lens element. For example, the first lens group 711 may include a single lens element. As described above, each of the other lens groups 712, 713, and 714 may include a plurality of lens elements. Figure 7 The scene image is restored by sensing information obtained by transmitting light through the lens array 710 of the structure, so that both the quality of the wide-angle image and the quality of the image corresponding to the zoom factor can be improved.

[0133] exist Figure 8 In the lens array 810 of FIG. 8 , the lens elements may be arranged based on the lens size. For example, one lens element among the plurality of lens elements may be arranged closer to the center position of the lens array 810 than another lens element having a lens size larger than the lens size of the one lens element. In addition, one lens element among the plurality of lens elements may be arranged farther from the center position of the lens array 810 than another lens element having a lens size smaller than the lens size of the one lens element.

[0134] Reference Figure 8 , a first lens element 811 included in a first lens group having a first lens size as a largest lens size among the four lens groups may be disposed on an edge of the lens array 810. A second lens element 812 having a second lens size smaller than the first lens size may be disposed closer to a center position than the first lens element 811. A third lens element 813 having a third lens size smaller than the second lens size may be disposed closer to a center position than the second lens element 812. A fourth lens element 814 having a fourth lens size smaller than the third lens size may be disposed closer to a center position than the third lens element 813.

[0135] Fig. 9 The lens array may include Figure 8 The lens array 810 is the same as or in the same manner as Figure 8 The lens elements of the lens array 810 are arranged in a similar manner. Fig. 9 , each of the plurality of lens elements may cover the same number of sensing elements. For example, each of the first lens elements 911 may cover 16 sensing elements arranged in a 4×4 shape. Similarly, each of the second lens element 912, the third lens element 913, and the fourth lens element 914 may also cover 16 sensing elements. In this example, the size of the sensing elements in the sensing array 920 may be designed so that each lens element may cover the same number of sensing elements. For example, the first size (e.g., pixel size) of the first sensing element covered by the first lens element 911 may be larger than the second size of the second sensing element covered by the second lens element 912. In addition, the third size of the third sensing element may be smaller than the second size, and the fourth size of the fourth sensing element may be smaller than the third size. In the present disclosure, for example, the size of the sensing element may be the pixel pitch of the sensing element. According to an example embodiment, a lens element providing a small FOV may cover the same number of sensing elements as the number of sensing elements covered by a lens element providing a large FOV. Therefore, the degree of reduction in resolution due to multiple lenses may be reduced, and the quality of the scene image to be restored may be improved.

[0136] Although for ease of description, Fig. 9 As shown in , the number of sensing elements covered by each lens element is set to an integer, but example embodiments are not limited thereto. For example, each lens element may cover a fractional number of sensing elements.

[0137] exist Fig.10 In the lens array 1010, each of the plurality of lens elements 1011 may be randomly arranged relative to the plurality of sensing elements on a plane corresponding to the lens array 1010. The number of lens elements 1011 may satisfy Equation 9 for each lens size, and the lens elements 1011 may be arranged on the plane of the lens array 1010. Based on a portion of the random arrangement of the lenses, the image sensor may uniformly restore a scene image with high resolution regardless of the zoom factor and wide angle supported by each lens size.

[0138] Fig.11 is a block diagram illustrating a structure of an image sensor according to example embodiments.

[0139] Reference Fig.11 , the image sensor 1100 includes a lens array 1111 , a sensing array 1120 , and a processor 1130 .

[0140] Reference Fig.11 , the lens array 1111 may include a plurality of lens elements. The plurality of lens elements may be arranged on a lens plane. All of the plurality of lens elements may have the same or similar focal length. The plurality of lens elements may be designed to have the same focal length, but due to manufacturing tolerances, the plurality of focal lengths may be slightly different from each other. For example, the difference between the focal lengths of the plurality of lens elements may be less than a threshold error. As described above, the lens size of at least one lens element of the plurality of lens elements may be different from the lens size of at least one other lens element. Each of the plurality of lens elements may refract light and form a focus at a point on the sensing array 1120 including the plurality of sensing elements.

[0141] The sensing array 1120 may include a plurality of sensing elements. The plurality of sensing elements may be disposed on a sensing plane parallel to the lens plane. The plurality of sensing elements may be disposed on a sensing plane spaced apart from the lens array 1111 by a distance equal to the focal length of the lens element. Each of the plurality of sensing elements may sense light passing through the lens array 1111. For example, each of the plurality of sensing elements may receive light passing through a lens element covering the corresponding sensing element.

[0142] The processor 1130 may restore a scene image based on the intensity of light sensed by the plurality of sensing elements. For example, the processor 1130 may acquire a CEV image based on sensing information sensed by the plurality of sensing elements. The processor 1130 may restore a scene image from the CEV image. The scene image may be an image output from the image sensor 1100 and may be restored to be the same as or similar to the original scene.

[0143] In one example, the processor 1130 may restore the scene image from the CEV image based on the geometric structure between the plurality of sensing elements and the plurality of lens elements. For example, the processor 1130 may rearrange the pixels of the captured image (e.g., the CEV image) to generate an output image, wherein, as described above with reference to Figure 3 As described above, pixels of sensing elements that sense similar LF information among the plurality of sensing elements are rearranged to be adjacent to each other.

[0144] In another example, the processor 1130 may restore the scene image from the CEV image based on an image restoration model that is fully trained before the CEV image is acquired. The image restoration model may be a model designed to output a scene image corresponding to an arbitrary CEV image, and may have a machine learning structure. For example, the image restoration model may have a neural network structure. The image restoration model may be trained to output a reference output image given as a ground truth in response to an input of an arbitrary reference CEV image. However, the image restoration model is not limited thereto.

[0145] Fig.12 and Fig.13is a diagram illustrating an example of a device in which an image sensor is implemented according to example embodiments.

[0146] Image sensors can be applied to various technical fields. The image sensor can be designed so that a lens array including a plurality of lenses can be separated from a sensor including a plurality of sensing elements by a relatively small distance equal to the focal length of the lens elements in the lens array. For example, the image sensor can be implemented as an ultra-thin camera with a large sensor for high-definition capture. In other words, a multi-lens array structure can be used to reduce the thickness of the image sensor. The image sensor can be implemented as an application processor (AP), a field programmable gate array (FPGA), or a chip, and can be used as an image signal processor for a camera.

[0147] In addition, the image sensor can use a lens array with an ultra-thin structure and lens elements with different lens sizes and the same focal length to obtain sensing information associated with multiple zoom factors. Therefore, the image sensor can restore high-resolution scene images for multiple zoom factors.

[0148] In one example, the image sensor may be implemented in a mobile terminal. The mobile terminal may be a movable terminal rather than a terminal fixed at any geographical location, and may include, for example, a portable device (e.g., a smart device such as a smart phone or a tablet computer), an artificial intelligence speaker, or a vehicle. Fig.12 and Fig.13 An example of a mobile terminal is shown in FIG. 4 , however, example embodiments are not limited thereto.

[0149] Reference Fig.12 , the image sensor 1210 may be applied to a front camera or a rear camera of a smart phone. The image sensor 1210 may have a structure combining a large full-frame sensor and a microlens array, and may be applied to a camera of a mobile phone. For example, Fig.12 As shown in , the image sensor 1210 may be included in a front camera or a rear camera of the smart device 1200. For example, a sensing array and a lens array of the image sensor 1210 may be implemented as a full-frame sensing array and a micro-lens array, respectively.

[0150] In another example, the image sensor may have a thin structure or a curved structure and may be implemented for a vehicle. Fig.13, the image sensor 1310 may be implemented as a curved front camera or rear camera of the vehicle 1300. However, example embodiments are not limited thereto, and the image sensor 1310 may be used for, for example, a digital single-lens reflex (DSLR) camera, a drone, a closed-circuit television (CCTV), a webcam camera, a panoramic camera, a camera for film or broadcasting, or a virtual reality / augmented reality (VR / AR) camera. In addition, the image sensor 1310 may be applicable to various fields, for example, a flexible / retractable camera, a compound eye camera, or a contact lens type camera.

[0151] In another example, the image sensor may also be applicable to multi-frame super-resolution image restoration for increasing the resolution of a video image based on information about a plurality of captured consecutive frames.

[0152] The example embodiments described herein may be implemented using hardware components, software components, or a combination thereof. For example, a processing device may be implemented using one or more general or special computers (such as a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor, or any other device capable of responding and executing instructions in a defined manner). The processing device may run an operating system (OS) and one or more software applications running on the OS. The processing device may also access, store, operate, process, and create data in response to the execution of the software. For simplicity, the processing device is described as a single processing device; however, those skilled in the art will appreciate that the processing device may include multiple processing elements and multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, different processing configurations (such as parallel processors) are feasible.

[0153] Software may include computer programs, code segments, instructions, or some combination thereof, to individually or collectively instruct or configure a processing device to operate as desired. Software and data may be permanently or temporarily contained in any type of machine, component, physical or virtual device, computer storage medium or device, or permanently or temporarily contained in a propagating signal wave capable of providing instructions or data to a processing device or capable of being interpreted by a processing device. Software may also be distributed on networked computer systems so that the software is stored and executed in a distributed manner. Software and data may be stored by one or more non-transitory computer-readable recording media.

[0154] The method according to the above-mentioned example embodiment may be recorded in a non-transitory computer-readable medium including program instructions for implementing various operations that can be performed by a computer. The medium may also include program instructions, data files, data structures, etc. alone, or may also include a combination of program instructions, data files, data structures, etc. The program instructions recorded on the medium may be program instructions specially designed and constructed for the purpose of the example embodiment, or they may be known and available to technicians in the field of computer software. Examples of non-transitory computer-readable media include: magnetic media (such as hard disks, floppy disks, and tapes); optical media (such as CD-ROM disks and DVDs); magneto-optical media (such as optical disks); hardware devices (such as read-only memory (ROM), random access memory (RAM), flash memory) and the like that are specially configured to store and execute program instructions. Examples of program instructions include both machine code (such as code generated by a compiler) and files containing high-level code that can be executed by a computer using an interpreter. The hardware device may be configured to be used as one or more software modules to perform the operations of the above-mentioned example embodiments, and vice versa.

[0155] Although the present disclosure includes example embodiments, it will be clear to those of ordinary skill in the art that various changes in form and detail may be made in these example embodiments without departing from the spirit and scope of the claims and their equivalents. The example embodiments described herein are considered to be illustrative only and not for limiting purposes. The description of features or aspects in each example will be considered to apply to similar features or aspects in other examples. If the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, appropriate results may be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as being included in the present disclosure.

Claims

1. An image sensor, comprising: A lens array comprising: a plurality of lens elements, wherein each of the plurality of lens elements has a same focal length; and a plurality of sensing elements spaced apart from the lens array by the focal length and configured to sense light passing through the lens array, wherein a lens size of at least one lens element among the plurality of lens elements is different from a lens size of another lens element among the plurality of lens elements, wherein a first lens element among the plurality of lens elements is arranged closer to a center position of the lens array than a second lens element among the plurality of lens elements, wherein the second lens element has a lens size greater than a lens size of the first lens element, wherein each of the plurality of lens elements covers the same number of sensing elements, Wherein, a first size of the first sensing element covered by the first lens element is smaller than a second size of the second sensing element covered by the second lens element.

2. The image sensor according to claim 1, wherein: The plurality of lens elements are classified into a plurality of lens groups based on lens size, wherein each lens group among the plurality of lens groups corresponds to a respective lens size, so that for each lens group among the plurality of lens groups, the lens elements included in the lens group have a corresponding lens size, The respective lens size corresponding to each lens group among the plurality of lens groups is different from the respective lens sizes of the remaining lens groups among the plurality of lens groups.

3. The image sensor according to claim 2, wherein: Each lens group among the plurality of lens groups is configured to cover the same number of sensing elements among the plurality of sensing elements, The number of lens elements included in each lens group among the plurality of lens groups is determined based on the number of covered sensing elements.

4. The image sensor according to claim 2, wherein: For each lens group among the plurality of lens groups, lens elements included in the lens group are arranged adjacent to each other.

5. The image sensor according to claim 2, wherein: A lens group among the plurality of lens groups includes a single lens element.

6. The image sensor according to claim 1, wherein: At least one lens element among the plurality of lens elements is arranged to cover a non-integer number of sensing elements.

7. The image sensor according to claim 1, further comprising: A processor is configured to restore an image based on sensing information sensed by the plurality of sensing elements so that a resolution of a central area within a field of view of the lens array is higher than a resolution of an area adjacent to the central area.

8. The image sensor according to claim 1, further comprising: The processor is configured to acquire a compound eye vision image based on the sensing information sensed by the plurality of sensing elements.

9. The image sensor according to claim 8, wherein: The processor is further configured to rearrange pixels included in the compound eye vision image based on the light field information sensed by the plurality of sensing elements.

10. The image sensor according to claim 8, wherein: The processor is also configured to restore a scene image from the compound eye vision image based on a geometric relationship between the plurality of sensing elements and the plurality of lens elements.

11. The image sensor according to claim 8, wherein: The processor is also configured to restore the scene image from the compound eye vision image based on an image restoration model that is fully trained before the compound eye vision image is acquired.

12. The image sensor according to claim 1, further comprising: a processor configured to: select target sensing information from among the sensing information sensed by the plurality of sensing elements, wherein the target sensing information corresponds to a scaling factor specified by a user, The processor is further configured to restore the scene image based on the selected target sensing information.

13. The image sensor according to claim 12, wherein: The processor is further configured to select information corresponding to a field of view corresponding to a specified zoom factor as the target sensing information.

14. The image sensor according to claim 1, wherein: Each lens element among the plurality of lens elements is configured to refract incident light and form a focus of the light exiting the lens element at a point on a sensing array including the plurality of sensing elements.

15. The image sensor according to claim 1, wherein: The plurality of lens elements are classified into a plurality of lens groups based on a field of view, wherein each lens group among the plurality of lens groups corresponds to a respective field of view, such that the respective field of view corresponding to each lens group among the plurality of lens groups is different from the respective fields of view of the remaining lens groups among the plurality of lens groups, Wherein, for each lens group among the plurality of lens groups, the sensing elements among the plurality of sensing elements included in the sensing area covered by the lens group are configured to: sense light corresponding to the field of view corresponding to the lens group.

16. The image sensor according to claim 1, wherein: The image sensor is included in a mobile terminal.

17. An image sensing method, comprising: sensing light passing through a plurality of lens elements by a plurality of sensing elements, wherein each of the plurality of lens elements has a same focal length; and restoring, by a processor, an image of a scene based on the intensity of light sensed by the plurality of sensing elements, wherein a lens size of at least one lens element among the plurality of lens elements is different from a lens size of another lens element among the plurality of lens elements, wherein a first lens element among the plurality of lens elements is arranged closer to a center position of the lens array than a second lens element among the plurality of lens elements, wherein the second lens element has a lens size greater than a lens size of the first lens element, wherein each of the plurality of lens elements covers the same number of sensing elements, Wherein, a first size of the first sensing element covered by the first lens element is smaller than a second size of the second sensing element covered by the second lens element.

18. A camera comprising: a lens array configured to: refract incident light reflected from an imaging subject; as well as A sensor configured to: sense light refracted by the lens array, The lens array includes a first lens having a first diameter and a second lens having a second diameter different from the first diameter, wherein the first lens is arranged closer to the center of the lens array than the second lens, wherein the second diameter is greater than the first diameter, wherein each of the first lens and the second lens covers the same number of sensing elements, The first size of the first sensing element covered by the first lens is smaller than the second size of the second sensing element covered by the second lens.

19. The camera of claim 18, wherein: The lens array includes a plurality of first lenses having a first diameter and a plurality of second lenses having a second diameter.

20. The camera of claim 19, wherein: A first area of ​​the sensor covered by the plurality of first lenses has the same size as a second area of ​​the sensor covered by the plurality of second lenses.

21. The camera of claim 19, wherein: The sensor includes a plurality of sensing elements; Wherein, each of the plurality of first lenses and the plurality of second lenses covers the same number of sensing elements among the plurality of sensing elements.

22. The camera of claim 21, wherein: The number of sensing elements covered by each of the plurality of first lenses and the plurality of second lenses is a non-integer.

23. The camera of claim 21, wherein: Each of the plurality of sensing elements is configured to sense a first light ray reflected from a first point of an imaging subject and a second light ray from a second point of the imaging subject.

24. The camera of claim 22, wherein: The plurality of sensing elements include a first sensing element configured to sense a first light refracted by a first lens having a first diameter and a second light refracted by another lens among the plurality of first lenses having the first diameter.

25. The camera of claim 19, wherein: The plurality of first lenses correspond to a first field of view, The plurality of second lenses correspond to a second field of view different from the first field of view.

Images