Image restoration method and image restoration apparatus

Abstract

An image restoration method and an image restoration device are disclosed. The image restoration device generates a plurality of warped images by warping each of a plurality of input images based on depths corresponding to a plurality of disparities, and generates an output image having a high resolution based on the plurality of input images and the plurality of warped images using an image restoration model.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2019-0136237, filed on October 30, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. TECHNICAL FIELD

[0002] One or more example embodiments provide methods and apparatuses related to techniques for restoring a multi-lens image. BACKGROUND

[0003] Due to development of optical technology and image processing technology, capturing devices are being utilized in a wide range of fields (e.g., multimedia content, security, and identification). For example, a capturing device can be installed in a mobile device, a camera, a vehicle, or a computer to capture an image, identify an object, or acquire data for controlling a device. The volume of a capturing device can be determined based on, for example, the size of a lens, the focal length of a lens, or the size of a sensor. In order to reduce the volume of a capturing device, a multi-lens including a small lens can be used. SUMMARY

[0004] One or more example embodiments can address at least the above problems and / or disadvantages and other disadvantages not described above. Also, the example embodiments do not require overcoming the above-described disadvantages, and the example embodiments can not overcome any of the above-described problems.

[0005] One or more example embodiments provide an image restoration method and apparatus.

[0006] According to an aspect of example embodiments, there is provided an image restoration method including: obtaining a plurality of pieces of input image information; generating a plurality of pieces of warped image information corresponding to a plurality of disparities from each piece of input image information among the plurality of pieces of input image information; and generating an output image based on the plurality of pieces of input image information and the plurality of pieces of warped image information using an image restoration model.

[0007] The obtaining of the plurality of pieces of input image information can include obtaining a plurality of input images captured using lenses located at different positions as the plurality of pieces of input image information.

[0008] The generating of the plurality of pieces of warped image information can include generating a warped image as a warped image information by warping each of the plurality of input images to a pixel coordinate system corresponding to a target image based on a depth corresponding to each of the plurality of disparities.

[0009] The generating of the warped image can include generating one of the plurality of warped images by warping all pixels in each of the plurality of input images to a pixel coordinate system corresponding to the target image based on a single depth corresponding to one of the plurality of disparities.

[0010] The depth corresponding to one of the plurality of disparities can be associated with a disparity set with respect to the target image for the input image and a gap between the sensing units that capture the target image and the input image.

[0011] The generating of the output image can include generating the output image by providing data obtained by concatenating the plurality of input images and the plurality of warped images as an input of an image restoration model.

[0012] The obtaining of the plurality of pieces of input image information can include extracting a plurality of input feature maps as the plurality of pieces of input image information from the plurality of input images using a feature extraction model.

[0013] The generating of the plurality of pieces of warped image information can include generating warped feature maps as the warped image information by warping each of the plurality of input feature maps to a pixel coordinate system corresponding to the target image based on a depth corresponding to each of the plurality of disparities.

[0014] The generating of the output image can include generating the output image by providing data obtained by concatenating the plurality of input feature maps and a plurality of warped feature maps as an input of an image restoration model.

[0015] The image restoration model can be a neural network including at least one convolution layer that applies a convolution filter to input data.

[0016] The plurality of disparities can be less than or equal to a maximum disparity and greater than or equal to a minimum disparity. The maximum disparity can be associated with a minimum capturing distance of the sensing units, a gap between the sensing units, and a focal length of the sensing units.

[0017] A limited number of disparities can be provided.

[0018] The generating of the output image can include generating the output image by skipping depth sensing until a target point corresponding to a single pixel.

[0019] The generating of the plurality of pieces of warped image information can include loading a pre-computed coordinate mapping function for a target sensing unit and a sensing unit that captures one of the plurality of input images; and generating the warped image information by applying the loaded coordinate mapping function to the input image.

[0020] A resolution of the output image can be higher than a resolution of each of the plurality of input images.

[0021] The acquiring of the plurality of input image information can include capturing, by an image sensor including a multi-lens array, a multi-lens image including a plurality of input images.

[0022] The acquiring of the plurality of input image information can include capturing, by each of a plurality of image sensors, an input image.

[0023] According to an aspect of another example embodiment, there is provided an image restoration apparatus including an image sensor configured to acquire a plurality of input image information, and a processor configured to generate a plurality of warped image information corresponding to a plurality of parallaxes from each of the plurality of input image information, and generate an output image based on the plurality of input image information and the plurality of warped image information using an image restoration model.

[0024] According to an aspect of another example embodiment, there is provided an image restoration apparatus including a lens array including a plurality of lenses, a sensing array including a plurality of sensing elements that sense light passing through the lens array and are classified based on a plurality of sensing regions respectively corresponding to the plurality of lenses, and a processor configured to generate a plurality of warped information corresponding to a plurality of parallaxes from each of a plurality of input information, and generate an output image based on the plurality of input information and the plurality of warped information using an image restoration model, wherein the plurality of input information is acquired from the plurality of sensing regions and slightly different from each other.

[0025] A resolution of the output image can be higher than a resolution corresponding to each of the plurality of input information.

[0026] The processor can be configured to generate the plurality of warped information by warping each of the plurality of input information to a pixel coordinate system corresponding to a target image based on a depth corresponding to each of the plurality of parallaxes.

[0027] The processor can be configured to generate one of the plurality of warped information by warping all pixels in each of the plurality of input information to a pixel coordinate system corresponding to a target image based on a single depth corresponding to one of the plurality of parallaxes.

[0028] The processor can be configured to generate the output image by providing data obtained by concatenating the plurality of input information and the plurality of warped information as an input of an image restoration model.

[0029] The processor can be configured to extract a plurality of input feature maps as the plurality of pieces of input information from the plurality of input images using a feature extraction model.

[0030] The processor can be configured to generate a warped feature map as warped information by warping each of the plurality of input feature maps to a pixel coordinate system corresponding to a target image based on a depth corresponding to each of the plurality of disparities.

[0031] The processor can be configured to generate an output image by providing data obtained by concatenating the plurality of input feature maps and the plurality of warped feature maps as an input of an image restoration model. BRIEF DESCRIPTION OF DRAWINGS

[0032] The above and / or other aspects will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A process of image restoration according to an example embodiment is illustrated;

[0034] Figure 2 is a flowchart illustrating an image restoration method according to an example embodiment;

[0035] Figure 3 is a diagram illustrating image restoration using an image restoration model according to an example embodiment;

[0036] Figure 4 Generation of a warped image to be input to an image restoration model according to an example embodiment is illustrated;

[0037] Figure 5 is a diagram illustrating matching between pixels of a warped image and pixels of a target image according to an example embodiment;

[0038] Figure 6 is a diagram illustrating generation of an output image through registration of a warped image according to an example embodiment;

[0039] Figure 7 is a diagram illustrating a camera calibration process according to an example embodiment;

[0040] Figure 8 is a diagram illustrating a structure of an image restoration model according to an example embodiment;

[0041] Figure 9 is a diagram illustrating an image restoration process using an image warping model and an image restoration model according to an example embodiment;

[0042] Figure 10 is a diagram illustrating a structure of an image warping model according to an example embodiment;

[0043] Figure 11 is a block diagram illustrating a configuration of an image restoration device according to an example embodiment; and

[0044] Figure 12 is a block diagram illustrating a computing device according to an example embodiment. DETAILED DESCRIPTION

[0045] Example embodiments will be described in detail with reference to the accompanying drawings. The same reference numbers are used throughout different drawings to refer to the same or like elements.

[0046] The following description of structures or functions is an example, and is used only to describe example embodiments, and the scope of example embodiments is not limited to the description provided in this specification. Various changes and modifications can be made to the example embodiments by those having ordinary skill in the art.

[0047] It should also be understood that when the terms "comprise" and / or "include" are used in this specification, it is intended to mean that there are the described features, integers, steps, operations, elements, components, and / or the combination thereof, but not to the exclusion of the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or the combination thereof. As used in the expression "at least one of A, B, and C," the phrase "at least one of" indicates that A, B, or C can be present, or any combination thereof, and that the number of A, B, and C is not limited to a particular number. For example, the expression "at least one of a, b, and c" should be understood to mean a, b, c, a and b, a and c, b and c, or a, b, and c.

[0048] Unless defined otherwise herein, all terms used herein, including technical or scientific terms or phrases, have the same meaning as commonly understood by one of ordinary skill in the art. Unless otherwise defined herein, terms defined in commonly used dictionaries are to be interpreted as having a meaning that matches the contextual meaning in the relevant art, and are not to be interpreted in an idealized or overly formal sense.

[0049] With regard to the reference numerals assigned to the elements in the drawings, it should be noted that the same element can be designated by the same reference numeral even though it is illustrated in different drawings. Wherever possible, the same reference numbers are used in the drawings and the accompanying description.

[0050] Figure 1 A process of image restoration according to an example embodiment is illustrated.

[0051] The quality of an image captured and recovered by the image sensor 110 can be determined based on the number of sensing elements included in the sensing array 112 and the amount of light incident on the sensing elements. For example, the resolution of an image can be determined by the number of sensing elements included in the sensing array 112, and the sensitivity of an image can be determined by the amount of light incident on the sensing elements. The amount of light incident on the sensing elements can be determined based on the size of the sensing elements. As the size of the sensing elements increases, the amount of light incident on the sensing elements and the dynamic range of the sensing array 112 can increase. Accordingly, the resolution of an image captured by the image sensor 110 can increase as the number of sensing elements included in the sensing array 112 increases. Further, as the size of the sensing elements increases, the image sensor 110 can advantageously operate to capture images with high sensitivity even in low illumination.

[0052] The volume of the image sensor 110 can be determined based on the focal length of the lens element 111. For example, the volume of the image sensor 110 can be determined based on the gap between the lens element 111 and the sensing array 112. In order to collect light refracted by the lens element 111, the lens element 111 and the sensing array 112 can need to be separated from each other by the focal length of the lens element 111.

[0053] The focal length of the lens element 111 can be determined based on the field of view (FOV) of the image sensor 110 and the size of the lens element 111. For example, when the FOV is fixed, the focal length can increase in proportion to the size of the lens element 111. Further, in order to capture an image within a predetermined FOV, the size of the lens element 111 can need to increase as the size of the sensing array 112 increases.

[0054] As described above, in order to increase the sensitivity of an image while maintaining the FOV and the resolution of the image, the volume of the image sensor 110 can increase. For example, in order to increase the sensitivity of an image while maintaining the resolution of the image, the size of each sensing element can need to increase while maintaining the number of sensing elements included in the sensing array 112. Accordingly, the size of the sensing array 112 can increase. In this example, in order to maintain the FOV, the size of the lens element 111 can increase as the size of the sensing array 112 increases, and the focal length of the lens element 111 can increase. Accordingly, the volume of the image sensor 110 can increase.

[0055] Referring to Figure 1The image sensor 110 includes a lens array and a sensing array 112. The lens array includes a plurality of lens elements, and the sensing array 112 includes a plurality of sensing elements. The lens elements can be arranged on a plane of the lens array, and the sensing elements can be arranged on a plane of the sensing array 112. The sensing elements of the sensing array 112 can be classified based on sensing regions corresponding to the lens elements, respectively. The plane of the lens array can be parallel to the plane of the sensing array 112, and can be separated from the plane of the sensing array 112 by a focal length of the lens elements 111 included in the lens array. The lens array can be referred to as a "micro multi-lens array (MMLA)" or a "multi-lens array."

[0056] For example, as the size of each lens element included in the lens array decreases, i.e., as the number of lenses included in the same area on the lens array increases, the focal length of the lens elements 111 and the thickness of the image sensor 110 can decrease. Accordingly, a thin camera can be implemented. In this example, the image sensor 110 can restore a high-resolution output image 190 by rearranging and combining low-resolution input images 120 captured by each lens element 111.

[0057] Individual lens elements 111 in the lens array can cover sensing regions 113 of the sensing array 112 corresponding to the lens size of the lens elements 111. The sensing regions 113 of the sensing array 112 covered by the lens elements 111 can be determined based on the lens size of the lens elements 111. The sensing regions 113 can represent regions on the sensing array 112 reached by light within a predetermined FOV by passing through the lens elements 111. The size of the sensing regions 113 can be represented as a distance or a diagonal length from a center point of the sensing regions 113 to an outermost point of the sensing regions 113, and the lens size can correspond to the diameter of the lens.

[0058] Each sensing element in the sensing array 112 can generate sensing information based on light passing through the lenses of the lens array. For example, the sensing elements can sense intensity values of light received by the lens elements 111. The image sensor 110 can determine intensity information corresponding to an original signal associated with a point included in the FOV of the image sensor 110 based on the sensing information output by the sensing array 112, and can restore a captured image based on the determined intensity information. For example, the sensing array 112 can be an image sensing module including a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS).

[0059] In addition, the sensing elements can include color filters to sense desired colors, and can generate color intensity values corresponding to predetermined colors as the sensing information. Each of the plurality of sensing elements included in the sensing array 112 can be positioned to sense a color different from a color of an adjacent sensing element spatially adjacent to each sensing element.

[0060] When diversity of sensing information is sufficiently ensured, and when a full rank relationship between the sensing information and the raw signal information corresponding to the points in the FOV of the image sensor 110 is formed, a captured image corresponding to the maximum resolution of the sensing array 112 can be obtained. The diversity of the sensing information can be ensured based on parameters of the image sensor 110, for example, the number of lens elements included in the lens array and the number of sensing elements included in the sensing array 112.

[0061] For example, the sensing region 113 covered by a single lens element 111 can include a non-integer number of sensing elements. In one example, the multi-lens array structure can be implemented as a fractional alignment structure. For example, when the lens elements included in the lens array have the same lens size, the number of lens elements included in the lens array and the number of sensing elements included in the sensing array 112 can be co-prime. A ratio P / L between the number of sensing elements P corresponding to one axis of the sensing array 112 and the number of lens elements L in the lens array can be determined as a real number. Each lens element can cover the same number of sensing elements as a pixel offset corresponding to P / L.

[0062] Based on the fractional alignment structure described above, in the image sensor 110, the optical center axis (OCA) of each lens element 111 can be slightly different from the optical center axis of the sensing array 112. For example, the lens element 111 can be arranged eccentrically with respect to the sensing element. Accordingly, the lens elements 111 of the lens array can receive different strips of light field (LF) information. The LF can be emitted from an arbitrary target point, and can represent a field indicating the intensity and direction of light reflected from an arbitrary point on an object. The LF information can represent information about a combination of a plurality of LFs. Since the direction of the chief ray of each lens element 111 is also changed, the sensing region 113 can receive different strips of LF information. Accordingly, a plurality of pieces of input information (for example, input image information) slightly different from each other can be acquired from a plurality of sensing regions. The image sensor 110 can optically acquire a larger amount of sensing information based on the plurality of pieces of input information.

[0063] The above-described image sensor 110 can be classified into a plurality of sensing units. Each of the plurality of sensing units can be distinguished in units of lenses included in the multi-lens array. For example, each sensing unit can include a lens and a sensing element of a sensing region 113 covered by the lens. In one example, the image sensor 110 can generate separate input images from sensing information acquired for each sensing region 113 corresponding to each lens. For example, each of the plurality of sensing units can individually acquire an input image. As described above, since the plurality of sensing units acquire different pieces of LF information, the input images captured by each sensing unit can represent slightly different scenes. For example, the image sensor 110 can include "N" lenses, and can be classified into "N" sensing units. Since the "N" sensing units individually capture input images, the image sensor 110 can acquire "N" input images. In this example, "N" can be an integer greater than or equal to "2". In Figure 1 In the middle, the multi-lens array can include "25" lenses (N = 5 x 5 = 25), and the image sensor 110 can capture "25" low-resolution input images 120. In another example, the multi-lens image can include "36" input images (N = 6 x 6 = 36). Although, as described above, the plurality of sensing units is included in a single image sensor 110, the sensing units are not limited thereto. For example, the sensing units can be independent image sensing modules (e.g., camera sensors). In this example, each sensing unit can be located at a position different from the position of another sensing unit.

[0064] In the following description, the image sensor 110 can generate a plurality of low-resolution input images 120 from a plurality of sensing information acquired as described above, and can restore a high-resolution output image 190 based on a target image 121 among the plurality of low-resolution input images 120. Although a center image among the plurality of low-resolution input images 120 is determined as the target image 121, the target image 121 is not limited thereto. For example, another input image can also be used as the target image 121. In addition, the image sensor 110 can use an image of a separate additional image sensor as the target image. For example, the additional image sensor can be a camera sensor capable of capturing an image having a relatively high resolution compared to the image sensor 110.

[0065] Figure 2 is a flowchart illustrating an image restoration method according to an example embodiment. Figure 3 is a diagram illustrating image restoration using an image restoration model according to an example embodiment.

[0066] Referring to Figure 2In operation 210, the image restoration device can acquire a plurality of pieces of input image information. The input image information can be, but is not limited to, input images themselves, and can include, for example, input feature maps extracted from the input images using a feature extraction model. Hereinafter, an example in which the input image information is input images will be described with reference to Figures 4 to 8 An example in which the input image information is input feature maps will be described below with reference to Figure 9 and Figure 10 An example in which the input image information is input feature maps will be described below with reference to

[0067] The image restoration device can capture a plurality of input images using an image sensor 310 of Figure 3 For example, in the image restoration device, the image sensor 310 including a multi-lens array can capture a multi-lens image 320 including a plurality of input images. In the multi-lens image 320, each input image can be captured by a separate sensing unit included in the image sensor 310. The first input image to the Nth input image can be separately captured by the first sensing unit C1 to the Nth sensing unit C N In another example, each of a plurality of image sensors 310 in the image restoration device can capture an input image. In this example, each sensing unit can be an independent image sensor 310.

[0068] In operation 220, the image restoration device can generate a plurality of pieces of warp (or deformation) information (e.g., warp (or deformation) image information 330) corresponding to a plurality of disparities from each of a plurality of pieces of input information (e.g., input image information). The disparity can represent a difference in position of the same target point between two images, and can be, for example, a difference between pixel coordinates. In one example, the disparity of a target image with respect to each input image can be set to an arbitrary value, and a virtual distance from the image sensor 310 to the target point can be determined based on the set disparity. The image restoration device can generate warp image information 330 based on the distance determined according to the set disparity. The warp image information 330 can be, but is not limited to, a warped (or deformed) image obtained by converting an input image to a pixel coordinate system of a target image, and can also be a warped (or deformed) feature map obtained by converting an input feature map extracted from an input image to a pixel coordinate system of a target sensing unit that captures a target image. Hereinafter, an example in which a virtual depth determined based on the above-described disparity and a warp (or deformation) based on the virtual depth will be described with reference to Figure 4 An example in which the input image information is input feature maps will be described below with reference to

[0069] For example, as shown in Figure 3 The image restoration device can generate a minimum disparity d minThe corresponding warped image information 330 to the maximum disparity d max The corresponding warped image information 330. When the minimum disparity d min is equal to 0, the warped image can be the input image itself. The camera calibration parameters 319 will be described below with reference to FIG. 3B. Figure 7 In one example, when the number of disparities is "D", the image restoration device can generate "D" pieces of warped image information 330 for each of "N" input image information, and thus a total of "NxD" pieces of warped image information 330 can be generated. In this example, "D" can be an integer greater than or equal to "1".

[0070] In operation 230, the image restoration device can generate an output image 390 based on the plurality of pieces of input image information and the plurality of pieces of warped image information 330 using an image restoration model 340. The image restoration model 340 can be a model trained to output the output image 390 in response to an input of the input image information. The image restoration model 340 can have, for example, a machine learning structure, and can be a neural network. The neural network can be used to restore an image based on image registration by mapping input data and output data in a non-linear relationship based on deep learning. Deep learning can be a machine learning technique for solving an image registration problem from a large data set. Through supervised or unsupervised learning by deep learning, input data and output data can be mapped to each other. The image restoration model 340 can include an input layer 341, a plurality of hidden layers 342, and an output layer 343. Data input through the input layer 341 can be propagated through the plurality of hidden layers 342, and can be output through the output layer 343. However, data can be directly input to the hidden layer 342, rather than the input layer 341 and the output layer 343, or can be directly output from the hidden layer 342. The neural network can be trained through, for example, backpropagation.

[0071] The image restoration model 340 described above can be implemented as a convolutional neural network (CNN). The CNN can denote a neural network including a convolution layer, and a hidden layer of the CNN can include a convolution layer. For example, the CNN can include a convolution layer having nodes connected by a kernel. The CNN can be a network pre-trained to output an output image having a high resolution in response to input of a plurality of pieces of input image information and a plurality of pieces of warped image information based on training data. For example, the output image can be an image in which pixels included in the input image and the warped image and matching the target image are registered, and the resolution of the output image can be higher than resolutions corresponding to the plurality of pieces of input information (e.g., the input image). The image restoration device can extract feature data by performing convolution filtering on data input to the convolution layer. The feature data can denote data obtained by abstracting features of an image, and can include, for example, a result value of a convolution operation based on a kernel of the convolution layer. The image restoration device can perform a convolution operation with respect to a pixel and a neighboring pixel at an arbitrary location in an image based on an element value of the kernel. The image restoration device can calculate a convolution operation value for each pixel of the image by sweeping the kernel with respect to the pixel. Hereinafter, an example in which the image restoration model 340 is implemented as a CNN will be described in more detail with reference to FIGS. 4A and 4B. Figure 8 An example in which the image restoration model 340 is implemented as a CNN will be further described.

[0072] For example, the image restoration device can provide the "N" pieces of input image information acquired in operation 210 and the "NxD" pieces of warped image information 330 generated in operation 220 to the image restoration model 340. As described above, the image restoration model 340 can include a convolution layer that applies convolution filtering to input data. Accordingly, the image restoration device can generate an output image 390 having a high resolution by applying convolution filtering to the "N" pieces of input image information and the "NxD" pieces of warped image information 330 using the image restoration model 340.

[0073] Figure 4 Generation of a warped image to be input to an image restoration model according to an example embodiment is illustrated.

[0074] The image restoration device can generate a plurality of pieces of warped information (e.g., warped images) by warping (or deforming) each piece of input information in the plurality of pieces of input information (e.g., input images) to a pixel coordinate system corresponding to the target image 430 based on a depth corresponding to each of a plurality of disparities. For example, Figure 4 A warped image obtained by warping (or deforming) an i-th input image 420 among the "N" input images to a pixel coordinate system corresponding to the target image 430 is illustrated.

[0075] In the following description, a world coordinate system can represent a coordinate system based on an arbitrary point on the world as a three-dimensional (3D) coordinate system. A camera coordinate system can represent a 3D coordinate system based on a camera, and a principal point of a sensing unit can be used as an origin, an optical axis direction of the sensing unit can be used as a z-axis, a vertical direction of the sensing unit can be used as a y-axis, and a horizontal direction of the sensing unit can be used as an x-axis. A pixel coordinate system can also be referred to as an "image coordinate system," and can represent two-dimensional (2D) coordinates of pixels in an image.

[0076] For example, a world coordinate of a target point 490 separated from an image sensor can be assumed as (X, Y, Z). An i-th sensing unit 411C i A sensed pixel coordinate can be assumed as (u, v). An i-th sensing unit 412C T A sensed pixel coordinate can be assumed as (u', v'). However, it can be difficult to accurately determine a distance to the target point 490 based on only a pixel value sensed by each sensing unit. An image restoration device can assume that an input image has an arbitrary disparity with respect to a target image 430, and can warp (or deform) the input image to a pixel coordinate system corresponding to the target image 430 based on a distance value corresponding to the disparity.

[0077] As shown in Equation 1 below, the image restoration device can calculate a normalized coordinate of the i-th input image 420 by normalizing pixel coordinates of individual pixels of the i-th input image 420 . .

[0078] [Equation 1]

[0079]

[0080]

[0081] In Equation 1, respectively, denote principal points of the i-th sensing unit 411C i with respect to x- and y-axes of the i-th sensing unit 411C i , and respectively, denote focal lengths on the x- and y-axes of the i-th sensing unit 411C i . As shown in Equation 1, the image restoration device can normalize individual pixels of the i-th input image 420 by dividing by the focal lengths using the principal point of the i-th sensing unit 411C i as an origin.

[0082] Further, as shown in Equation 2 below, the image restoration device can calculate a depth corresponding to the normalized coordinate To calculate the i-th sensing unit 411C i 3D camera coordinates

[0083] [Equation 2]

[0084]

[0085]

[0086]

[0087] As shown in Equation 2 above, the image restoration device can restore depth With each normalized coordinate Multiply to obtain Image restoration equipment can retrieve the depth value from the coordinates of a 3D camera. Set to depth Therefore, the image restoration device can be based on the i-th sensing unit 411C i The optical axis is used to calculate the 3D camera coordinates, and the i-th sensing unit 411C i The corresponding input image is captured based on the depth corresponding to each pixel.

[0088] As mentioned above, it may be difficult to accurately estimate the depth value of the target point 490 indicated by the pixels of the input image based solely on the pixel values ​​of the pixels. Therefore, the image restoration device can use the depth value corresponding to a portion of the parallax within a limited range to perform the coordinate transformation based on Equation 2. The range of parallax can be limited to [d]. min d max Furthermore, the depth value can also be limited to [Z]. min (∞). Furthermore, Z min This represents the minimum capture distance of the image sensor, and can be, for example, 10 centimeters (cm). For example, in... Figure 4 In this process, the image restoration device may assume that the i-th input image 420 has a disparity d of "1" relative to the target image 430 (i.e., d = 1), and may use a depth value (e.g., z1) corresponding to the disparity d of "1". In Equation 2, z1 may be used as the depth value. In this example, the image restoration device can set all pixels of the i-th input image 420 to have the same disparity relative to the target image 430, and can transform the coordinates of all pixels based on the same depth value (e.g., the depth value corresponding to "d=1"). Similarly, the image restoration device can assume that the i-th input image 420 has "2", "3", "4" and "d" relative to the target image 430. max The parallax d of “” can be used with “2”, “3”, “4” and “d”. maxThe image restoration device can individually acquire the 3D camera coordinates transformed using the depth value z2 corresponding to the parallax d with respect to "2", the 3D camera coordinates transformed using the depth value z3 corresponding to the parallax d with respect to "3", the 3D camera coordinates transformed using the depth value z4 corresponding to the parallax d with respect to "4", and the depth values ​​transformed using the depth value z4 corresponding to the parallax d with respect to "2". max The depth value z corresponding to the parallax d is... min Transformed 3D camera coordinate values. Although the above... Figure 4 Integer disparity has been described, but the example implementation is not limited thereto.

[0089] As shown in Equation 3 below, the image restoration device can transform the 3D camera coordinates of the i-th input image 420 based on parallax transformation using Equation 2 into the target sensing unit 412C. T 3D camera coordinates

[0090] [Equation 3]

[0091]

[0092] In equation 3, R T Indicates target sensing unit 412C T The rotation information of the world coordinate system, and T T Indicates target sensing unit 412C T Translation information of the world coordinate system. In addition, R i Indicates the i-th sensing unit 411C i The rotation information of the world coordinate system, and T i Indicates the i-th sensing unit 411C i Translation information of the world coordinate system. Rotation and translation information can be calibration information, which will be referred to below. Figure 7 As described in Equation 3, the image restoration device can be based on rotation information R. i and R T And translation information T i and T T To transform 3D camera coordinates To calculate based on target sensing unit 412C T 3D camera coordinates

[0093] As shown in Equation 4 below, the image restoration device can calculate the target sensing unit 412C based on the coordinates of each pixel in the input image. T To match 3D camera coordinates Normalize.

[0094] [Equation 4]

[0095]

[0096]

[0097] In Equation 4, the image restoration device can acquire normalized coordinates of the target sensing unit 412C T by dividing each of the 3D camera coordinates based on the target sensing unit 412C by the depth T

[0098] As shown in Equation 5 below, the image restoration device can calculate pixel coordinates of a pixel coordinate system corresponding to the target image 430 from the normalized coordinates of the target sensing unit 412C T

[0099] [Equation 5]

[0100]

[0101]

[0102] In Equation 5, respectively denote principal points of the x-axis and the y-axis of the target sensing unit 412C T with respect to the target sensing unit 412C T respectively denote focal lengths on the x-axis and the y-axis of the target sensing unit 412C T The following will describe Figure 7 and

[0103] Based on Equations 1 to 5 described above, the image restoration device can warp (or deform) the i-th input image 420 to the pixel coordinate system corresponding to the target image 430 by transforming the pixel coordinates i of the i-th sensing unit 411C to the pixel coordinates of the target sensing unit 412C T A series of operations based on Equations 1 to 5 can be referred to as a "warping operation". For ease of description, the warping operation has been described in a time series, however, the example embodiments are not limited thereto. An operation (e.g., a uniform matrix operation) including a combination of operations based on Equations 1 to 5 can also be used.

[0104] ​​​​​​The image restoration device can generate a single warped image by warping (or deforming) all pixels of an input image for each of a plurality of input information (e.g., input images) to a pixel coordinate system corresponding to the target image 430 based on a single depth corresponding to one of the plurality of disparities. For example, based on a depth value z j All pixels of the j-th warped image generated from the i-th input image 420 can be based on the same depth value z j The warped (or deformed) pixels. In this example, j can be an integer greater than or equal to "1" and less than or equal to "d max ", but is not limited thereto, and can be a real number greater than or equal to "0" and less than or equal to "d max ". In addition, the maximum disparity d max may be determined as shown in the following Equation 6.

[0105] [Equation 6]

[0106]

[0107] In Equation 6, b denotes a gap between two sensing units, f denotes a focal length of the sensing unit, and z min denotes a minimum capturing distance of the sensing unit. For example, the plurality of disparities can be less than or equal to a maximum disparity d min determined based on the minimum capturing distance z max , the gap b, and the focal length f, and can be greater than or equal to a minimum disparity d min .

[0108] The depth corresponding to one of the plurality of disparities can be determined based on a disparity set with respect to the target image 430 for the input image and based on a gap b between a sensing unit C i that captures the input image and a sensing unit C T that captures the target image 430. For example, when the depth z object of all target points 490 appearing in the external scene is equal to z j , all pixels of the j-th warped image can be accurately aligned with respect to the target image 430. However, since real objects have various depths, a portion of the pixels in the input image can be aligned with the target image 430.

[0109] For example, as Figure 4As shown in FIG. 4B, the image restoration device can generate warped images corresponding to a plurality of disparities from the i-th input image 420. The plurality of warped images can include a first warped image 421 generated using a depth z1 corresponding to a disparity d of "1", a second warped image 422 generated using a depth z2 corresponding to a disparity d of "2", a third warped image 423 generated using a depth z3 corresponding to a disparity d of "3", a fourth warped image 424 generated using a depth z4 corresponding to a disparity d of "4", and a fifth warped image 425 generated using a depth z max min Although the partial input images and the partial warped images are shown in one dimension for convenience of description, example embodiments are not limited thereto. For example, each image can be a 2D image.

[0110] The target point 490 can be sensed at a target pixel 439 in the target image 430 and at an input pixel 429 in the i-th input image 420. When a disparity d between the i-th input image 420 and the target image 430 is set to "1", the image restoration device can generate the first warped image 421 by warping (or distorting) the i-th input image 420 such that a pixel in the i-th input image 420 separated from the target pixel 439 by the disparity d of "1" is aligned with the target pixel 439 in the target image 430. The second warped image 422 can be generated by warping (or distorting) the i-th input image 420 such that a pixel in the i-th input image 420 separated from the target pixel 439 by the disparity d of "2" is aligned with the target pixel 439. Similarly, each of the third warped image 423 to the fifth warped image 425 can be generated by warping (or distorting) the i-th input image 420 such that a pixel in the i-th input image 420 separated from the target pixel 439 by the disparity d set for each of the third warped image 423 to the fifth warped image 425 is aligned with the target pixel 439. As shown in FIG. 4B, the input pixel 429 can be aligned at a different position from the target pixel 439 in the first warped image 421, the second warped image 422, and the fifth warped image 425. In the third warped image 423 and the fourth warped image 424, the input pixel 429 can be aligned with the target pixel 439 with an error of one pixel or less. Alignment of pixels between warped images and target images is described below with reference to FIGS. 5A to 5D. Figure 4 Figure 5

[0111] Figure 5 is a diagram illustrating matching between pixels of a warped image and pixels of a target image according to an example embodiment.

[0112] ​​​An error between at least one pixel among pixels included in each of the plurality of images warped (or deformed) from the input image 520 based on a plurality of disparities and a target pixel included in the target image 530 can be less than or equal to one pixel. As a result, although accurate estimation of a depth of a target point is omitted, the image restoration device can generate a plurality of warped images based on depths corresponding to preset disparities to match at least one pixel included in at least one of the plurality of warped images with the target point. For example, in Figure 5 a first pixel 501 of a first warped image 521 warped (or deformed) from the input image 520 can be matched with a pixel 531 of the target image 530. Also, a second pixel 502 of a second warped image 522 can be matched with a pixel 532 of the target image 530.

[0113] Although an example in which any pixel of a warped image is matched with the target image 530 has been described with reference to Figure 5 an input image, an arbitrary region of an input image can include the same optical information as optical information of a region of a target image corresponding to the arbitrary region, and a predetermined region of a portion of an image warped (or deformed) from the input image can be matched with a region of the target image corresponding to the predetermined region.

[0114] Figure 6 is a diagram illustrating generation of an output image by registration of warped images according to an example embodiment.

[0115] The image restoration device can generate a first warped image 631 to a fifth warped image 635 from a plurality of input images 620. For example, the image restoration device can generate the first warped image 631 from a first input image 621 based on a depth value corresponding to an arbitrary disparity. The second warped image 632 can be an image warped (or deformed) from a second input image 622, the third warped image 633 can be an image warped (or deformed) from a third input image 623, the fourth warped image 634 can be an image warped (or deformed) from a fourth input image 624, and the fifth warped image 635 can be an image warped (or deformed) from a fifth input image 625. In each of the first input image 621 to the fifth input image 625, a first pixel 601 can be matched with a target image. The target image can be selected from the input images, however, example embodiments are not limited thereto. A second pixel 602 in the second warped image 632, a third pixel 603 in the third warped image 633, and a fourth pixel 604 in the fourth warped image 634 can each be matched with the target image. Other warped images can also include pixels matched with the target image, but further description of the pixels is omitted herein for simplicity of description.

[0116] The image restoration device can provide the plurality of input images 620 and the first warped image 631 to the fifth warped image 635 to an image restoration model 640. The image restoration model 640 can include a CNN including convolutional layers as described above, and can be trained to output an output image 690 having a high resolution in response to an input of input image information and warped image information. For example, the image restoration device can generate the output image 690 having a high resolution by registering pixels matching the target image in various image information using the image restoration model 640.

[0117] Figure 7 is a diagram illustrating a camera calibration process according to an example embodiment.

[0118] The image restoration device can pre-store information for generating warped image information.

[0119] For example, in operation 710, the image restoration device can perform camera calibration. A plurality of sensing units included in an image sensor can be designed in a state 701 in which all of the sensing units are aligned, however, the sensing units of an actually manufactured image sensor can be in a state 702 in which the sensing units are not aligned. The image restoration device can perform camera calibration using a check plate. The image restoration device can calculate, through camera calibration, principal points of the sensing units with respect to x-axes and y-axes of the sensing units and focal lengths as internal camera parameters. Further, the image restoration device can calculate, through camera calibration, rotation information R i and translation information T i of a world coordinate system of the sensing units as external parameters.

[0120] In operation 720, the image restoration device can generate and store depth information for each disparity. For example, the image restoration device can calculate a depth value corresponding to an arbitrary disparity between input images sensed by two sensing units based on an arrangement relationship between the sensing units (for example, an angle formed between optical axes or a gap between the sensing units). As described above, a limited number of disparities can be provided within a limited range. For example, the disparity can be an integer disparity, however, example embodiments are not limited thereto.

[0121] The image restoration device can pre-calculate a coordinate mapping function to be applied as a warping operation based on the internal camera parameters and the external parameters. The coordinate mapping function can represent a function of transforming a coordinate of each pixel in an input image to a pixel coordinate system corresponding to a target image based on a depth corresponding to an arbitrary disparity, the internal camera parameters, and the external parameters, and can include, for example, a function including a series of integrated operations according to Equations 1 to 5. The image restoration device can pre-calculate and store the coordinate mapping function for each disparity and for each sensing unit.

[0122] To generate the warped image information in operation 220 of Figure 2 To generate the warped image information in operation 220 of

[0123] However, the coordinate mapping function can not need to be pre-computed and stored. The image restoration device can store the internal camera parameters and the extrinsic parameters, instead of storing the pre-computed coordinate mapping function. The image restoration device can load the stored internal camera parameters and the stored extrinsic parameters, can compute the coordinate mapping function, and can generate the warped image information of the input image using the computed coordinate mapping function.

[0124] Figure 8 is a diagram illustrating a structure of an image restoration model according to an example embodiment.

[0125] The image restoration device can provide data obtained by concatenating a plurality of pieces of input information (e.g., input images) and a plurality of pieces of warped information (e.g., warped images) as an input of the image restoration model to generate an output image.

[0126] For example, the image restoration device can generate concatenated data 841 by concatenating the input image information 820 and the plurality of pieces of warped image information 829 generated from the input image information 820 as described above. For example, the image restoration device can concatenate "N" input images acquired from "N" sensing units with "D" warped images generated from the "N" input images. As described in Figure 8 As the input image information and the warped image information are concatenated, the concatenated data 841 can include "(D+1)×N" images, as shown in

[0127] The image restoration device can extract feature data from concatenated data 841 via convolutional layer 842. The image restoration device can perform a shuffle 843, making the pixel values ​​indicating the same points in multiple feature data sets closer to each other. The image restoration device can generate a high-resolution output image from the feature data via residual blocks 844 and 845. A residual block can represent a block of residual data between the output data input to the block and the feature data extracted from the input data. The output image has a resolution of "(A×H)×(A×W)," which can be higher than the resolution of "H×W" of each of the multiple input images.

[0128] For example, refer to Figure 5 and Figure 6 When the object is located at a distance from the image sensor [z min , z max When the distance is within the range of [D+1], each region of the target image may include information similar to that of the region at the same location in at least one of the (D+1)×N) reconstructed images included in the concatenated data 841 described above. Therefore, the image restoration device can provide the concatenated data 841 to the image restoration model 340, thus allowing the use of information including regions similar to the target image in each input image and distorted image, thereby enhancing the performance of image restoration. Although depth information of the target point indicated by individual pixels of the input image is not provided, the image restoration device can generate an output image with relatively high resolution. Furthermore, even if the input image is not aligned, the image restoration device can restore the image if the camera parameter information is known.

[0129] Although it has been referenced Figures 1 to 8 This section primarily describes examples of direct distortion (or deformation) of the input image, but the example embodiments are not limited to this. See below for further details. Figure 9 Examples describing the distortion (or deformation) of feature data extracted from an input image.

[0130] Figure 9 This is a diagram illustrating image restoration processing using an image distortion (or deformation) model and an image restoration model according to an example embodiment.

[0131] The image restoration device may use an image warping model 950 and an image restoration model 340. The image warping model 950 may include a feature extraction model 951 and a warping operation 952. The image warping model 950 may be a model trained to extract feature maps from each input image 920 and warp (or deform) the extracted feature maps. The parameters of the feature extraction model 951 (e.g., connection weights) can be changed through training; however, the warping operation 952 may include operations based on Equations 1 to 5 described above and may remain unchanged.

[0132] For example, the image restoration device can extract a plurality of input feature maps from a plurality of input images as a plurality of pieces of input image information using the feature extraction model 951. The feature extraction model 951 can include at least one convolution layer, and the input feature maps can be result values obtained by performing convolution filtering. The image restoration device can warp (or deform) each of the plurality of input feature maps to a pixel coordinate system corresponding to the target image based on a depth corresponding to each of the plurality of disparities to generate warped feature maps as warped image information. A feature map obtained by warping (or deforming) an input feature map to a pixel coordinate system of a target sensing unit based on a depth corresponding to a predetermined disparity can be referred to as a "warped feature map". The warping operation 952 applied to the input feature map is the same as the warping operation 952 applied to the input image 920 based on Equations 1 to 5 described above, and thus a further description thereof is not repeated here.

[0133] For example, when an input image captured in a Bayer pattern is directly warped (or deformed) to a pixel coordinate system of a target sensing unit, the Bayer pattern can be lost in the warped image. When color information is mixed by warping (or deforming), color information of each channel can be lost in the warped image. The image restoration device can extract an input feature map from an input image before color information is lost by the warping operation 952, and thus the color information is preserved in the input feature map. The image restoration device can calculate a warped feature map by applying the warping operation 952 to the input feature map extracted in a state in which the color information is preserved. Accordingly, the image restoration device can provide data obtained by concatenating the plurality of input feature maps and the plurality of warped feature maps as an input of the image restoration model, and can generate an output image 990 having a high resolution and preserved color information. As described above, the image restoration device can minimize loss of color information.

[0134] Hereinafter, reference will be made to Figure 10 An example of describing the structure of the image warping model 950.

[0135] Figure 10 FIG. 10 is a diagram illustrating a structure of an image warping model according to an example embodiment.

[0136] The image restoration device can generate input feature maps and warped feature maps from a plurality of input images using the image warping model 950. For example, the image restoration device can extract an input feature map from each of the plurality of input images using a feature extraction model. The feature extraction model can include at least one convolution layer 1051 as described above. In addition, the feature extraction model can include a residual block 1052. For example, in the case of a feature extraction model including a residual block 1052, the image restoration device can extract an input feature map from each of the plurality of input images using the residual block 1052. Figure 10In this case, the feature extraction model can include one convolutional layer and "M" residual blocks. In this example, "M" can be an integer greater than or equal to "1". The image restoration device can extract input feature maps as result values obtained by applying a convolutional filter to the individual input images 1020.

[0137] Further, the image restoration device can apply a warping operation to the extracted input feature maps. As described above, the image restoration device can warp (or deform) the input feature map corresponding to each sensing unit to the pixel coordinate system of the target sensing unit based on the depth corresponding to each of the plurality of disparities and based on the calibration information 1019 (e.g., internal camera parameters and external parameters) of the image sensor 1010. For example, the image restoration device can perform a warping operation for each input feature map based on the depths corresponding to the "D" number of disparities to generate "D" warped feature maps for one input feature map. The image restoration device can generate concatenated data 1053 obtained by concatenating the plurality of input feature maps and the warped feature maps. The concatenated data 1053 can include information associated with the "N" number of input feature maps and the "NxD" number of warped feature maps.

[0138] The image restoration device can provide the concatenated data 1053 as input to the image restoration model 340 to generate an output image 1090 having a high resolution (e.g., the resolution is increased to "A" times the resolution of the individual input images). For example, the image restoration model 340 can include one convolutional layer 1042 and a plurality of residual blocks 1044 and 1045. The residual block 1044 among the plurality of residual blocks 1044 and 1045 can receive the concatenated data 1053 as input, and can receive data to which shuffling 1043 is applied so that pixel values indicating the same point in the concatenated data 1053 are close to each other.

[0139] The image warping model 950 and the image restoration model 340 described above can be trained simultaneously or sequentially during training. Since the warping operation causing loss of color information is included in the image warping model 950, the image warping model 950 can learn parameters that minimize the color loss during training. The image warping model 950 and the image restoration model 340 can be trained by backpropagation. For example, the image warping model 950 and the image restoration model 340 can be trained to output training output (e.g., ground truth images having high resolution) having high resolution in response to input of training input (e.g., a plurality of low-resolution images) having low resolution. The image warping model 950 and the image restoration model 340 being trained can be referred to as a "temporary image warping model 950" and a "temporary image restoration model 340," respectively. The temporary image warping model 950 and the temporary image restoration model 340 can generate temporary output from arbitrary training input, and the parameters (e.g., connection weights between nodes) of the temporary image warping model 950 and the temporary image restoration model 340 can be adjusted so that the loss between the temporary output and the ground truth images can be minimized.

[0140] Figure 11 FIG. 1 is a block diagram illustrating a configuration of an image restoration device according to an example embodiment.

[0141] Referring to Figure 11 , the image restoration device 1100 can include an image sensor 1110, a processor 1120, and a memory 1130.

[0142] The image sensor 1110 can acquire a plurality of pieces of input image information. The image sensor 1110 can acquire a plurality of input images captured using lenses located at different positions as the plurality of pieces of input image information. For example, the image sensor 1110 can include a sensing unit configured to acquire each piece of input image information among the plurality of pieces of input image information. To acquire "N" pieces of input image information, the image sensor 1110 can include "N" sensing units. Although the "N" sensing units are included in a single image sensor (i.e., the image sensor 1110), example embodiments are not limited thereto. For example, each of the "N" image sensors 1110 can include a sensing unit.

[0143] The processor 1120 can generate a plurality of pieces of warped image information corresponding to a plurality of disparities from each piece of input image information among the plurality of pieces of input image information, and can generate an output image using an image restoration model based on the plurality of pieces of input image information and the plurality of pieces of warped image information. The processor 1120 can skip depth sensing until a target point corresponding to a single pixel, and can generate the output image.

[0144] However, the operation of the processor 1120 is not limited thereto, and the processor 1120 can simultaneously or sequentially perform the above-described operations with reference toFigures 1 to 10 at least one of the described operations.

[0145] The memory 1130 can temporarily or permanently store data for performing the image restoration method. For example, the memory 1130 can store input image information, warped image information, and an output image. Also, the memory 1130 can store an image warping model, a parameter of the image warping model, an image restoration model, and a parameter of the image restoration model. The parameters can be pre-trained.

[0146] Figure 12 is a block diagram illustrating a computing device according to an example embodiment.

[0147] Referring to Figure 12 , the computing device 1200 is a device configured to generate a high-resolution image using the above-described image restoration method. For example, the computing device 1200 can correspond to the image restoration device 1100 of Figure 11 . The computing device 1200 can include, for example, an image processing device, a smart phone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), or a head-mounted display (HMD). In one example, the computing device 1200 can also be implemented as a vision camera device for a vehicle, a drone, or a closed-circuit television (CCTV). In another example, the computing device 1200 can be implemented as a network camera device for a video call, a 360-degree virtual reality (VR) camera device, or a VR / augmented reality (AR) camera device.

[0148] Referring to Figure 12 , the computing device 1200 can include a processor 1210, a storage 1220, a camera 1230, an input device 1240, an output device 1250, and a network interface 1260. The processor 1210, the storage 1220, the camera 1230, the input device 1240, the output device 1250, and the network interface 1260 can communicate with each other through a communication bus 1270.

[0149] The processor 1210 can perform functions and execute instructions within the computing device 1200. For example, the processor 1210 can process instructions stored in the storage 1220. The processor 1210 can perform one or more operations described above with reference to Figures 1 to 11 .

[0150] The storage 1220 can store information or data for execution of the processor 1210. The storage 1220 can include a computer readable storage medium or a computer readable storage device. The storage 1220 can store instructions to be executed by the processor 1210, and information associated with execution of software or applications when the software or applications are being executed by the computing device 1200.

[0151] The camera 1230 can capture a plurality of input images. Also, although still images have been primarily described as images, example embodiments are not limited thereto. For example, the camera 1230 can capture an image including one or more image frames. For example, the camera 1230 can generate a frame image corresponding to each of a plurality of lenses. In this example, the computing device 1200 can generate a high-resolution output image for each frame from a plurality of input images corresponding to separate frames using the image warping model and the image restoration model described above.

[0152] The input device 1240 can receive input from a user through a tactile input, a video input, an audio input, or a touch input. For example, the input device 1240 can detect input from a keyboard, a mouse, a touch screen, a microphone, or a user, and can include other devices configured to transmit the detected input.

[0153] The output device 1250 can provide output of the computing device 1200 to a user through a visual channel, an audio channel, or a tactile channel. The output device 1250 can include, for example, a display, a touch screen, a speaker, a vibration generator, or other devices configured to provide output to a user. The network interface 1260 can communicate with external devices through a wired network or a wireless network. For example, the output device 1250 can provide a result obtained by processing data based on at least one of visual information, auditory information, and tactile information to a user. The computing device 1200 can visualize an output image generated with high resolution on a display.

[0154] Example embodiments described herein can be implemented using hardware components, software components, or combinations thereof. A processing device can be implemented using one or more general purpose computers or special purpose computers such as, for example, processors, controllers and arithmetic logic units, digital signal processors, microcomputers, field programmable arrays, programmable logic units, microprocessors, or any other devices capable of responding to and executing instructions in a defined manner. The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and create data in response to the execution of software. For simplicity, the processing device is described as a single processor; however, one of ordinary skill in the art will appreciate that the processing device can include a plurality of processing elements and a plurality of types of processing elements. For example, the processing device can include a plurality of processors or a processor and a controller. In addition, different processing configurations are possible, such as parallel processors.

[0155] Software can include programs, code, instructions, or some combination thereof, for carrying out an operation on the processing device described above. The software and data can be permanently or temporarily stored in any type of machine, component, physical or virtual equipment, computer storage medium or device, or in a propagation signal that can be read by a processing device. The software can also be distributed over network coupled computer systems so that the software is stored and executed in a distributed fashion. The software and data can be stored by one or more non-transitory computer-readable recording mediums.

[0156] The method according to the above-described example embodiments can be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by the computer. The media can also include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the media can be those specially designed and constructed for the purposes of the example embodiments, or they can be of the type well known and available to those having skill in the computer software arts. Examples of non-transitory computer-readable media include magnetic media, such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVDs; magneto-optical media, such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files containing a high-level code that can be executed by the computer using an interpreter. The described hardware devices can be configured to act as one or more software modules in order to perform the operations of the above-described example embodiments, or vice versa.

[0157] While the disclosure includes example embodiments, it will be clear to those having ordinary skill in the art that various changes can be made to the example embodiments without departing from the spirit and scope of the claims and their equivalents. The example embodiments described herein are to be considered merely exemplary, and are not for the purpose of limitation. Descriptions of features or aspects within each example should be considered to apply to similar features or aspects within other examples. If the described techniques are performed in a different order, and / or if the components within the described systems, architectures, devices, or circuits are combined in a different manner, or replaced or supplemented by other components or their equivalents, appropriate results can be achieved. Accordingly, the scope of the disclosure is not limited by the specific embodiments described herein, but only by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.

Claims

1. An image restoration method comprising: acquiring a plurality of pieces of input image information; generating, from each piece of the input image information, a plurality of pieces of warped image information corresponding to a plurality of parallaxes; and generating, using an image restoration model, an output image based on the plurality of pieces of input image information and the plurality of pieces of warped image information, wherein the generating the plurality of pieces of warped image information comprises generating the plurality of pieces of warped image information by warping each piece of the plurality of pieces of input image information to a pixel coordinate system corresponding to a target image based on a depth corresponding to each of the plurality of parallaxes, wherein the generating the output image comprises generating the output image by providing, as an input to the image restoration model, data obtained by concatenating the plurality of pieces of input image information and the plurality of pieces of warped image information, and wherein the image restoration model is a neural network including at least one convolutional layer that applies a convolutional filter to an input to the image restoration model. The acquiring the plurality of pieces of input image information comprises acquiring, as the plurality of pieces of input image information, a plurality of input images captured using lenses located at different positions.

2. The image restoration method according to claim 1, wherein The generating the plurality of pieces of warped image information comprises generating, as a warped image, a warped image by warping each of the plurality of input images to the pixel coordinate system corresponding to the target image based on a depth corresponding to each of the plurality of parallaxes.

3. The image restoration method according to claim 2, wherein The generating the warped image comprises generating one of a plurality of warped images by warping all pixels in each of the plurality of input images to the pixel coordinate system corresponding to the target image based on a single depth corresponding to one of the plurality of parallaxes.

4. The image restoration method of claim 3, wherein, The depth corresponding to one of the plurality of parallaxes is associated with a parallax set with respect to the target image and a gap between sensing units that capture the target image and the input image.

5. The image restoration method of claim 3, wherein, The acquiring the plurality of pieces of input image information comprises extracting, using a feature extraction model, a plurality of input feature maps from the plurality of input images as the plurality of pieces of input image information.

6. The image restoration method of claim 1, wherein The generating the plurality of pieces of warped image information comprises generating, as a warped feature map, a warped feature map by warping each of the plurality of input feature maps to the pixel coordinate system corresponding to the target image based on a depth corresponding to each of the plurality of parallaxes.

7. The image restoration method of claim 6, wherein, 8.The image restoration method according to any one of claims 1 to 7, wherein the plurality of parallaxes are smaller than or equal to a maximum parallax and greater than or equal to a minimum parallax, and the maximum parallax is associated with a minimum capturing distance of the sensing units, a gap between the sensing units, and a focal length of the sensing units. a limited number of parallaxes are provided.

9. The image restoration method of claim 8, wherein, The generating the output image comprises generating the output image by skipping depth sensing of a target point corresponding to a single pixel.

10. The image restoration method according to any one of claims 1 to 7, wherein The generating the plurality of pieces of warped image information comprises:

11. The image restoration method according to any one of claims 1 to 7, wherein loading a pre-computed coordinate mapping function for a target sensing unit and a sensing unit that captures one of the plurality of input images; and ​ The warped image information is generated by applying a loaded coordinate mapping function to the input image.

12. The image restoration method according to any one of claims 1 to 7, wherein The resolution of the output image is higher than the resolution of each of the plurality of input images.

13. The image restoration method of claim 1, wherein, The acquiring of the plurality of input image information includes capturing, by an image sensor including a multi-lens array, a multi-lens image including a plurality of input images.

14. The image restoration method of claim 1, wherein, The acquiring of the plurality of input image information includes capturing, by each of a plurality of image sensors, an input image. 15.A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image restoration method of any one of claims 1 to 14. 16.An image restoration device comprising: an image sensor configured to acquire a plurality of input image information; and a processor configured to generate, from each of the plurality of input image information, a plurality of warped image information corresponding to a plurality of disparities, and generate, using an image restoration model, an output image based on the plurality of input image information and the plurality of warped image information, wherein the processor is further configured to generate the plurality of warped image information by warping each of the plurality of input image information to a pixel coordinate system corresponding to a target image based on a depth corresponding to each of the plurality of disparities, wherein the processor is further configured to generate the output image by providing, as an input of the image restoration model, data obtained by concatenating the plurality of input image information and the plurality of warped image information, and wherein the image restoration model is a neural network including at least one convolution layer that applies a convolutional filter to an input of the image restoration model. 17.An image restoration device comprising: a lens array including a plurality of lenses; a sensing array including a plurality of sensing elements that sense light passing through the lens array, and the plurality of sensing elements are classified based on a plurality of sensing regions respectively corresponding to the plurality of lenses; and a processor configured to generate, from each of a plurality of input information, a plurality of warped information corresponding to a plurality of disparities, and generate, using an image restoration model, an output image based on the plurality of input information and the plurality of warped information, wherein the plurality of input information is acquired from the plurality of sensing regions and is different from each other, wherein the processor is further configured to generate the plurality of warped information by warping each of the plurality of input information to a pixel coordinate system corresponding to a target image based on a depth corresponding to each of the plurality of disparities, wherein the processor is further configured to generate the output image by providing, as an input of the image restoration model, data obtained by concatenating the plurality of input information and the plurality of warped information, and wherein the image restoration model is a neural network including at least one convolution layer that applies a convolutional filter to an input of the image restoration model.

18. The image restoration apparatus according to claim 17, wherein The resolution of the output image is higher than the resolution corresponding to each of the plurality of input information.

19. The image restoration apparatus according to claim 17, wherein The processor is configured to generate a piece of the plurality of pieces of warped information by warping all pixels in each piece of the plurality of pieces of input information to a pixel coordinate system corresponding to a target image based on a single depth corresponding to one of the plurality of parallaxes.

20. The image restoration apparatus according to any one of claims 17 to 19, wherein The processor is configured to extract a plurality of input feature maps as the plurality of pieces of input information from the plurality of input images using a feature extraction model.

21. The image restoration apparatus according to claim 20, wherein The processor is configured to generate a warped feature map as the warped information by warping each of the plurality of input feature maps to the pixel coordinate system corresponding to the target image based on a depth corresponding to each of the plurality of parallaxes.

Images