Image sensing device and image processing device

Abstract

The present application relates to an image sensing device and an image processing device. Disclosed is an image sensing device comprising: a first module adapted to generate a plurality of interpolated images separated for each color channel based on an original image and a plurality of first convolutional layers; a second module adapted to generate a plurality of refined images separated for each color channel based on the plurality of interpolated images and a plurality of second convolutional layers; and a third module adapted to generate at least one output image corresponding to the original image based on the plurality of refined images and a plurality of third convolutional layers.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate to semiconductor design technology, and more particularly, to image sensing devices. Background Art

[0002] Image sensors are devices that capture images using the properties of semiconductors that react to light. Image sensors are generally classified as charge-coupled device (CCD) image sensors and complementary metal oxide semiconductor (CMOS) image sensors. CMOS image sensors are increasingly common because they allow both analog and digital control circuits to be directly implemented on a single integrated circuit (IC). Summary of the Invention

[0003] Various embodiments of the present disclosure are directed to image sensing devices capable of end-to-end learning when generating output images from original images.

[0004] According to an embodiment, an image sensing device may include: a first module, which is adapted to generate a plurality of interpolated images separated for each color channel based on an original image and a plurality of first convolutional layers; a second module, which is adapted to generate a plurality of refined images separated for each color channel based on the plurality of interpolated images and a plurality of second convolutional layers; and a third module, which is adapted to generate at least one output image corresponding to the original image based on the plurality of refined images and a plurality of third convolutional layers.

[0005] The second module can learn weights included in each of the plurality of second convolutional layers to generate a plurality of refined images.

[0006] The third module can learn weights included in each of the plurality of third convolutional layers to generate an output image.

[0007] Each of the plurality of first convolutional layers may include weights each having a fixed value or weights in which fixed values ​​and variable values ​​are mixed.

[0008] According to an embodiment of the present disclosure, an image sensing device may include: a first module adapted to generate, based on an original image generated by a set color filter pattern, a plurality of interpolated images separated for each color channel while maintaining alignment in the set color filter pattern; a second module adapted to generate a plurality of refined images based on the plurality of interpolated images and a first learning algorithm; and a third module adapted to generate, based on the plurality of refined images and the second learning algorithm, at least one output image corresponding to the original image.

[0009] The first module may generate a plurality of interpolated images using a plurality of first convolutional layers.

[0010] Each of the plurality of first convolutional layers may include weights each having a fixed value or weights in which fixed values ​​and variable values ​​are mixed.

[0011] The second module may generate a plurality of refined images using a plurality of second convolutional layers, and learn weights included in each of the plurality of second convolutional layers.

[0012] The third module may use a plurality of third convolutional layers to generate an output image, and learn weights included in each of the plurality of third convolutional layers.

[0013] According to an embodiment of the present disclosure, an image sensing device may include: a first module, which is suitable for receiving an original image and generating a plurality of interpolated images based on the original image and a first convolutional layer, a second convolutional layer, and a third convolutional layer, the plurality of interpolated images including an interpolated image of a first subset associated with a first color channel and the first convolutional layer, an interpolated image of a second subset associated with a second color channel and the second convolutional layer, and an interpolated image of a third subset associated with a third color channel and the third convolutional layer; a second module, which is suitable for generating a plurality of refined images based on the plurality of interpolated images and a fourth convolutional layer, a fifth convolutional layer, and a sixth convolutional layer, the plurality of refined images including a refined image of a first subset associated with the first color channel and the fourth convolutional layer, a refined image of a second subset associated with the second color channel and the fifth convolutional layer, and a refined image of a third subset associated with the third color channel and the sixth convolutional layer; and a third module, which is suitable for correcting the plurality of refined images to generate an output image.

[0014] According to an embodiment of the present disclosure, an image sensing device may include: an image sensor, which includes a pixel array having a predetermined color filter pattern and is suitable for generating an original image; an image processor, which is suitable for generating an output image based on the original image and supports an end-to-end learning network using multiple convolutional layers when generating the output image based on the original image.

[0015] The learned network can be designed to maintain alignment in a predetermined color filter pattern when interpolating the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram illustrating an image sensing device according to an embodiment of the present disclosure.

[0017] Figure 2 It is an example Figure 1 Block diagram of the image sensor shown in .

[0018] Figure 3 It is an example Figure 2 An example of a pixel array is shown in FIG.

[0019] Figure 4It is an example Figure 1 Block diagram of the image processor shown in .

[0020] Figure 5 It is an example Figure 4 A block diagram of an example of the first module is shown in .

[0021] Figure 6 It is an example Figure 4 A block diagram of another example of the first module shown in .

[0022] Figure 7 It is an example Figure 4 Block diagram of the second module shown in .

[0023] Figures 8 to 13 This is an example of an embodiment according to the present disclosure. Figure 1 The operation of the image sensing device is shown in the figure. DETAILED DESCRIPTION

[0024] Various embodiments of the present disclosure are described below with reference to the accompanying drawings in order to describe the present disclosure in detail so that a person having ordinary skill in the art to which the present disclosure pertains can easily implement the technical spirit of the present disclosure.

[0025] It will be understood that throughout the specification, when an element is referred to as being “connected to” or “coupled to” another element, the element may be directly connected to or coupled to the other element, or electrically connected to or coupled to the other element with one or more elements interposed therebetween. In addition, it will be understood that the terms “comprises,” “includes,” and their derivatives used in this specification do not exclude the presence of one or more other elements, but may further include or have one or more other elements, unless otherwise specified. Throughout the description of the specification, some components are described in the singular, but the present disclosure is not limited thereto, and it will be understood that these components may be formed in the plural.

[0026] Figure 1 is a block diagram illustrating an image sensing device 10 according to an embodiment of the present disclosure.

[0027] Reference Figure 1 , the image sensing device 10 may include an image sensor 100 and an image processor 200 .

[0028] The image sensor 100 may generate a raw image IMG according to incident light.

[0029] The image processor 200 may generate an output image DIMG based on an original image IMG. When generating the output image DIMG based on the original image IMG, the image processor 200 may support an end-to-end learning network using multiple convolutional layers. The image processor 200 may generate an output image DIMG with improved clarity by improving artifacts included in the original image IMG and / or artifacts generated when processing the original image IMG through the end-to-end learning network.

[0030] Figure 2 It is an example Figure 1 A block diagram of an image sensor 100 is shown in FIG.

[0031] Reference Figure 2 , the image sensor 100 may include a pixel array 110 and a signal converter 120 .

[0032] The pixel array 110 may include a plurality of pixels arranged in row and column directions (see Figure 3 ). Pixel array 110 may generate analog-type image values ​​VPX for each row. For example, pixel array 110 may generate image values ​​VPX from pixels arranged in the first row during a first row time, and generate image values ​​VPX from pixels arranged in the nth row during an nth row time, where "n" is an integer greater than 2.

[0033] The signal converter 120 may convert the analog image value VPX into a digital image value DPX. The original image IMG may include the image value DPX. For example, the signal converter 120 may include an analog-to-digital converter.

[0034] Figure 3 It is an example Figure 2 FIG. 1 is a diagram of an example of a pixel array 110 shown in FIG.

[0035] Reference Figure 3 , the pixel array 110 can be arranged in a set color filter pattern. For example, the set color filter pattern can be a Bayer pattern. The Bayer pattern can be composed of repeating units each having 2×2 pixels. In each unit, two pixels G and G (hereinafter referred to as “green”) each having a green filter can be arranged to face each other diagonally at their corners, and a pixel B (hereinafter referred to as “blue”) having a blue filter and a pixel R (hereinafter referred to as “red”) having a red filter can be arranged at the other corners of each unit. The four pixels G, R, B, and G are not necessarily limited to Figure 3 The arrangement structure shown is not limited to the one shown, but can be arranged in various ways based on the above-mentioned Bayer pattern.

[0036] Although the present embodiment is described by taking an example in which the pixel array 110 has a Bayer pattern, the present disclosure is not limited thereto, and may have various patterns such as a quadrilateral pattern.

[0037] Figure 4 It is an example Figure 1 A block diagram of the image processor 200 is shown in FIG.

[0038] Reference Figure 4 , the image processor 200 may include a first module 210 , a second module 220 and a third module 230 .

[0039] The first module 210 can generate a plurality of interpolated images IIMG separated for each color channel based on the original image IMG. For example, the plurality of interpolated images IIMG may include a first interpolated image IIMG1 according to a green channel, a second interpolated image IIMG2 according to a red channel, and a third interpolated image IIMG3 according to a blue channel. The first module 210 can use a plurality of first convolutional layers when generating the plurality of interpolated images IIMG. For example, the first module 210 can generate a first interpolated image IIMG1 based on the original image IMG and a first convolutional layer CL1, a second interpolated image IIMG2 based on the original image IMG and a second convolutional layer CL2, and a third interpolated image IIMG3 based on the original image IMG and a third convolutional layer CL3. The first module 210 can interpolate the original image IMG for each color channel while maintaining alignment in the Bayer pattern and generate a plurality of interpolated images IIMG (see Figures 8 to 10 ).

[0040] The second module 220 can generate multiple refined images RIMG separated for each color channel based on the multiple interpolated images IIMG and the first learning algorithm. For example, the multiple refined images RIMG can include a first refined image RIMG1 based on the green channel, a second refined image RIMG2 based on the red channel, and a third refined image RIMG3 based on the blue channel. The second module 220 can use multiple second convolutional layers to generate the multiple refined images RIMG. For example, the second module 220 can generate the first refined image RIMG1 based on the first to third interpolated images IIMG1 to IIMG3 and a fourth convolutional layer CL4. Furthermore, the second module 220 can generate the second refined image RIMG2 based on the first to third interpolated images IIMG1 to IIMG3 and a fifth convolutional layer CL5. Furthermore, the second module 220 can generate the third refined image RIMG3 based on the first to third interpolated images IIMG1 to IIMG3 and a sixth convolutional layer CL6. The second module 220 can learn the weights included in each of the multiple second convolutional layers based on the first learning algorithm.

[0041] The third module 230 can generate at least one output image DIMG based on the multiple refined images RIMG and the second learning algorithm. The third module 230 can learn and correct defects, noise, etc. in the multiple refined images RIMG, and generate a high-quality output image DIMG. Since the third module 230 can use a well-known deep learning network as the second learning algorithm, a detailed description of the third module 230 will be omitted below. For reference, well-known deep learning networks can be extended to various fields according to the desired purpose and use ordinary images such as JPEG and BMP as input images, rather than images such as the refined images of the present disclosure.

[0042] Figure 5 It is an example Figure 4 . By way of example, the first module 210 generating the first to third interpolation images IIMG1 to IIMG3 will be described.

[0043] Reference Figure 5 The first module 210 may include a first storage module 211 , a first interpolation module 213 , a second interpolation module 215 and a third interpolation module 217 .

[0044] The first storage module 211 can store the first convolution layer CL1, the second convolution layer CL2, and the third convolution layer CL3. For example, the first convolution layer CL1 may include the first convolution kernel CK11, the second convolution kernel CK12, the third convolution kernel CK13, and the fourth convolution kernel CK14 corresponding to the color filter pattern of the pixel array 110 (see Figure 8 ). Each of the first to fourth convolution kernels CK11 to CK14 may include a weight for interpolating the green channel. All weights may each have a fixed value. The second convolution layer CL2 may include a first convolution kernel CK21, a second convolution kernel CK22, a third convolution kernel CK23, and a fourth convolution kernel CK24 corresponding to the color filter pattern of the pixel array 110 (see Figure 9 ). For example, each of the first to fourth convolution kernels CK21 to CK24 may include a weight for interpolating the red channel. All weights may each have a fixed value. The third convolution layer CL3 may include a first convolution kernel CK31, a second convolution kernel CK32, a third convolution kernel CK33, and a fourth convolution kernel CK34 corresponding to the color filter pattern of the pixel array 110 (see Figure 10 ). For example, each of the first to fourth convolution kernels CK31 to CK34 may include a weight for interpolating the blue channel. All weights may each have a fixed value.

[0045] The first interpolation module 213 may generate a first interpolated image IIMG1 of a green channel based on the original image IMG and the weights included in the first convolutional layer CL1 .

[0046] The second interpolation module 215 may generate a second interpolated image IIMG2 of a red channel based on the original image IMG and the weights included in the second convolutional layer CL2.

[0047] The third interpolation module 217 may generate a third interpolated image IIMG3 of a blue channel based on the original image IMG and the weights included in the third convolutional layer CL3.

[0048] Figure 6 It is an example Figure 4 . A block diagram of another example of the first module 210 is shown in FIG. By way of example, the first module 210 generating the first to third interpolation images IIMG1 to IIMG3 will be described.

[0049] Reference Figure 6 The first module 210 may include a calculation module 219 , a first storage module 211 , a first interpolation module 213 , a second interpolation module 215 and a third interpolation module 217 .

[0050] The calculation module 219 may calculate a variable value VV among weights included in each of the first to third convolutional layers CL1 to CL3 based on the original image IMG.

[0051] The first storage module 211 can store the first convolution layer CL1, the second convolution layer CL2 and the third convolution layer CL3. For example, the first convolution layer CL1 can include the first convolution kernel CK11 to the fourth convolution kernel CK14 corresponding to the color filter pattern of the pixel array 110 (see Figure 8 ). Each of the first to fourth convolution kernels CK11 to CK14 may include a weight for interpolating the green channel. The weight may have a fixed value and a variable value. The variable value may be some of the variable values ​​VV. The second convolution layer CL2 may include first to fourth convolution kernels CK21 to CK24 corresponding to the color filter pattern of the pixel array 110 (see Figure 9 ). For example, each of the first to fourth convolution kernels CK21 to CK24 may include a weight for interpolating the red channel. The weight may have a fixed value and a variable value. The variable value may be another variable value in the variable value VV. The third convolution layer CL3 may include the first to fourth convolution kernels CK31 to CK34 corresponding to the color filter pattern of the pixel array 110 (see Figure 10). For example, each of the first to fourth convolution kernels CK31 to CK34 may include a weight for interpolating the blue channel. The weight may have a fixed value and a variable value. The variable value may be another variable value in the variable value VV.

[0052] The first interpolation module 213 may generate a first interpolated image IIMG1 of a green channel based on the original image IMG and the weights included in the first convolutional layer CL1 .

[0053] The second interpolation module 215 may generate a second interpolated image IIMG2 of the red channel based on the original image IMG and the weights included in the second convolutional layer CL2.

[0054] The third interpolation module 217 may generate a third interpolated image IIMG3 of a blue channel based on the original image IMG and the weights included in the third convolutional layer CL3.

[0055] Figure 7 It is an example Figure 4 . By way of example, the second module 220 generating the first to third refined images RIMG1 to RIMG3 will be described.

[0056] Reference Figure 7 The second module 220 may include a second storage module 221 , a first learning module 223 , a second learning module 225 and a third learning module 227 .

[0057] The second storage module 221 can store the fourth convolution layer CL4, the fifth convolution layer CL5 and the sixth convolution layer CL6. For example, the fourth convolution layer CL4 can include a first convolution kernel CK41, a second convolution kernel CK42 and a third convolution kernel CK43. The first convolution kernel CK41 to the third convolution kernel CK43 are convolved with the first interpolation image IIMG1 to the third interpolation image IIMG3 respectively to generate the first refined image RIMG1 of the green channel (see Figure 11 ). Each of the first to third convolution kernels CK41 to CK43 may include a weight. The weight may be a variable value. The fifth convolution layer CL5 may include a first convolution kernel CK51, a second convolution kernel CK52, and a third convolution kernel CK53, which are convolved with the first to third interpolation images IIMG1 to IIMG3, respectively, to generate a second refined image RIMG2 of the red channel (see Figure 12). Each of the first to third convolution kernels CK51 to CK53 may include a weight. The weight may be a variable value. The sixth convolution layer CL6 may include a first convolution kernel CK61, a second convolution kernel CK62, and a third convolution kernel CK63, which are convolved with the first to third interpolation images IIMG1 to IIMG3, respectively, to generate a third refined image RIMG3 of the blue channel (see Figure 13 ). Each of the first to third convolution kernels CK61 to CK63 may include a weight. The weight may be a variable value.

[0058] The first learning module 223 may generate a first refined image RIMG1 based on the fourth convolutional layer CL4 and the first to third interpolated images IIMG1 to IIMG3. The first learning module 223 may change or update the weights included in the fourth convolutional layer CL4 through learning when generating the first refined image RIMG1. The first learning module 223 may repeat the convolution operation at least once when generating the first refined image RIMG1. When the first learning module 223 repeats the convolution operation "N" times, more than "N" fourth convolutional layers CL4 may be required.

[0059] The second learning module 225 may generate a second refined image RIMG2 based on the fifth convolutional layer CL5 and the first to third interpolated images IIMG1 to IIMG3. The second learning module 225 may change or update the weights included in the fifth convolutional layer CL5 through learning when generating the second refined image RIMG2. The second learning module 225 may repeat the convolution operation at least once when generating the second refined image RIMG2. When the second learning module 225 repeats the convolution operation "N" times, more than "N" fifth convolutional layers CL5 may be required.

[0060] The third learning module 227 may generate a third refined image RIMG3 based on the sixth convolutional layer CL6 and the first to third interpolated images IIMG1 to IIMG3. The third learning module 227 may change or update the weights included in the sixth convolutional layer CL6 through learning when generating the third refined image RIMG3. The third learning module 227 may repeat the convolution operation at least once when generating the third refined image RIMG3. When the third learning module 227 repeats the convolution operation "N" times, more than "N" sixth convolutional layers CL6 may be required.

[0061] Hereinafter, the operation of the image sensing device 10 having the above-described configuration according to the embodiment of the present disclosure is described.

[0062] When the image sensor 100 generates a raw image IMG according to incident light, the image processor 200 may generate an output image DIMG with improved clarity by improving artifacts included in the raw image IMG and / or artifacts generated when processing the raw image IMG through an end-to-end learning network.

[0063] Figures 8 to 10 is a diagram illustrating an operation of the first module 210 included in the image processor 200.

[0064] Reference Figure 8 , the first module 210 can generate a first interpolated image IIMG1 based on the original image IMG and the first convolution layer CL1. The first module 210 can selectively use the first to fourth convolution kernels CK11 to CK14 according to the positions of the image values ​​included in the original image IMG. For example, the first module 210 can use the first convolution kernel CK11 when interpolating image value 1 at the intersection of odd rows and odd columns among the image values ​​corresponding to the Bayer pattern. In addition, the first module 210 can use the second convolution kernel CK12 when interpolating image value 2 at the intersection of odd rows and even columns among the image values ​​corresponding to the Bayer pattern. In addition, the first module 210 can use the third convolution kernel CK13 when interpolating image value 3 at the intersection of even rows and odd columns among the image values ​​corresponding to the Bayer pattern. In addition, the first module 210 can use the fourth convolution kernel CK14 when interpolating image value 4 at the intersection of even rows and even columns among the image values ​​corresponding to the Bayer pattern. In other words, the first module 210 may interpolate the original image IMG according to the green channel while maintaining the green alignment of the Bayer pattern and generate the first interpolated image IIMG1 .

[0065] In an embodiment, each of the first to fourth convolution kernels CK11 to CK14 may include weights each having a fixed value. For example, in the first convolution kernel CK11, since the image value 1 at the intersection of odd rows and odd columns corresponds to green, the weight applied to the image value 1 may be "1", and the weight applied to the surrounding image values ​​may be "0". In the second convolution kernel CK12, since the image value 2 at the intersection of odd rows and even columns corresponds to red, the weight applied to the image value 2 may be "0", the weight applied to the surrounding image values ​​corresponding to green may be "1 / 4", and the weight applied to the surrounding image values ​​corresponding to blue may be "0". In the third convolution kernel CK13, since the image value 3 at the intersection of even rows and odd columns corresponds to blue, the weight applied to the image value 3 may be "0", the weight applied to the surrounding image values ​​corresponding to green may be "1 / 4", and the weight applied to the surrounding image values ​​corresponding to red may be "0". In the fourth convolution kernel CK14, since the image value 4 at the intersection of an even row and an even column corresponds to green, the weight applied to the image value 4 may be "1" and the weight applied to the surrounding image values ​​may be "0".

[0066] In an embodiment, each of the first to fourth convolution kernels CK11 to CK14 may include weights each having a fixed value or a mixture of fixed and variable values. For example, in the first convolution kernel CK11, since the image value 1 at the intersection of odd rows and odd columns corresponds to green, the weight applied to the image value 1 may be "1", and the weight applied to the surrounding image values ​​may be "0". In the second convolution kernel CK12, since the image value 2 at the intersection of odd rows and even columns corresponds to red, the weight applied to the image value 2 may be "RGAIN / 9", the weight applied to the surrounding image values ​​corresponding to green may be "1 / 9", and the weight applied to the surrounding image values ​​corresponding to blue may be "BGAIN / 9". "RGAIN" is calculated according to the following formula 1, and "BGAIN" is calculated according to the following formula 2.

[0067] [Formula 1]

[0068] RGAIN=TGV / TRV

[0069] Here, “TGV” represents the average value of all image values ​​corresponding to green among image values ​​included in the original image IMG, and “TRV” represents the average value of all image values ​​corresponding to red among image values ​​included in the original image IMG.

[0070] [Formula 2]

[0071] BGAIN=TGV / TBV

[0072] Here, “TBV” represents an average value of all image values ​​corresponding to blue among image values ​​included in the original image IMG.

[0073] Next, in the third convolution kernel CK13, since the image value 3 at the intersection of an even row and an odd column corresponds to blue, the weight applied to the image value 3 may be "BGAIN / 9," the weight applied to the surrounding image values ​​corresponding to green may be "1 / 9," and the weight applied to the surrounding image values ​​corresponding to red may be "RGAIN / 9." "RGAIN" is calculated according to the above formula 1, and "BGAIN" is calculated according to the above formula 2. In the fourth convolution kernel CK14, since the image value 4 at the intersection of an even row and an even column corresponds to green, the weight applied to the image value 4 may be "1," and the weight applied to the surrounding image values ​​may be "0."

[0074] Reference Figure 9 , the first module 210 can generate a second interpolated image IIMG2 based on the original image IMG and the second convolution layer CL2. The first module 210 can selectively use the first to fourth convolution kernels CK21 to CK24 based on the positions of image values ​​included in the original image IMG. For example, the first module 210 can use the first convolution kernel CK21 when interpolating image value 1 at the intersection of odd rows and odd columns among image values ​​corresponding to the Bayer pattern. Furthermore, the first module 210 can use the second convolution kernel CK22 when interpolating image value 2 at the intersection of odd rows and even columns among image values ​​corresponding to the Bayer pattern. Furthermore, the first module 210 can use the third convolution kernel CK23 when interpolating image value 3 at the intersection of even rows and odd columns among image values ​​corresponding to the Bayer pattern. Furthermore, the first module 210 can use the fourth convolution kernel CK24 when interpolating image value 4 at the intersection of even rows and even columns among image values ​​corresponding to the Bayer pattern. That is, the first module 210 may interpolate the original image IMG according to the red channel while maintaining the red alignment of the Bayer pattern and generate the second interpolated image IIMG2.

[0075] In an embodiment, each of the first to fourth convolution kernels CK21 to CK24 may include weights each having a fixed value. For example, in the first convolution kernel CK21, since the image value 1 at the intersection of odd rows and odd columns corresponds to green, the weight applied to the image value 1 may be "0", the weight applied to the surrounding image values ​​corresponding to red may be "1 / 2", and the weight applied to the surrounding image values ​​corresponding to blue and green may be "0". In the second convolution kernel CK22, since the image value 2 at the intersection of odd rows and even columns corresponds to red, the weight applied to the image value 2 may be "1", and the weight applied to the surrounding image values ​​may be "0". In the third convolution kernel CK23, since the image value 3 at the intersection of even rows and odd columns corresponds to blue, the weight applied to the image value 3 may be "0", the weight applied to the surrounding image values ​​corresponding to red may be "1 / 4", and the weight applied to the surrounding image values ​​corresponding to green may be "0". In the fourth convolution kernel CK24, since the image value 4 at the intersection of even rows and even columns corresponds to green, the weight applied to the image value 4 can be "0", the weight applied to the surrounding image values ​​corresponding to red can be "1 / 2", and the weight applied to the surrounding image values ​​corresponding to blue and green can be "0".

[0076] In an embodiment, each of the first to fourth convolution kernels CK21 to CK24 may include weights each having a fixed value or a mixture of fixed and variable values. For example, in the first convolution kernel CK21, since the image value 1 at the intersection of odd rows and odd columns corresponds to green, the weight applied to the image value 1 may be "1 / (9*RGAIN)", the weight applied to the peripheral image value corresponding to red may be "1 / 9", the weight applied to the peripheral image value corresponding to blue may be "BGAIN / (9*RGAIN)", and the weight applied to the peripheral image value corresponding to green may be "1 / (9*RGAIN)". In the second convolution kernel CK22, since the image value 2 at the intersection of odd rows and even columns corresponds to red, the weight applied to the image value 2 may be "1", and the weight applied to the peripheral image value may be "0". In the third convolution kernel CK23, since the image value 3 at the intersection of even rows and odd columns corresponds to blue, the weight applied to the image value 3 may be "BGAIN / (9*RGAIN)", the weight applied to the surrounding image values ​​corresponding to red may be "1 / 9", and the weight applied to the surrounding image values ​​corresponding to green may be "1 / (9*RGAIN)". In the fourth convolution kernel CK24, since the image value 4 at the intersection of even rows and even columns corresponds to green, the weight applied to the image value 4 may be "1 / (9*RGAIN)", the weight applied to the surrounding image values ​​corresponding to red may be "1 / 9", the weight applied to the surrounding image values ​​corresponding to blue may be "BGAIN / (9*RGAIN)", and the weight applied to the surrounding image values ​​corresponding to green may be "1 / (9*RGAIN)". "RGAIN" is calculated according to the above formula 1, and "BGAIN" is calculated according to the above formula 2.

[0077] Reference Figure 10, the first module 210 can generate a third interpolated image IIMG3 based on the original image IMG and the third convolutional layer CL3. The first module 210 can selectively use the first to fourth convolution kernels CK31 to CK34 based on the positions of the image values ​​included in the original image IMG. For example, the first module 210 can use the first convolution kernel CK31 when interpolating image value 1 at the intersection of odd rows and odd columns among the image values ​​corresponding to the Bayer pattern. Furthermore, the first module 210 can use the second convolution kernel CK32 when interpolating image value 2 at the intersection of odd rows and even columns among the image values ​​corresponding to the Bayer pattern. Furthermore, the first module 210 can use the third convolution kernel CK33 when interpolating image value 3 at the intersection of even rows and odd columns among the image values ​​corresponding to the Bayer pattern. Furthermore, the first module 210 can use the fourth convolution kernel CK34 when interpolating image value 4 at the intersection of even rows and even columns among the image values ​​corresponding to the Bayer pattern. That is, the first module 210 may interpolate the original image IMG according to the blue channel while maintaining the blue alignment of the Bayer pattern and generate the third interpolated image IIMG3.

[0078] In an embodiment, each of the first to fourth convolution kernels CK31 to CK34 may include weights each having a fixed value. For example, in the first convolution kernel CK31, since the image value 1 at the intersection of odd rows and odd columns corresponds to green, the weight applied to the image value 1 may be "0", the weight applied to the surrounding image values ​​corresponding to blue may be "1 / 2", and the weight applied to the surrounding image values ​​corresponding to red and green may be "0". In the second convolution kernel CK32, since the image value 2 at the intersection of odd rows and even columns corresponds to red, the weight applied to the image value 2 may be "0", the weight applied to the surrounding image values ​​corresponding to blue may be "1 / 4", and the weight applied to the surrounding image values ​​corresponding to green may be "0". In the third convolution kernel CK33, since the image value 3 at the intersection of even rows and odd columns corresponds to blue, the weight applied to the image value 3 may be "1", and the weight applied to the surrounding image values ​​may be "0". In the fourth convolution kernel CK34, since the image value 4 at the intersection of even rows and even columns corresponds to green, the weight applied to the image value 4 can be "0", the weight applied to the surrounding image values ​​corresponding to blue can be "1 / 2", and the weight applied to the surrounding image values ​​corresponding to red and green can be "0".

[0079] In an embodiment, each of the first to fourth convolution kernels CK31 to CK34 may include weights each having a fixed value or a mixture of fixed and variable values. For example, in the first convolution kernel CK31, since the image value 1 at the intersection of odd rows and odd columns corresponds to green, the weight applied to the image value 1 may be "1 / (9*BGAIN)", the weight applied to the peripheral image value corresponding to blue may be "1 / 9", the weight applied to the peripheral image value corresponding to red may be "RGAIN / (9*BGAIN)", and the weight applied to the peripheral image value corresponding to green may be "1 / (9*BGAIN)". In the second convolution kernel CK32, since the image value 2 at the intersection of odd rows and even columns corresponds to red, the weight applied to the image value 2 may be "RGAIN / (9*BGAIN)", the weight applied to the peripheral image value corresponding to blue may be "1 / 9", and the weight applied to the peripheral image value corresponding to green may be "1 / (9*BGAIN)". In the third convolution kernel CK33, since the image value 3 at the intersection of the even row and the odd column corresponds to blue, the weight applied to the image value 3 may be "1", and the weight applied to the surrounding image values ​​may be "0". In the fourth convolution kernel CK34, since the image value 4 at the intersection of the even row and the even column corresponds to green, the weight applied to the image value 4 may be "1 / (9*BGAIN)", the weight applied to the surrounding image values ​​corresponding to blue may be "1 / 9", the weight applied to the surrounding image values ​​corresponding to red may be "RGAIN / (9*BGAIN)", and the weight applied to the surrounding image values ​​corresponding to green may be "1 / (9*BGAIN)". "RGAIN" is calculated according to the above formula 1, and "BGAIN" is calculated according to the above formula 2.

[0080] Figures 11 to 13 is a diagram illustrating an operation of the second module 220 included in the image processor 200 according to an embodiment of the present disclosure.

[0081] Reference Figure 11, the second module 220 can generate a first refined image RIMG1 by convolving the first to third interpolated images IIMG1 to IIMG3 with the fourth convolution layer CL4. The fourth convolution layer CL4 may include first to third convolution kernels CK41 to CK43 assigned to each color channel. Each of the first to third convolution kernels CK41 to CK43 may include a weight. For example, the first convolution kernel CK41 may include first to ninth weights W41_1 to W41_9, the second convolution kernel CK42 may include first to ninth weights W42_1 to W42_9, and the third convolution kernel CK43 may include first to ninth weights W43_1 to W43_9. When representatively describing a target image value T1 among image values ​​included in the first refined image RIMG1, the target image value T1 may be calculated based on the following formula 3.

[0082] [Formula 3]

[0083]

[0084] When generating the first refined image RIMG1, the second module 220 may learn and change or update the weights included in the fourth convolutional layer CL4 through a first learning algorithm.

[0085] For example, the first learning algorithm may change the weights included in the fourth convolutional layer CL4 in a direction that reduces the difference between the first refined image RIMG1 and the first desired image. When the first learning algorithm calculates the gradient using the difference between the first refined image RIMG1 and the first desired image, backpropagates the gradient to the fourth convolutional layer CL4, and updates the weights included in the fourth convolutional layer CL4, a first refined image RIMG1 of a desired shape can be obtained from the original image IMG.

[0086] Reference Figure 12 , the second module 220 can generate a second refined image RIMG2 by convolving the first to third interpolated images IIMG1 to IIMG3 with the fifth convolution layer CL5. The fifth convolution layer CL5 may include first to third convolution kernels CK51 to CK53 assigned to each color channel. Each of the first to third convolution kernels CK51 to CK53 may include a weight. For example, the first convolution kernel CK51 may include first to ninth weights W51_1 to W51_9, the second convolution kernel CK52 may include first to ninth weights W52_1 to W52_9, and the third convolution kernel CK53 may include first to ninth weights W53_1 to W53_9. When representatively describing a target image value T2 among image values ​​included in the second refined image RIMG2, the target image value T2 may be calculated based on the following formula 4.

[0087] [Formula 4]

[0088]

[0089] When generating the second refined image RIMG2, the second module 220 may learn and change or update the weights included in the fifth convolutional layer CL5 through the first learning algorithm.

[0090] For example, the first learning algorithm may change the weights included in the fifth convolutional layer CL5 in a direction that reduces the difference between the second refined image RIMG2 and the second desired image. When the first learning algorithm calculates the gradient using the difference between the second refined image RIMG2 and the second desired image, backpropagates the gradient to the fifth convolutional layer CL5, and updates the weights included in the fifth convolutional layer CL5, a second refined image RIMG2 of a desired shape can be obtained from the original image IMG.

[0091] Reference Figure 13 , the second module 220 can generate a third refined image RIMG3 by convolving the first to third interpolated images IIMG1 to IIMG3 with the sixth convolution layer CL6. The sixth convolution layer CL6 may include first to third convolution kernels CK61 to CK63 assigned to each color channel. Each of the first to third convolution kernels CK61 to CK63 may include a weight. For example, the first convolution kernel CK61 may include first to ninth weights W61_1 to W61_9, the second convolution kernel CK62 may include first to ninth weights W62_1 to W62_9, and the third convolution kernel CK63 may include first to ninth weights W63_1 to W63_9. When representatively describing a target image value T3 among image values ​​included in the third refined image RIMG3, the target image value T3 may be calculated based on the following formula 5.

[0092] [Formula 5]

[0093]

[0094] When generating the third refined image RIMG3, the second module 220 may learn and change or update the weights included in the sixth convolutional layer CL6 through the first learning algorithm.

[0095] For example, the first learning algorithm may change the weights included in the sixth convolutional layer CL6 in a direction that reduces the difference between the third refined image RIMG3 and the third desired image. When the first learning algorithm calculates the gradient using the difference between the third refined image RIMG3 and the third desired image, backpropagates the gradient to the sixth convolutional layer CL6, and updates the weights included in the sixth convolutional layer CL6, a third refined image RIMG3 of a desired shape can be obtained from the original image IMG.

[0096] The third module 230 may generate the output image DIMG corresponding to the first to third refined images RIMG1 to RIMG3 using a second learning algorithm. For example, the third module 230 may use at least one convolution layer.

[0097] According to the aforementioned embodiments of the present disclosure, when the first to third modules operate based on their respective convolutional layers or convolution kernels to generate an output image from an original image, an end-to-end learning network can be implemented. When the first to third modules do not operate based on their respective convolutional layers, backpropagation may be difficult to implement, and thus an end-to-end learning network may not be implemented.

[0098] According to an embodiment of the present disclosure, when generating an output image from an original image, end-to-end learning can be achieved, thereby improving the performance of an image sensing device.

[0099] In addition, according to the embodiments of the present disclosure, the end-to-end learning network can be easily compatible with conventionally developed deep learning networks and thus used in various applications.

[0100] Although the present disclosure has been illustrated and described with reference to various embodiments, the disclosed embodiments are provided for illustration and not for limitation. Furthermore, it should be noted that, as will be appreciated by those skilled in the art in light of this disclosure, the present disclosure may be implemented in various ways by substitutions, changes, and modifications that fall within the scope of the appended claims.

[0101] CROSS-REFERENCE TO RELATED APPLICATIONS

[0102] This application claims priority to Korean Patent Application No. 10-2020-0172539, filed on December 10, 2020, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image sensing device, comprising: A first module that generates a plurality of interpolated images separated for each color channel based on the original image and a plurality of first convolutional layers; a second module that generates a plurality of refined images separated for each color channel based on the plurality of interpolated images and the plurality of second convolutional layers, and simultaneously changes or updates weights included in the plurality of second convolutional layers by a first learning algorithm; as well as A third module is configured to generate at least one output image corresponding to the original image based on the multiple refined images and multiple third convolutional layers.

2. The image sensing device according to claim 1, wherein The second module learns weights included in each of the plurality of second convolutional layers to generate the plurality of refined images.

3. The image sensing device according to claim 1, wherein The third module learns weights included in each of the plurality of third convolutional layers to generate the output image.

4. The image sensing device according to claim 1, wherein Each of the plurality of first convolutional layers includes weights each having a fixed value or weights in which fixed values ​​and variable values ​​are mixed.

5. The image sensing device according to claim 1, wherein The first module includes: A first storage module, the first storage module storing the plurality of first convolutional layers; and A plurality of interpolation modules are configured to generate the plurality of interpolation images based on weights included in corresponding first convolutional layers among the plurality of first convolutional layers.

6. The image sensing device according to claim 5, wherein: The first module further includes a calculation module that calculates a variable value among weights included in each of the plurality of first convolutional layers based on the original image.

7. The image sensing device according to claim 1, wherein The second module includes: a second storage module, the second storage module storing the plurality of second convolutional layers; and A plurality of learning modules are configured to generate the plurality of refined images based on the plurality of second convolutional layers and the plurality of interpolated images, and to learn a weight included in each of the plurality of second convolutional layers when generating the plurality of refined images.

8. The image sensing device according to claim 7, wherein: Each of the plurality of learning modules repeats a convolution operation at least once to generate the plurality of refined images.

9. The image sensing device according to claim 1, wherein: The first module generates the plurality of interpolated images while maintaining alignment in color filter patterns of the original images.

10. The image sensing device according to claim 1, wherein Each of the plurality of interpolation modules included in the first module generates a corresponding interpolation image based on the original image, and each of the plurality of learning modules included in the second module generates a corresponding refinement image based on the plurality of interpolation images.

11. An image sensing device, comprising: a first module that generates, based on an original image generated by a set color filter pattern, a plurality of interpolated images separated for each color channel while maintaining alignment in the set color filter pattern; a second module, which generates a plurality of refined images using a plurality of second convolutional layers based on the plurality of interpolated images and the first learning algorithm, and simultaneously changes or updates weights included in the plurality of second convolutional layers using the first learning algorithm; as well as A third module is configured to generate at least one output image corresponding to the original image based on the multiple refined images and a second learning algorithm.

12. The image sensing device according to claim 11, wherein The first module generates the plurality of interpolated images using a plurality of first convolutional layers.

13. The image sensing device according to claim 12, wherein: Each of the plurality of first convolutional layers includes weights each having a fixed value or weights in which fixed values ​​and variable values ​​are mixed.

14. The image sensing device according to claim 11, wherein The second module learns weights included in each of the plurality of second convolutional layers.

15. The image sensing device according to claim 11, wherein The third module generates the output image using a plurality of third convolutional layers and learns weights included in each of the plurality of third convolutional layers.

16. The image sensing device according to claim 11, wherein The first module includes: A first storage module, the first storage module storing a plurality of first convolutional layers; and A plurality of interpolation modules are configured to generate the plurality of interpolation images based on weights included in corresponding first convolutional layers in the plurality of first convolutional layers.

17. The image sensing device according to claim 16, wherein: The first module further includes a calculation module that calculates a variable value among weights included in each of the plurality of first convolutional layers based on the original image.

18. The image sensing device according to claim 11, wherein The second module includes: A second storage module, the second storage module storing a plurality of second convolutional layers; and A plurality of learning modules are configured to generate the plurality of refined images based on the plurality of second convolutional layers and the plurality of interpolated images, and to learn a weight included in each of the plurality of second convolutional layers when generating the plurality of refined images.

19. The image sensing device according to claim 18, wherein: When generating the plurality of refined images, the plurality of learning modules each repeat a convolution operation at least once.

20. The image sensing device according to claim 11, wherein Each of the plurality of interpolation modules included in the first module generates a corresponding interpolation image based on the original image, and each of the plurality of learning modules included in the second module generates a corresponding refinement image based on the plurality of interpolation images.

21. An image processing device, comprising: a first module that receives an original image and generates a plurality of interpolated images based on the original image and a first convolutional layer, a second convolutional layer, and a third convolutional layer, the plurality of interpolated images comprising an interpolated image of a first subset associated with a first color channel and the first convolutional layer, an interpolated image of a second subset associated with a second color channel and the second convolutional layer, and an interpolated image of a third subset associated with a third color channel and the third convolutional layer; a second module, which generates a plurality of refined images based on the plurality of interpolated images and the fourth, fifth, and sixth convolutional layers, and simultaneously changes or updates weights contained in the fourth, fifth, and sixth convolutional layers by using a first learning algorithm, wherein the plurality of refined images include a refined image of a first subset associated with the first color channel and the fourth convolutional layer, a refined image of a second subset associated with the second color channel and the fifth convolutional layer, and a refined image of a third subset associated with the third color channel and the sixth convolutional layer; as well as A third module corrects the plurality of refined images to generate an output image.

22. The image processing apparatus according to claim 21, wherein: Each of the plurality of interpolation modules included in the first module generates a corresponding interpolation image based on the original image, and each of the plurality of learning modules included in the second module generates a corresponding refinement image based on the plurality of interpolation images.

Images