Image sensor, image acquisition device including image sensor, and method of operating the same.

Abstract

To provide an image sensor, an image acquisition device including an image sensor, and an operation method thereof.SOLUTION: An image acquisition device includes a display layer containing hole areas through which external light passes and a pixel area arranged between the hole areas, an image sensor that is placed below the display layer and detects external light that has passed through the hole areas to generate a raw image, and a processor that performs image processing on the raw image based on blur information due to the arrangement of the hole areas.SELECTED DRAWING: Figure 7A

Description

TECHNICAL FIELD

[0001] The following embodiments relate to an image sensor, an image acquisition apparatus including the image sensor, and an operating method thereof.

BACKGROUND ART

[0002] Cameras, which are devices for capturing images, are widely mounted and used in various electronic devices. In mobile devices such as smartphones, cameras have become essential components, and are becoming higher in performance over time while also being reduced in size. In general, a smartphone includes a front camera and a rear camera, and the front camera is disposed in an upper region of the smartphone and is frequently used to take selfies.

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0003] An object of the present invention is to provide an image sensor, an image acquisition apparatus including the image sensor, and an operating method thereof.

MEANS FOR SOLVING THE PROBLEMS

[0004] An image acquisition apparatus according to an embodiment includes: a display layer including a pixel region disposed between a hole region through which external light passes and the hole region; an image sensor disposed below the display layer, the image sensor detecting external light passing through the hole region to generate a raw image; and a processor configured to perform image processing on the raw image based on blur information resulting from the arrangement of the hole region.

[0005] Each of the hole regions may be disposed between the pixel regions in the display layer, and may be arranged such that the minimum distance between the boundary of the hole region and the boundary of the pixel region is within 1 / 5 of the diameter of the hole region.

[0006] Each of the aforementioned hole regions has the shape of a single circle, and the hole regions may be identical in shape and size to one another.

[0007] Each of the aforementioned hole regions has a larger area than the aforementioned pixel region and can be distinguished from one another by a wiring region.

[0008] The processor can perform image preprocessing on the raw image to generate a preprocessed image, and then perform image restoration on the preprocessed image to generate an improved image.

[0009] The blur information may include information about the blur of the raw image, which is determined by at least one of the size, shape, depth, and placement of the hole region.

[0010] The blur information may include information about the blur of the raw image, which is determined by at least one of the morphology, size, depth, and spacing of the hole region.

[0011] An image acquisition device according to another embodiment includes a display layer including pixel regions and circular hole regions that transmit external light, and an image sensor positioned below the display layer to detect external light that has passed through the hole regions and generate an image, wherein each of the hole regions is positioned between pixel regions in the display layer and adjacent to the pixel regions, the hole regions and the pixel regions intersect each other in the display layer, and the hole regions are in the form of a single circle with the same shape and size.

[0012] An image sensor according to one embodiment includes a camera that detects external light passing through a hole region included in a display layer and generates a raw image, and a processor that performs image processing on the raw image based on blur information due to the arrangement of the hole regions.

[0013] An electronic device according to one embodiment includes a display panel including a display layer having a hole region through which external light passes and a pixel region for outputting a display image; an image sensor disposed below the display panel and detecting external light that has passed through the hole region to generate a raw image; a processor that generates an improved image by performing image processing on the raw image based on blur information due to the arrangement of the hole region; and storage for storing at least one of the raw image and the improved image.

[0014] An operating method of an image acquisition device according to one embodiment includes the steps of acquiring a raw image by detecting external light that has passed through a hole area located in the display layer using an image sensor located below the display layer, and performing image processing on the raw image based on blur information caused by the arrangement of the hole area. [Effects of the Invention]

[0015] According to one embodiment, it is possible to acquire high-brightness and high-quality images with an under-display camera. [Brief explanation of the drawing]

[0016] [Figure 1] This figure shows an electronic device to which an image acquisition device according to one embodiment is applied, and an enlarged view of the shooting area on the display screen. [Figure 2] This is a diagram illustrating the structure of an image acquisition device according to one embodiment. [Figure 3] This figure shows the arrangement of hole regions placed on a display layer according to one embodiment. [Figure 4] This figure shows the arrangement of hole regions placed on a display layer according to one embodiment. [Figure 5] This figure shows the point diffusion function based on the arrangement structure of the hall region according to one embodiment. [Figure 6]This is a diagram for explaining the change of the point spread function due to the arrangement structure of the hole region according to an embodiment. [Figure 7A] This is a diagram for explaining an image restoration process according to an embodiment. [Figure 7B] This is a diagram for explaining an image restoration process according to an embodiment. [Figure 7C] This is a diagram for explaining an image restoration process according to an embodiment. [Figure 8] This is a flowchart for explaining the operation of an image processing method according to an embodiment. [Figure 9] This is a diagram for explaining a process of training an image restoration model according to an embodiment. [Figure 10] This is a diagram showing the configuration of an image acquisition apparatus according to an embodiment. [Figure 11] This is a diagram showing the configuration of an image sensor according to an embodiment. [Figure 12] This is a diagram showing the configuration of an electronic device according to an embodiment. [Description of Embodiments]

[0017] The specific structural or functional descriptions for the embodiments are disclosed for illustrative purposes only, and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents or alternatives included in the technical idea.

[0018] Terms such as first or second may be used to describe a plurality of components, but such terms must be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be named a second component, and similarly, a second component may also be named a first component.

[0019] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the possibility of the presence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0020] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0021] The embodiments will be described below with reference to the attached drawings. Regardless of the reference numerals used in the drawings, the same components will be given the same reference numerals, and redundant explanations will be omitted. If it is determined that a specific explanation of the relevant prior art in the description of the embodiments would unnecessarily obscure the gist of the present invention, such detailed explanation will be omitted.

[0022] Figure 1 is a diagram showing an electronic device to which an image acquisition device according to one embodiment is applied, and an enlarged view of the shooting area on the display screen.

[0023] Referring to Figure 1, the image acquisition device is a device that captures images and acquires image data, and operates embedded in an electronic device 110 such as a smartphone, tablet computer, wearable device, netbook, laptop, PDA (personal digital assistant), or smart home device. The image acquisition device can operate with any electronic device 110, including a display 120, built in without any limitations.

[0024] In an image acquisition device, the camera that receives external light may be placed inside the electronic device 110 without being exposed to the outside. The camera of the image acquisition device is placed below the display 120 of the electronic device 110, and an image acquisition device having such a configuration is called an Under Display Camera (UDC). Because the camera is placed inside the electronic device 110, the area in which the camera is placed can also be realized as part of the display area. Therefore, it is not necessary to realize the display in a notch shape or to place an independent camera area within the display area, and a rectangular display can be fully realized.

[0025] Assuming that an image sensor is located inside a region 130 of the display 120, the display layer 140 corresponding to the region 130 may include a plurality of pixel regions 150 and a circular hole region 160. However, the shape of the hole region 160 is not limited to a circle; the hole region 160 may be realized in various shapes such as an ellipse or a square. In this specification, the hole region 160 is referred to as a micro-hole region. The arrangement pattern of the pixel regions 150 and the hole regions 160 may be repeated in the display layer 140 corresponding to the region 130. As shown in the figure, each hole region 160 may be located between and adjacent to the pixel regions 150. In one embodiment, the hole regions 160 may be arranged such that the minimum distance between the boundary of the hole region 160 and the boundary of the pixel region 150 is within 1 / 5 of the diameter of the hole region 160. Here, the value of 1 / 5 is merely an example, and it should not be interpreted as limiting the scope of the embodiment to that value.

[0026] The image acquisition device can acquire image data based on external light passing through the hole area 160 of the display layer 140. The image may be output through the pixel area 150, similar to the pixel areas included in other areas of the display 120. The display layer 140 is a component of the display panel, and is the layer on which the pixel area 150 is located. As shown in the figure, only one area 130 on which the image acquisition device is located has a hole area 160 that allows external light to pass into the interior of the electronic device 10. In the figure, the configuration of the display layer 140 on which the area 130 and the hole area 160 are located is shown as a circle, but this is just one example, and the area 130 and the display layer 140 can be realized in various configurations.

[0027] Figure 2 is a diagram illustrating the structure of an image acquisition device according to one embodiment.

[0028] Referring to Figure 2, an example of a cross-section of one region 130 of the electronic device 110 in Figure 1, viewed from the inside, is shown. The image acquisition device includes a display layer 210 and an image sensor 220. The display layer 210 includes a pixel region 230 capable of outputting color and a circular hole region 240 that transmits external light 250. The pixel region 230 and the hole region 240 may be arranged to intersect on the display layer 210. In the display layer 210, all areas except the hole region 240 are closed, and external light can reach the image sensor 220 by passing only through the hole region 240.

[0029] A protective layer 260 made of a transparent material is placed on the display layer 210 to protect the display layer 210. The protective layer 260 may be made of, for example, tempered glass or reinforced plastic. In addition, the display layer 210 may include other components for realizing a display panel other than the pixel area 230, and a display including the pixel area 230 can be realized through a display method such as LCD or OLED.

[0030] The image sensor 220 is positioned below the display layer 210 and can generate a raw image by detecting external light 250 that has passed through the hole region 240. The image sensor 220 can be designed to be ultra-compact, and multiple images may exist depending on the embodiment. The raw image is an image generated when the image sensor 220 detects the hole region 240, and may include, for example, a demosaiced RGB image. Since the light 250 reaching the image sensor 220 is a portion of the light incident on the display layer 210 that passes through the hole region 250, the raw image acquired by the image sensor 220 exhibits a lower level of image quality than the desired level. For example, the raw image has relatively low brightness and is noisy due to the obstruction of the pixel region 230. Also, because the hole region 240 acts like a slit, artifacts due to diffraction are shown in the raw image. For example, blur and flare phenomena may occur in the raw image.

[0031] When considering the image quality degradation factors described above, in a structure such as an under-display camera, image processing is necessary to improve the low image acquired by the image sensor 220. The image acquisition device may further include a processor (not shown) for performing such image processing. Depending on the embodiment, the image processing may be performed within the image sensor. The image processing may include a process to restore the low image acquired by the image sensor 220 to exhibit image quality similar to that of a typical image taken using a camera in a typical environment. The image acquisition device can efficiently perform image restoration through an image processing process that takes into account the arrangement of the hole areas 240 (e.g., shape, size, depth, spacing of the hole areas 240, etc.), and can provide high-quality, clear images even in an under-display camera environment.

[0032] On the other hand, the arrangement of the hole regions 240 included in the display layer 210 is determined according to the shape, size, depth, and spacing of each hole region 240, and can be optimized to minimize image degradation in low-resolution images or to be advantageous for image processing. The optimized arrangement is determined based on the blur characteristics of the arrangement of the hole regions 240. An example of such an arrangement of hole regions 240 is shown in Figures 3 and 4.

[0033] Figures 3 and 4 show a part of a display layer according to one embodiment, illustrating different arrangements of hole regions located within the display layer.

[0034] Referring to Figure 3, the entire area of ​​the display layer contains pixel areas 312, 314, 316, and 318, while the hole areas 320, 322, 324, 326, and 328 are located only in a portion of the display layer. The hole areas 320, 322, 324, 326, and 328 and the pixel areas 312, 314, 316, and 318 may intersect each other within the display layer.

[0035] The hole regions 320, 322, 324, 326, and 328 have the form of a single circle and may be identical in shape and size to one another. For example, a single-circle hole region 320 may be positioned between the pixel regions 312, 314, 316, and 318. Each of the hole regions 320, 322, 324, 326, and 328 may be positioned adjacent to four other hole regions and four pixel regions in the display layer. For example, hole region 320 may be positioned adjacent to hole regions 322, 324, 326, and 328 and pixel regions 312, 314, 316, and 318. Adjacent hole regions may be separated from each other by a wiring region for wiring, and the four hole regions may be positioned to surround one pixel region. The hole regions 320, 322, 324, 326, and 328 may be designed to have the largest size in the display layer excluding the pixel regions 312, 314, 316, and 318 and the wiring regions for wiring. The hole regions 320, 322, 324, 326, and 328 may be positioned adjacent to the pixel regions 312, 314, 316, and 318. For example, the hole region 320 may be positioned such that the minimum distance between the boundary of the hole region 320 and the boundary of each adjacent pixel region 312, 314, 316, and 318 is within 1 / 5 of the diameter of the hole region 160.

[0036] Referring to Figure 4, the arrangement of hole regions according to another embodiment is shown. Similar to the arrangement example shown in Figure 3, the hole regions 420, 422, 424, 426, 428 and the pixel regions 412, 414, 416, 418 may be arranged intersecting each other in the display layer. The hole regions 420, 422, 424, 426, 428 have the form of a single circle and may be identical in shape and size to each other. Each hole region 420, 422, 424, 426, 428 may be arranged adjacent to four adjacent hole regions and four pixel regions. Each hole region 420, 422, 424, 426, 428 is separated from each other by a wiring region and can be designed to have the largest size in the area excluding the pixel regions 412, 414, 416, 418 and the wiring region.

[0037] As illustrated with reference to Figures 3 and 4, by designing a hole region to have the maximum possible size, more external light can pass through, allowing for the acquisition of a high-brightness low-image. Furthermore, by arranging hole regions of the same size and shape adjacent to each other, image quality degradation due to diffraction can be reduced. The optimal arrangement structure for such hole regions can be derived through the following steps: determining the basic shape of the curved hole region while the pixel region that satisfies the given display requirements is arranged in the display layer; arranging the hole region between the furthest adjacent pixel regions from the current pixel region; increasing the size of the arranged hole region until it is within a certain distance from the pixel region or wiring region for wiring; and preventing all minute holes caused by the display structure other than the hole region.

[0038] The image acquisition device can generate higher-quality images by performing image preprocessing on the low image, taking into account the optical properties resulting from the arrangement structure of the hall area as determined above, and then performing image restoration. This image processing process will be explained in more detail below.

[0039] Figure 5 shows the point diffusion function based on the arrangement structure of the hole region according to one embodiment.

[0040] Figure 5 shows a region 505 of a display layer having a diameter of 510, and hole regions 520 and 525 and a pixel region 515 arranged within that region 505. This arrangement of hole regions 520 and 525 allows the blur morphology of the raw image detected via the image sensor to be simulated in a form similar to the point spread function (PSF) 540.

[0041] The point diffusion function 540 is a mathematical or numerical representation of how a single pixel region or point contained in the raw image spreads, and the blur information shown in the raw image can be estimated via the point diffusion function 540. The shape of the point diffusion function 540 can change depending on the size, shape, depth, spacing between them, and / or arrangement on a two-dimensional plane of the hole regions 520,525. For example, the size of the hole regions 520,525 determines the shape of the envelope of the point diffusion function 540 and the distance 550 to the main lobe, and the spacing 530 between adjacent hole regions 520,525 can determine the position and intensity of the first side-lobe. In addition, the ratio of the spacing to the size of each hole region 520,525 can determine the intensity of the first side-lobe.

[0042] The larger the size of the hole regions 520 and 525, the greater the distance 550 from the center 542 to the main lobe, and the smaller the intensity of the first side lobe. Conversely, the closer the spacing 530 between the hole regions 520 and 525, the greater the distance 545 from the center 542 to the first side lobe, and the smaller the intensity of the first side lobe. The relationship between the arrangement structure of the hole regions and the point diffusion function will be explained with reference to Figure 6.

[0043] Figure 6 illustrates the change in the point diffusion function due to the arrangement structure of the hole regions according to one embodiment. Referring to Figure 6, in case 610 and case 630, the diameter w corresponding to the size of the hole region is the same, but it is assumed that the spacing s1 between the hole regions in case 610 is larger than the spacing s2 between the hole regions in case 630. Otherwise, it is assumed that the remaining conditions are the same. The point diffusion functions 620 and 640 derived from simulations according to the arrangement of the hole regions in case 610 and case 630 are illustrated on the side. When comparing the point diffusion functions 620 and 640, it can be seen that the smaller the spacing between the hole regions, the greater the distance of the first side lobe at the centers 622 and 642, and the smaller the intensity of the first side lobe. Since the first side lobe is an element corresponding to blur and image afterimage in the low image, a smaller intensity of the first side lobe is advantageous.

[0044] Referring again to Figure 5, when considering the relationship between the arrangement of the hole regions 520 and 525 described above and the shape of the point diffusion function 540, it is preferable that the size of each hole region 520 and 525 is large and the spacing 530 between adjacent hole regions 520 and 525 is small. Furthermore, it is preferable that the hole regions 520 and 525 are arranged intersecting the pixel regions 515, adjacent to each pixel region 515, and have the shape of a single circle. When the arrangement of the hole regions 520 and 525 is determined, a margin for wiring or the like may be considered between adjacent hole regions 520 and 525.

[0045] Blur information shown in the raw image can be estimated from the point diffusion function 540 based on the arrangement structure of the Hall regions 520 and 525, and an image processing method based on the corresponding arrangement structure can be applied. For example, if the spacing 530 between the Hall regions 520 and 525 is large, a strong double image is shown. In this case, the image acquisition device can use the information from the point diffusion function 540 to remove the double image and blur from the raw image, and then perform an image reconstruction process. If the spacing 530 between the Hall regions 520 and 525 is small, strong noise or a widely spreading blur may occur. In this case, the image acquisition device can perform a deconvolution process or an image reconstruction process using a neural network model.

[0046] Figures 7A to 7C are diagrams illustrating the image restoration process according to one embodiment.

[0047] Referring to Figure 7A, below the display layer 710, where multiple pixel regions 720 and hole regions 715 are arranged, the image sensor 730 can detect external light passing through the multiple hole regions 715 and generate raw data. In step S735, blur information is extracted from the arrangement of the multiple hole regions 715 arranged in the display layer 710. The blur information may vary depending on the shape, size, depth and / or spacing of the hole regions 715, and may be derived from a point diffusion function determined through simulation or mathematical formulas.

[0048] In step S740, the image preprocessing performed on the raw data based on blur information includes image processing steps to reduce double images and blur artifacts contained in the raw data. The image preprocessing may include, for example, a filtering step that applies a filter to the raw image, a deconvolution step that uses a well-known kernel, or a step that obtains an image with reduced double images and blur via a neural network model.

[0049] In step S745, an image restoration process is performed on the pre-processed image that has undergone image preprocessing. The image restoration process is a step in which the resulting image is brought closer to the image acquired by the image sensor in an environment where there are no parts obstructed by multiple pixel regions 720. A neural network-based image restoration model trained for such image restoration can be used. The image restoration model can take the pre-processed image as input and output an improved image after the image restoration process has been performed.

[0050] To obtain favorable results from the above-described image preprocessing and image restoration processes, it is preferable that the size of each hole region 715 located on the display layer 710 is as large as possible, and that the distance between the hole regions 715 is as close as possible. The improved image after the image restoration process is completed is transmitted to the image signal processor (ISP) 750. The ISP 750 plays a role in processing the improved image in a favorable direction and can perform image processing such as noise reduction, white defect correction, RGB shading, RGB interpolation, color correction, and image format conversion.

[0051] According to another embodiment shown in Figure 7B, the image preprocessing and image restoration processes for the raw image described above are performed after the raw image has been processed by the ISP755. The image sensor 730 acquires a raw image, and the acquired raw image is transmitted to the ISP755. The ISP755 performs image processing on the raw image, such as noise reduction. Then, in step S760, the image preprocessing corresponding to step S740 is performed on the image processed by the ISP755, and in step S765, the image restoration process is performed.

[0052] According to a further embodiment shown in Figure 7C, the raw image preprocessing and image restoration processes described above may be performed between the first ISP 770 and the second ISP 785. The raw image acquired by the image sensor 730 is transmitted to the first ISP 770, which can perform image processing such as noise reduction. Subsequently, in step S775, image preprocessing is performed based on blur information, and in step S780, image restoration is performed. The resulting image from the image restoration process is transmitted to the second ISP 785, which can perform image processing on the resulting image, such as noise reduction, white defect correction, RGB shading, RGB interpolation, color correction, and image format conversion, as described above.

[0053] Figure 8 is a flowchart illustrating the operation of an image processing method according to one embodiment.

[0054] Referring to Figure 8, in step S810, the image acquisition device acquires a raw image using an image sensor. The image sensor is positioned below the display and can generate a raw image by detecting external light that has passed through the hole area of ​​the display layer.

[0055] In steps S820 and S830, the image acquisition device can generate an improved image by performing image processing on the raw image based on blur information resulting from the arrangement of the hole regions. The arrangement structure of the hole regions is determined by the shape, size, depth, and / or spacing of the hole regions. Specifically, in step S820, the image acquisition device can generate a pre-processed image by performing image preprocessing on the raw image based on the blur information. For example, the image acquisition device can generate a pre-processed image by performing preprocessing that applies a filter to the raw image and / or preprocessing that performs image deconvolution on the raw image.

[0056] In step S830, the image acquisition device performs an image restoration process on the preprocessed image to generate an improved image. For example, the image acquisition device can acquire the improved image using a neural network-based image restoration model that takes the preprocessed image as input. The image restoration model may be a model that has been trained to output an improved image as output data when a preprocessed image is input as input data. The learning process of such an image restoration model will be explained in more detail with reference to Figure 9.

[0057] Figure 9 is a diagram illustrating the process of training an image reconstruction model according to one embodiment.

[0058] Referring to Figure 9, a neural network-based image reconstruction model can be trained based on a general image and a preprocessed image. The general image can be generated by the image sensor 920 detecting external light that has passed through the display layer 910 where the main aperture 915 exists, rather than the fine hole regions. The general image thus generated is stored in the general image database (DB) 930.

[0059] In the display layer 940, where multiple hole regions 945 and pixel regions 950 are arranged alternately, the image sensor 960 detects external light passing through the hole regions 945 and can acquire a raw image. In the display layer 940, the arrangement of the hole regions 945 and pixel regions 950 is as described with reference to the above diagram. The image sensor 960 may be the same image sensor as the image sensor 920, and it is assumed that there are no differences between the image sensors 920 and 960 in acquiring images other than the difference between the display layers 910 and 940. In step S970, preprocessing is performed on the raw image acquired by the image sensor 960, and the preprocessed image generated by the preprocessing is stored in the preprocessed image database 980. The preprocessing of the image is performed based on blur information derived from the optical characteristics due to the arrangement of the hole regions 945, and includes a process of removing double images from the raw data using the blur information. The blur information includes information on the point diffusion function determined based on the arrangement of the hole regions 945.

[0060] In step S990, the image reconstruction model is trained based on the general image and preprocessed image acquired as described above. The preprocessed image is input to the image reconstruction model, and the image reconstruction process is performed. The resulting image is then compared to the general image. Based on the loss function that defines the difference between the resulting image and the general image, the parameters of the neural network constituting the image reconstruction model can be updated to reduce this difference. Through this process, the image reconstruction model can be trained to produce a resulting image that closely resembles the general image.

[0061] The process described above is repeated for various training data, allowing the image reconstruction model to be gradually updated in a favorable direction. In some embodiments, synthetic data in which noise or blur has been artificially added to a general image may be used to train the image reconstruction model. For example, the synthetic data can be input to the image reconstruction model, and the parameters of the image reconstruction model can be updated so that the difference between the resulting image output by the image reconstruction model and the original general image before synthesis decreases.

[0062] Figure 10 shows the configuration of an image acquisition device according to one embodiment.

[0063] Referring to Figure 10, the image acquisition device 1000 includes a display layer 1010, an image sensor 1020, and a processor 1030. The display layer 1010 may include pixel regions positioned between hole regions through which external light is transmitted. On the display layer 1010, the hole regions and pixel regions may intersect each other, and each hole region may have a larger area than the pixel region and have the shape of a single circle. The hole regions may be identical in shape and size, or they may be separated from each other by a wiring region.

[0064] The image sensor 1020 is positioned below the display layer 1010 and detects external light that has passed through the hole area of ​​the display layer 1010 to generate a raw image, which is then transmitted to the processor 1030. The image sensor 1020 may also include a camera that receives external light and generates image data.

[0065] The processor 1030 controls the overall operation of the image acquisition device 1000 and can execute instructions for performing the operations of the image acquisition device as described with reference to Figures 1 to 9. For example, the processor 1030 can generate an improved image by performing image processing on the raw image based on blur information due to the arrangement of hole regions. The blur information may include information about the blur of the raw image determined by the shape, size, depth and / or spacing of the hole regions present in the display layer 1010. The blur information may also include, for example, information about the point diffusion function due to the arrangement structure of the hole regions. The processor 1030 can generate a pre-processed image by performing image preprocessing on the raw image acquired by the image sensor 1020, and generate an improved image by performing image restoration processing on the pre-processed image. In relation to the preprocessing process, the processor 1030 can generate a pre-processed image by performing image preprocessing such as applying a filter to the raw image and / or performing image deconvolution on the raw image. The processor 1030 can acquire an improved image using a neural network-based image reconstruction model that takes a preprocessed image as input.

[0066] Figure 11 shows the configuration of an image sensor according to one embodiment.

[0067] According to one embodiment, the image sensor 1110 can not only acquire a raw image, but also perform image processing on the acquired raw image to generate an improved image. Through this operation, the image sensor 1110 can output an improved image.

[0068] Referring to Figure 11, the image sensor 1110 includes a camera 1120 and a processor 1130. The camera 1120 can detect external light passing through hole regions included in the display layer and generate a raw image. The display layer has intersecting hole regions and pixel regions, each hole region having a larger area than the pixel regions and being separated from each other by wiring regions. For example, each hole region may be adjacent to four other hole regions and four pixel regions.

[0069] The processor 1130 can perform image processing on the raw image based on blur information caused by the arrangement of the hole regions to generate an improved image. The processor 1130 can perform image preprocessing such as filtering on the raw image acquired by the camera 1120 and perform image restoration processing on the preprocessed image based on a neural network-based image restoration model. An improved image can be generated as a result of the image restoration processing.

[0070] Figure 12 shows the configuration of an electronic device according to one embodiment.

[0071] Referring to Figure 12, the electronic device 1200 may be a device including a display, such as a smartphone, tablet computer, wearable device, netbook, or laptop. The electronic device 1200 includes a processor 1210, memory 1220, image sensor 1230, storage device 1240, input device 1250, output device 1260, and communication device 1270. Each component of the electronic device 1200 can communicate with each other via a communication bus. The electronic device 1200 can perform all the functions of the image acquisition device 1000 shown in Figure 10.

[0072] The processor 1210 controls the overall operation of the electronic device 1200 and executes functions and instructions to be performed within the electronic device 1200. The processor 1210 may perform the operations described above with reference to Figures 1 to 11. The processor 1210 can perform image processing on the raw image acquired via the image sensor 1230 to generate an improved image. For example, the processor 1210 can perform image preprocessing and image restoration on the raw image based on blur information due to the arrangement of hole areas in the display layer to generate an improved image.

[0073] Memory 1220 stores information necessary for the processor 1210 to perform processing operations. For example, memory 1220 can store instructions to be executed by the processor 1210 and related information while software or programs are being executed in the electronic device 1200. Memory 1220 may include RAM, DRAM, SRAM, or other forms of non-volatile memory known in the art.

[0074] The image sensor 1230 is positioned below the display panel, which includes a display layer, and can detect external light passing through a hole area located in the display layer to generate a low image. The image sensor 1230 may also include a camera for receiving external light, and depending on the embodiment, image processing may be performed on the low image.

[0075] The storage device 1240 includes a computer-readable storage medium or computer-readable storage device that can store raw images and improved images. For example, the storage device 1240 may include a storage device, a magnetic hard disk, an optical disk, flash memory, an electrically programmable memory (EPROM), and the like.

[0076] The input device 1250 can receive input from the user via haptics, video, audio, or touch input. For example, the input device 1250 may include a keyboard, mouse, touchscreen, microphone, retinal scanner, or any other device that can detect input from the user and transmit the detected input to the electronic device 1200.

[0077] The output device 1260 can provide the user with the output of the electronic device 1200 via a visual, auditory, or tactile channel. The output device 1260 may include, for example, a display panel for a liquid crystal display or LED / OLED display, a touchscreen, a speaker, a vibration generator, or any other device that can provide output to the user. In the case of a display panel, it may include a display layer in which a hole area through which external light passes and a pixel area for outputting a display image are arranged.

[0078] The communication device 1270 can communicate with external devices via a wired or wireless network. The communication device 1270 can send and receive data and information with external devices.

[0079] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in this embodiment are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will understand that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Other processing configurations, such as a parallel processor, are also possible.

[0080] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by a processing unit or for providing instructions or data to a processing unit. Software can be distributed across a network of computer systems and stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0081] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0082] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0083] Therefore, other embodiments, other embodiments, and claims equivalent to those described below also fall within the scope of the claims. [Explanation of Symbols]

[0084] 110:Electronic equipment 120: Display 140, 210: Display Layer 150, 230: Pixel area 160, 240: Hall area 220: Image sensor 260: Protective layer

Claims

1. A display layer including a hole region through which external light passes and a pixel region positioned between the hole region, An image sensor is positioned below the display layer and detects external light that has passed through the hole region below the display layer to generate a low image. A processor that performs image processing on the raw image based on blur information resulting from the arrangement of the hole regions, Includes, The processor generates a preprocessed image by performing at least one of the following image preprocessing steps: applying a filter to the raw image and performing image deconvolution on the raw image. Image restoration is performed on the pre-processed image to generate an improved image with reduced blur compared to the raw image. The processor is an image acquisition device that acquires the improved image using a neural network-based image restoration model that takes the preprocessed image as input.

2. The image acquisition apparatus according to claim 1, wherein each of the hole regions is positioned between pixel regions in the display layer, and is positioned such that the minimum distance between the boundary of the hole region and the boundary of the pixel region is within 1 / 5 of the diameter of the hole region.

3. Each of the aforementioned hole regions has the shape of a single circle, The image acquisition apparatus according to claim 1 or 2, wherein the aforementioned hole regions are identical in shape and size to one another.

4. The image acquisition apparatus according to any one of claims 1 to 3, wherein each of the hole regions has a larger area than the pixel region and is separated from one another by a wiring region.

5. The image acquisition apparatus according to any one of claims 1 to 4, wherein the blur information includes information regarding the blur of the raw image determined by at least one of the morphology, size, depth, and spacing of the hole region.

6. The image acquisition apparatus according to any one of claims 1 to 5, wherein the blur information includes information regarding the point diffusion function due to the arrangement structure of the hall region.

7. A display layer including a pixel area and a circular hole area that transmits external light, An image sensor is positioned below the display layer and detects external light that has passed through the hole region below the display layer to generate an image. A processor that performs image processing on the aforementioned image, Includes, Each of the aforementioned hole regions is In the aforementioned display layer, it is positioned between pixel regions and adjacent to the pixel regions, In the aforementioned display layer, the hole region and the pixel region are arranged to intersect each other. The aforementioned hole region is in the form of a single circle, with the same shape and size as each other. The processor performs image processing on the image based on the blur information resulting from the arrangement of the hole regions. The processor generates a preprocessed image by performing at least one of the following image preprocessing: applying a filter to the image and performing image deconvolution on the image. An image restoration process is performed on the pre-processed image to generate an improved image with reduced blur compared to the original image. The processor is an image acquisition device that acquires the improved image using a neural network-based image restoration model that takes the preprocessed image as input.

8. The image acquisition apparatus according to claim 7, wherein adjacent hole regions are separated from each other by a wiring region for wiring, and the four hole regions are arranged to surround one pixel region.

9. A camera positioned below the display layer to detect external light passing through a hole region included in the display layer and generate a low image, the camera being configured to detect the external light passing through the hole region below the display layer, A processor that performs image processing on the raw image based on blur information resulting from the arrangement of the hole regions, Includes, The processor generates a preprocessed image by performing at least one of the following image preprocessing steps: applying a filter to the raw image and performing image deconvolution on the raw image. Image restoration is performed on the pre-processed image to generate an improved image with reduced blur compared to the raw image. The processor is an image sensor that acquires the improved image using a neural network-based image restoration model that takes the preprocessed image as input.

10. Each of the aforementioned hole regions is positioned between pixel regions in the display layer, such that the minimum distance between the boundary of the hole region and the boundary of the pixel region is within 1 / 5 of the diameter of the hole region. The image sensor according to claim 9, wherein each of the aforementioned hole regions has a larger area than the aforementioned pixel region and is separated from one another by a wiring region.

11. A display panel including a display layer in which a hole area through which external light passes and a pixel area for outputting a display image are arranged, An image sensor is positioned below the display panel and detects external light that has passed through the hole region below the display panel to generate a low image. A processor that generates an improved image by performing image processing on the raw image based on blur information resulting from the arrangement of the hole regions, A storage device that stores at least one of the raw image and the improved image, Includes, The processor generates a preprocessed image by performing at least one of the following image preprocessing steps: applying a filter to the raw image and performing image deconvolution on the raw image. Image restoration is performed on the pre-processed image to generate the improved image in which blur is reduced compared to the raw image. The processor is an electronic device that acquires the improved image using a neural network-based image restoration model that takes the preprocessed image as input.

12. Each of the aforementioned hole regions is positioned between pixel regions in the display layer, such that the minimum distance between the boundary of the hole region and the boundary of the pixel region is within 1 / 5 of the diameter of the hole region. The electronic device according to claim 11, wherein each of the hole regions has a larger area than the pixel region and is separated from one another by a wiring region.

13. In the operation method of an image acquisition device, A step of acquiring a low image by detecting external light that has passed through a hole region located in a display layer using an image sensor located below the display layer, wherein the image sensor is located below the display layer and detects external light that has passed through the hole region below the display layer. The steps include: performing image processing on the raw image based on blur information resulting from the arrangement of the hole regions; Includes, The step of performing the aforementioned image processing is: A step of generating a preprocessed image by performing at least one of the following image preprocessing steps: applying a filter to the raw image and performing image deconvolution on the raw image. The steps include: performing an image restoration process on the pre-processed image to generate an improved image with reduced blur compared to the raw image; Includes, The method of performing the image processing step includes the step of obtaining the improved image using a neural network-based image reconstruction model that takes the preprocessed image as input.

14. A computer-readable storage medium for storing instructions for performing the method described in claim 13.

Images