Image sensor, image processing system, and operating method thereof

Abstract

This disclosure relates to image sensors, image processing systems, and methods of operation thereof. The image sensor according to this disclosure includes a lens configured to receive light; a pre-operation actuator configured to generate position correction information based on a difference between first pixel values, the first pixel values ​​corresponding to pixels of a first microlens for light received through the lens in a first image; and a lens position controller configured to change the position of the lens for a second image based on the position correction information.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2022-0011454, filed with the Korean Intellectual Property Office on January 26, 2022, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The various embodiments generally relate to image processing systems, and more specifically to image sensors, image processing systems, and methods of operating image processing systems. Background Technology

[0004] An image sensor is a device that captures images using semiconductors that react to light. Developments in the computer and communications industries have led to an increased demand for image sensors used in a wide variety of products, such as smartphones, digital cameras, gaming devices, the Internet of Things (IoT), robots, surveillance cameras, and medical miniature cameras.

[0005] Image sensors can be broadly classified into charge-coupled device (CCD) image sensors and complementary metal-oxide-semiconductor (CMOS) image sensors. Compared to CMOS image sensors, CCD image sensors offer lower noise and better image quality. However, CMOS image sensors have a simpler driving method and can be implemented with various scanning methods. Furthermore, CMOS image sensors can be integrated with signal processing circuitry on a single chip, facilitating miniaturization, resulting in very low power consumption, and lower manufacturing costs due to the interchangeability of CMOS process technologies. Therefore, CMOS image sensing systems are increasingly used in mobile devices. Summary of the Invention

[0006] Various implementations relate to an image processing system and a method of operating the image processing system, which performs image sensing operations by changing the position of a lens through a preparatory operation to obtain the maximum difference between pixel values.

[0007] According to an embodiment, an image sensor may include a lens, a pre-operation actuator, and a lens position controller, wherein the lens is configured to receive light, the pre-operation actuator is configured to generate position correction information based on the difference between first pixel values ​​of light received through the lens in a first image, the first pixel values ​​corresponding to pixels of a first microlens, and the lens position controller is configured to change the position of the lens for a second image based on the position correction information.

[0008] According to one embodiment, an image processing system may include a lens, an image sensor, and an image processor. The lens may be configured to receive light and transmit the received light to a plurality of microlenses included in the image sensor.

[0009] The image sensor can be configured to generate position correction information about a test image received through the lens based on the phase difference between pixels corresponding to a first microlens among a plurality of microlenses. The image sensor can be configured to control the position of the lens used to capture the image based on the position correction information.

[0010] The image processor can be configured to perform resolution restoration operations based on the pixel values ​​of the captured image and output a high-resolution image with the restored resolution.

[0011] According to an embodiment, an operation method of an image processing system may include sensing a test image while performing lens oscillation; generating position correction information about the test image based on the difference between a first pixel value corresponding to a pixel of a first microlens in an image sensor; and changing the position of the lens based on the position correction information.

[0012] According to an embodiment, the operation method of the image processing system may include sensing a test image while performing lens oscillation. The method may include detecting a focus position and a correction position, wherein, at the focus position, a first pixel corresponding to a first microlens in the image sensor has the same phase; and at the correction position, the phase difference between the first pixels is maximum.

[0013] The method may further include generating position correction information about the test image, including information about the distance and direction of movement of the lens from the focus position to the correction position. The method may also include changing the position of the lens to a position corresponding to the position correction information. Attached Figure Description

[0014] Figure 1 This is a diagram illustrating an example of an image processing system according to an embodiment of the present disclosure;

[0015] Figure 2 This illustrates an embodiment according to the present disclosure. Figure 1 An example diagram of an image sensor;

[0016] Figure 3 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure;

[0017] Figure 4 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure;

[0018] Figure 5 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure;

[0019] Figure 6 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure;

[0020] Figure 7 This is a block diagram illustrating an example of an image processing system according to an embodiment of the present disclosure;

[0021] Figure 8 This is a diagram illustrating an example of an operation method for a preparatory operation according to an embodiment of the present disclosure;

[0022] Figure 9 This is a diagram illustrating an example of a method for detecting focus position and correcting position according to an embodiment of the present disclosure;

[0023] Figure 10 This is a diagram illustrating an example of a method for additionally moving a lens using a second microlens position for an image, according to an embodiment of the present disclosure;

[0024] Figure 11 This is a flowchart illustrating an example of image sensing operation according to an embodiment of the present disclosure;

[0025] Figure 12 This is a flowchart illustrating an example of a method for performing a preparatory operation according to an embodiment of the present disclosure;

[0026] Figure 13 This is a flowchart illustrating an example of a method for changing the position of a lens according to an embodiment of the present disclosure;

[0027] Figure 14 This is a flowchart illustrating an example of an additional method for changing the position of a lens according to an embodiment of this disclosure; and

[0028] Figure 15 This is a block diagram illustrating an example of an electronic device including an image sensor according to an embodiment of the present disclosure. Detailed Implementation

[0029] The descriptions of specific structures and / or functions disclosed in this specification are merely illustrative examples of embodiments based on the concepts provided in this disclosure. The examples of embodiments provided may be implemented in various forms, but this disclosure is not limited to the examples of embodiments described in this specification or their various forms.

[0030] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement the technical essence of the present disclosure.

[0031] Figure 1 This is a diagram illustrating an example of an image processing system 10 according to an embodiment of the present disclosure.

[0032] refer to Figure 1 The image processing system 10 may include an image sensor 100 and an image processor 200.

[0033] The image processing system 10 can acquire images. Furthermore, the image processing system 10 can store, display, or output the resulting image to an external device. The image processing system 10 can output the image to the host computer upon request.

[0034] According to the implementation, the image processing system 10 can be implemented as a packaged module or component, which may include hardware circuitry and / or executable instructions such as, for example, software instructions and / or firmware instructions. The image processing system 10 can be implemented as part of an electronic device or as a separate unit used in conjunction with an electronic device. For example, the electronic device may be a digital camera, mobile device, smartphone, personal computer (PC), tablet computer, laptop computer, personal digital assistant (PDA), enterprise digital assistant (EDA), portable multimedia player (PMP), wearable device, black box, robot, or autonomous vehicle, etc.

[0035] Electronic devices can also include, for example, imaging devices, portable camcorders, closed-circuit television (CCTV), webcams, security cameras, industrial vision cameras, in-vehicle vision cameras, set-top boxes, game consoles, electronic dictionaries, e-book readers, desktop computers, servers, MP3 players, smart medical devices, televisions, digital video disc (DVD) players, stereos, refrigerators, air conditioners, cleaners, ovens, microwave ovens, washing machines, air purifiers, smart mirrors, smart windows, electronic keys, electronic photo frames, digital billboards, and security control panels. Wearable devices can include smartwatches, rings, bracelets, anklets, necklaces, glasses, contact lenses, head-mounted displays (HMDs), skin pads, tattoos, or bio-implanted circuits.

[0036] Image sensor 100 can be implemented as, for example, a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, or any other type of image sensor. Image sensor 100 can generate images for transmission through lens 812 ( Figure 8 ) Input (or capture) object 811 ( Figure 8 Image data. For example, lens 812 can receive light emitted by the object and / or light reflected by the object 811. Lens 812 may include at least one lens forming an optical system. According to embodiments of this disclosure, lens 812 may be included in image sensor 100 or provided externally to image sensor 100.

[0037] Image sensor 100 may include multiple imaging elements for generating a representation of an object (a captured image). The imaging elements may use, for example, CCD technology, CMOS technology, or other forms of imaging technology. Image sensor 100 can generate multiple digital pixel values ​​(DPXs) corresponding to multiple pixels used in the captured image. The multiple digital pixel values ​​(DPXs) generated by image sensor 100 can be transmitted to image processor 200. That is, image sensor 100 can generate multiple digital pixel values ​​(DPXs) corresponding to a single frame of a photograph or video. For ease of description, the term "pixel" may refer to an imaging element or an element of the captured image.

[0038] The image processor 200 can process pixel data received from the image sensor 100 to generate processed image data with improved image quality. This processing can include electronic image stabilization (EIS), interpolation, tone correction, image quality correction, or resizing.

[0039] like Figure 1 As shown, the image processor 200 can be implemented independently of the image sensor 100. For example, the image sensor 100 and the image processor 200 can be implemented as separate chips in a single package (such as a multi-chip package) including a chip for the image sensor 100 and a chip for the image processor 200. In another embodiment of this disclosure, the image processor 200 and the image sensor 100 can be in a single chip. Various embodiments of this disclosure can also have each of the image sensor 100 and / or the image processor 200 as a multi-chip.

[0040] According to an embodiment, the image processing system 10 may further include a memory. The memory may be implemented as, for example, a non-volatile storage element. For example, the memory may be one of a variety of storage elements, such as a read-only memory (ROM) that can only read data, an one-time programmable (OTP) memory that can only be written once, an erasable programmable read-only memory (EPROM) that can erase and write stored data, NAND flash memory, and NOR flash memory.

[0041] Figure 2 This illustrates an embodiment according to the present disclosure. Figure 1 Image of image sensor 100.

[0042] refer to Figure 2 The image sensor 100 may include a pixel array 110, a line decoder 120, a timing generator 130, and a signal transducer 140.

[0043] According to an embodiment, the pixel array 110 may include a color filter array 111 and a photoelectric conversion layer 113. The photoelectric conversion layer 113 may be formed beneath, for example, the color filter array 111 and includes a plurality of photoelectric conversion elements corresponding to corresponding pixels of the color filter array 111. The pixel array 110 may include a plurality of pixels for outputting color information included in incident light. Each of the plurality of pixels may output a pixel signal corresponding to incident light passing through the corresponding color filter array 111.

[0044] The color filter array 111 may include color filters that allow only light of a specific wavelength (e.g., red, blue, or green) incident on each pixel to pass through. In this disclosure, the color filters may be represented as color channels. The color filter array 111 may allow pixel data for each pixel to represent a value corresponding to the intensity of light of a specific wavelength.

[0045] More specifically, each of the plurality of pixels can accumulate photocharge generated according to the incident light and can generate a pixel signal corresponding to the accumulated photocharge. Each pixel may include photoelectric conversion elements (e.g., photodiodes, phototransistors, photogates, and clamping photodiodes, etc.) to convert the optical signal into an electrical signal, and at least one transistor is provided to process the electrical signal.

[0046] Pixel array 110 may include a plurality of pixels arranged in both row and column directions. Pixel array 110 may generate a plurality of pixel signals VPXs for each row. The plurality of pixel signals VPXs may, for example, be analog pixel signals.

[0047] The line decoder 120 can select one of a plurality of rows in the pixel array 110 arranged with a plurality of pixels in response to the address and control signals output from the timing generator 130.

[0048] Signal converter 140 can convert multiple analog pixel signals VPXs into multiple digital pixel values ​​DPXs. The multiple digital pixel values ​​DPXs can be output in various modes. Signal converter 140 can perform, for example, correlated double sampling (CDS) on the signal output from pixel array 110 in response to a control signal output from timing generator 130, and can perform analog-to-digital conversion on the signal to which CDS is performed to output a digital signal. Each of the digital signals can correspond to the intensity of the wavelength of incident light passing through the corresponding color filter array 111.

[0049] Signal converter 140 may include a CDS block (not shown) and an analog-to-digital converter (ADC) block (not shown). The CDS block may sequentially sample and hold a set of reference signals and image signals provided to each of the plurality of column lines included in the pixel array 110. That is, the CDS block may sample and hold the level of the image signal and the level of the reference signal corresponding to each column. The ADC block may output pixel data, which is obtained by converting the CDS signals output from the CDS block for each column into digital signals. To output pixel data, the ADC block may include a counter and a comparator corresponding to each column.

[0050] Additionally, the image sensor 100 according to an embodiment of this disclosure may further include an output buffer 150. The output buffer 150 may be implemented as a plurality of buffers storing the digital signal output from the signal converter 140. More specifically, the output buffer 150 may latch and output each column of pixel data provided from the signal converter 140. The output buffer 150 may temporarily store the pixel data output from the signal converter 140 and may output pixels sequentially. According to another embodiment of this disclosure, the image sensor 100 may not provide an output buffer 150. Therefore, in one embodiment, the image sensor 100 may directly output the digital pixel value DPX from the signal converter 140 without using the output buffer 150.

[0051] Figure 3 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure.

[0052] refer to Figure 3 The image shows an example where one microlens corresponds to four adjacent pixels used to detect the same color. (Reference) Figure 3Four green pixels can correspond to one microlens. Four adjacent red pixels or four adjacent blue pixels can also correspond to one microlens.

[0053] like Figure 3 As shown, sixteen pixels can correspond to four microlenses. A microlens can correspond to a group of four pixels configured to detect the same color. For example, when a red pixel corresponds to a microlens, light from that microlens can be incident on the red pixel, and the pixel value of the red pixel can be processed to obtain phase information.

[0054] Four blue pixels or four green pixels can also correspond to a microlens. In the same way, four blue pixels or four green pixels can also be processed to obtain phase information.

[0055] However, the embodiments of this disclosure are not limited to the example where one microlens corresponds to four pixels. That is, the number of pixels corresponding to one microlens can vary. Therefore, a microlens can be referred to as corresponding to a group of pixels that detect the same color. Pixels that detect color can be referred to as pixels that have the same color, or simply pixels with the same color.

[0056] Therefore, the image sensor 100 according to the embodiments of the present disclosure can generate pixel values ​​for all pixels, and the pixel values ​​can be used to obtain phase information about the pixels.

[0057] Figure 4 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure.

[0058] refer to Figure 4 Nine adjacent pixels of the same color can correspond to a microlens. Each of the pixel values ​​of the pixels corresponding to a microlens can be used to determine phase information about those pixels.

[0059] According to this disclosure, an image sensor comprising a pixel array of nine pixels corresponding to a microlens can be used. According to embodiments of this disclosure, the number of pixels corresponding to a microlens is not limited to a specific number. The number of pixels in the microlens can be designed based on requirements such as, for example, image sharpness.

[0060] Figure 5 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure.

[0061] like Figure 5 As shown, sixty-four pixels correspond to four microlenses, with each microlens corresponding to sixteen pixels. Pixels corresponding to a microlens can be adjacent to each other and have the same color.

[0062] Figure 3 , Figure 4 as well as Figure 5 This is just an example, and the number of pixels corresponding to a microlens is not limited to this. Pixels corresponding to microlenses are grouped together and can be referred to as adjacent to each other (or neighboring pixels), although two pixels in a microlens may not be directly adjacent to each other. For example, in Figure 4 In this model, although the nine pixels of a microlens can be considered adjacent pixels, the pixels in the top row are not directly adjacent to the pixels in the bottom row. Additionally, the pixels corresponding to a microlens can be referred to as the microlens pixels.

[0063] Figure 6 This is a diagram illustrating an example of the arrangement of a pixel array and microlenses according to an embodiment of the present disclosure.

[0064] refer to Figure 6 One microlens can correspond to four pixels. Additionally, in Figure 6 In this system, four adjacent microlenses can all correspond to pixels with the same color.

[0065] According to the implementation method, pixels corresponding to a microlens can have the same color. Furthermore, as... Figure 6 As shown, four adjacent microlenses can all have pixels of the same color.

[0066] According to the implementation method ( Figure 8 As shown, the pixel array can be located below the microlens (e.g., 813). However, various embodiments of this disclosure are not limited thereto. One or more objects may exist between the microlens 813 and the pixel array.

[0067] Figure 7 This is a block diagram illustrating an example of an image processing system according to an embodiment of the present disclosure. Figure 7 An example of an image processing system including components is shown, and it can be shown that... Figure 1 An embodiment of the image processing system 10 shown in the figure.

[0068] refer to Figure 7 The image sensor 710 can generate image data, and the image processor 720 can receive the image data and generate an output image. The image sensor 710 may include a lens 711, a pre-operation actuator 712, a lens position controller 713, and a storage device 714.

[0069] Lens 711 can collect light received from the outside of image sensor 710. According to embodiments of this disclosure, lens 711 can be an optical lens. Lens 711 can transmit the received light to a plurality of microlenses included in image sensor 710.

[0070] like Figure 7 As shown, in one embodiment, the image sensor 710 may include a lens 711. However, in another embodiment, the lens 711 may be located outside the image sensor 710.

[0071] The pre-operation actuator 712 can generate position correction information based on the difference between first pixel values ​​among a plurality of pixel values ​​of a first image received through the lens 711, the first pixel values ​​corresponding to a first microlens. According to an embodiment, the first microlens may be one of a plurality of microlenses included in the image sensor 710. The pre-operation actuator 712 can generate position correction information based on the first pixel values ​​of a plurality of pixels corresponding to microcells, having the same color, and being adjacent to each other.

[0072] According to an implementation, the first image may be, for example, a pre-selected test image. The test image may be provided to test the image sensor 710.

[0073] When, for example, the first pixel values ​​of the micro-units used for the first image have the same value, the pre-operation actuator 712 can detect the focus position of the lens 711. When, for example, the difference between the first pixel values ​​of the first image is the largest, the pre-operation actuator 712 can detect the correction position of the lens 711.

[0074] According to another embodiment, the pre-operation actuator 712 can generate position correction information about a test image received through the lens 711 based on the phase difference between pixels of the first microlens among a plurality of microlenses. When, for example, the first pixels of the test image have the same phase, the pre-operation actuator 712 can detect the focus position of the lens 711. When, for example, the phase difference between the first pixels of the test image is maximum, the pre-operation actuator 712 can detect the correction position of the lens 711.

[0075] The pre-operation actuator 712 can generate position correction information about the first image, including movement information of the lens 711 from the focus position to the correction position. According to an embodiment of this disclosure, the pre-operation actuator 712 can detect the correction position based on the difference between the maximum and minimum values ​​in the first pixel values. According to another embodiment, the pre-operation actuator 712 can detect the correction position based on the pixel values ​​of pixels located diagonally in the first pixel values.

[0076] The lens position controller 713 can change the position of the lens 711 with respect to the second image based on position correction information. The lens position controller 713 can also move the lens 711 from the focus position of the second image to a position corresponding to the movement information based on the position correction information with respect to the first image.

[0077] According to an embodiment, the second image can be a captured image. The captured image can be an actual image of the object sensed or captured by the image sensor 710. The lens position controller 713 can move the lens 711 to the focus position of the second image. The lens position controller 713 can move the position of the lens 711 to a position corresponding to the position correction information to obtain a recoverable image with high resolution.

[0078] According to embodiments of this disclosure, the pre-operation actuator 712 can generate edge correction information based on a second pixel value corresponding to a second microlens among a plurality of microlenses, the second microlens being configured to receive light from the edge of the first image. This can also be expressed as having a second microlens located at the edge of the first image. The lens position controller 713 can additionally move the lens 711, which has been moved to a position corresponding to the movement information, based on the edge correction information.

[0079] According to an embodiment, the operation of moving the lens 711 can be referred to as "performing a lens wobbling" or "lens wobbling operation." According to an embodiment, when performing a lens wobbling operation on the lens 711, the image sensor 710 can sense or capture a test image. The lens 711 can move, for example, within a predetermined range. The lens 711 can be a lens wobbled to a position other than the focus position. The image sensor 710 can obtain pixel values ​​by changing the position of the lens 711. The pixel values ​​of the image sensor 710 can change in response to the movement of the lens 711. The image sensor 710 can detect the focus position and the correction position based on the pixel values ​​obtained by performing a lens wobbling operation on the lens 711.

[0080] Image sensor 710 can store position correction information of lens 711 in storage device 714. Storage device 714 may include non-volatile and / or volatile storage devices. Image sensor 710 can update the position correction information stored in storage device 714. Image sensor 710 can update the position correction information based on the results of performing image sensing / capture operations and recovery operations of captured images.

[0081] Image processor 720 can perform resolution restoration operations based on the pixel values ​​of the captured image. Image processor 720 can output a high-resolution image with restored resolution. According to one embodiment, when the position of lens 711 corresponds to the focus position, there may be no phase difference between pixels, which may make it difficult to perform high-resolution image restoration operations. When lens 711 moves to a position corresponding to position correction information about the test image, the difference between pixel values ​​obtained from image sensor 710 can be maximized, thus enabling image processor 720 to perform high-resolution image restoration operations.

[0082] Figure 8 This is a diagram illustrating an example of an operation method for a preparatory operation according to an embodiment of the present disclosure.

[0083] refer to Figure 8 This shows the focus position 810 and the correction position 820 of lens 812 with respect to the test image. Figure 8 In this process, when the lens position corresponds to the focus position 810, high-resolution image restoration may be difficult to perform due to small or no phase difference between pixels. Image restoration may not be performed, for example, when the phase difference between pixels is less than a threshold. In various embodiments, the threshold may be a predetermined default value, which may be fixed. In various embodiments, the threshold may be a predetermined default value, which may be adjustable.

[0084] According to the implementation, the high-resolution image restoration operation can utilize the phase difference between pixels. Since there are multiple pixels corresponding to a microlens, pixels at the focus position 810 can have the same phase. Therefore, the image sensor can detect the correction position where the phase difference between pixels is maximum by performing lens wobbling. In other words, the image sensor 710 can detect the correction position where the phase difference between pixels is maximum based on pixel values ​​obtained by moving the lens within a predetermined range (performing lens wobbling). The phase difference can be, for example, the phase difference between two pixels.

[0085] Regarding the focus position 810, light reflected from the object 811 can enter the lens 812. The lens 812 can transmit the light to the microlens 813. The light can then enter the pixel 814 in the pixel array.

[0086] At focus position 810, different images of object 811 can be combined at pixel 814. Therefore, it may be difficult to perform a restoration operation on object 811 based on the pixel values ​​of pixel 814.

[0087] According to one embodiment, the image sensor 710 can sense a test image by performing lens oscillation. At the calibration position 820, light reflected from the object 821 can pass through the lens 812 and be incident on the microlens 813. The light can then be incident on the pixels 814 in the pixel array.

[0088] At the correction position 820, different images of object 821 can be separated at pixel 814. Therefore, since pixel 814 can have different pixel values, image processor 720 can be able to perform high-resolution image restoration operations.

[0089] According to the embodiment, when performing lens oscillation, the image sensor 710 can detect the focus position 810 and the correction position 820. The position 810 of the lens 812 can be the focus position 810 when the pixel values ​​corresponding to the microlens 813 are the same. The position 820 of the lens 812 can be the correction position when the difference between the pixel values ​​corresponding to the microlens 813 is at its maximum.

[0090] According to one embodiment, the image sensor 710 can generate position correction information indicating the positional difference of the lens 812 corresponding to the focus position 810 and the correction position 820. The position correction information may include movement information of the lens 812 from the focus position 810 to the correction position 820. According to another embodiment, the position correction information may include information about the distance and direction of movement of the lens from the focus position 810 to the correction position 820.

[0091] According to another embodiment, the image sensor 710 can sense a test image by tilting the microlens 813 and the position of the corresponding pixel 814 without performing lens tilting. The image sensor 710 can generate position correction information indicating changes in the positions of the microlens 813 and the pixel 814. This may include moving the microlens 813 and the pixel 814 instead of moving the lens 812.

[0092] According to another embodiment, the image sensor 710 can sense a test image while the positions of the pixels 814 and the lens 812 are swung. The image sensor 710 can then generate position correction information indicating changes in the positions of the lens 812 and the pixels 814. This may include moving the lens 812 and the pixels 814 without moving the microlens 813. Typically, any one or more of the lens 812, the microlens 813, or the pixels 814 can be moved to determine the correction position.

[0093] Figure 9 This is a diagram illustrating an example of a method for detecting focus position and correcting position according to an embodiment of the present disclosure.

[0094] refer to Figure 9One microlens can correspond to four pixels 910. The four pixels 910 can have the same color and be adjacent to each other. Figure 9 In the process, the pre-operation actuator 712 can generate position correction information based on the pixel values ​​of green pixels G11, G12, G21 and G22.

[0095] The pre-operation actuator 712 can calculate the difference between the maximum and minimum values ​​among the four pixels 910. According to an embodiment, the pre-operation actuator 712 can perform lens wobbling of the lens 711 to detect the correction position of the lens 711 when the difference between the maximum and minimum values ​​among the four pixels 910 is greatest. For example, lens wobbling may result in the discovery that green pixel G11 can have a maximum value and green pixel G21 can have a minimum value among the four pixels 910. Therefore, the pre-operation actuator 712 can detect the position of the lens 711 when the difference between G11 and G21 is greatest as the correction position.

[0096] According to another embodiment, the pre-operation actuator 712 can calculate the pixel value difference between pixels located diagonally across the four pixels 910. The pre-operation actuator 712 can calculate the pixel value difference between G11 and G22 and the pixel value difference between G12 and G21 by tilting the lens 711. The pre-operation actuator 712 can calculate the average value between the two pixel value differences. The pre-operation actuator 712 can detect the correction position of the lens 711 based on the pixel values ​​of the diagonally located pixels in the pixel corresponding to a microlens.

[0097] Figure 9 Only one microlens corresponding to four pixels 910 is shown. However, this disclosure is not limited to this. The microlens may, for example, include nine pixels, sixteen pixels, or another number of pixels.

[0098] Figure 10 This is a diagram illustrating an example of a method for additionally moving the camera based on the position in the image.

[0099] refer to Figure 10 The position correction information corresponding to the center 1010 of the image can be different from the position correction information corresponding to the edge 1020 of the image. Since the lens can have a circular shape, the lens 711 can move more based on the position correction information corresponding to the edge 1020 of the image compared to using the position correction information corresponding to the center 1010 of the image.

[0100] According to embodiments of this disclosure, the pre-operation actuator 712 can generate edge correction information based on a second pixel value corresponding to a second microlens located at the edge of the first image among a plurality of microlenses. The lens position controller 713 can additionally move the lens 711, which has been moved to a position corresponding to the movement information, based on the edge correction information.

[0101] For example, when performing an image sensing operation based on position correction information corresponding to the first image, the position of the lens 711 can change, and the first image can be a test image. The position correction information can correspond to the center 1010 of the image or the edge 1020 of the image. When performing an image sensing operation based on the position correction information corresponding to the center 1010 of the image, the pixel value difference between pixels at the edge 1020 of the image may not be the maximum.

[0102] According to the embodiment, the pre-operation actuator 712 can generate edge correction information, which may be position correction information corresponding to the edge 1020 of the test image. The lens position controller 713 can further move the position of the lens 711 to be changed based on the position correction information, for example, based on the edge correction information.

[0103] According to another embodiment, the pre-operation actuator 712 can store position correction information corresponding to different microlenses in, for example, a storage device 714. The stored position correction information can correspond to the center 1010 or the edge 1020 of the test image. The pre-operation actuator 712 can change the position of the lens 711 from the focus position during image sensing operation by using, for example, the average of multiple segments of stored position correction information.

[0104] Figure 11 This is a flowchart illustrating an example of image sensing operation according to an embodiment of the present disclosure.

[0105] refer to Figure 11 , similar to, for example Figure 7 The image processing system 710 can generate position correction information by sensing a test image, and obtain pixel values ​​by changing the position of lens 711 during image sensing operations. The image processing system 710 can use the obtained pixel values ​​to perform high-resolution image restoration operations.

[0106] In step S1110, the image sensor 710 can sense the test image while performing lens oscillation. The image sensor 710 can sense the test image by changing the position of the lens 711. When the position of the lens 711 changes, the focus of the light collected by the pixel array of the image sensor 710 can change.

[0107] In step S1120, the pre-operation actuator 712 may perform a lens oscillation operation to generate position correction information about the test image. This information can be used to determine the maximum difference in value of a first pixel corresponding to the first microlens among the plurality of microlenses included in the image sensor.

[0108] Therefore, the pre-operation actuator 712 can generate position correction information based on the difference between first pixel values, where the first pixel value corresponds to a first microlens among a plurality of pixel values ​​for the first image received by the lens 711. According to an embodiment, the first microlens may be one of a plurality of microlenses included in the image sensor 710. The pre-operation actuator 712 can generate position correction information based on the first pixel values ​​of a plurality of pixels having the same color and being adjacent to each other.

[0109] According to an implementation, the first image may be a test image. The test image may be provided to test the image sensor 710.

[0110] In step S1130, the lens position controller 713 can change the position of the lens 711 based on position correction information during the image sensing operation. When performing the image sensing operation, the lens 711 can move to a focus position with respect to the captured image. The resolution of the pixel values ​​of the image obtained at the focus position can be reduced during the image restoration operation. The lens position controller 713 can change the position of the lens 711 based on position correction information generated by sensing a test image. The lens position controller 713 can move the lens 711 to a position corresponding to the position correction information of the test image. Light can be collected through the lens 711, which has been repositioned, and transmitted to pixels (such as, for example, pixel 814). Since the pixel values ​​of the image generated by the image sensor 710 have different phases, high resolution can be maintained during the image restoration operation.

[0111] In step S1140, the image processor 720 can restore the resolution of the captured image based on the pixel values ​​of the captured image and can output the restored image. The image processor 720 can perform a resolution restoration operation based on the pixel values ​​of the captured image. The image processor 720 can output a high-resolution image with the restored resolution.

[0112] According to the implementation method, since the pixel values ​​of all pixels include phase information, the resolution of the sensed image can be maintained even when the resolution restoration operation is performed.

[0113] Figure 12 This is a flowchart illustrating an example of a method for performing preparatory operations according to an embodiment of the present disclosure.

[0114] refer to Figure 12The pre-operation actuator 712 can generate position correction information for the lens 711 by sensing the test image.

[0115] In step S1210, the pre-operation actuator 712 can detect the focus position of the lens, where first pixel values ​​of adjacent pixels with the same color in the test image have the same value. According to another embodiment, the pre-operation actuator 712 can generate position correction information about the test image received through the lens 711 based on the phase difference between pixels corresponding to the first microlens in the plurality of microlenses. The pre-operation actuator 712 can detect the focus position of the lens 711, where first pixels of the test image have the same phase.

[0116] In step S1220, the pre-operation actuator can detect a correction position, which is the position of the lens when the difference between the first pixel values ​​of the test image is maximum. According to another embodiment, the pre-operation actuator can detect a correction position, which is the position of the lens when the phase difference between the first pixels of the test image is maximum.

[0117] The pre-operation actuator 712 can generate position correction information about the first image, including movement information of the lens 711 that has moved from the focus position to the correction position. According to one embodiment, the pre-operation actuator 712 can detect the correction position based on the difference between the maximum and minimum values ​​in the first pixel values. According to another embodiment, the pre-operation actuator 712 can detect the correction position based on the pixel values ​​of pixels located diagonally in the first pixel values.

[0118] In step S1230, the pre-operation actuator 712 can detect movement information of the lens 711, which has moved from the focusing position to the correction position. According to another embodiment, the position correction information may include information about the distance and direction of movement of the lens 711, which has moved from the focusing position to the correction position.

[0119] Figure 13 This is a flowchart illustrating an example of a method for changing the position of a lens according to an embodiment of the present disclosure.

[0120] refer to Figure 13 During image sensing operation, the lens position controller 713 can change the position of the lens 711 from the focus position.

[0121] In step S1310, the lens position controller 713 can move the lens 711 to a focus position. The lens position controller 713 can move the position of the lens 711 to a focus position where the pixels have the same phase by using pixel values ​​obtained by sensing the image.

[0122] In step S1320, the lens position controller 713 can move the lens 711 from a focus position with respect to the captured image to a position corresponding to the movement information. During the image sensing operation, the lens position controller 713 can change the position of the lens 711 based on position correction information about the test image.

[0123] For example, the position of lens 711 can be changed as much during image sensing operations as it changes from the focus position to the correction position by sensing a test image.

[0124] In step S1330, the image sensor 710 can perform an image sensing operation. The difference between the pixel values ​​obtained by performing the image sensing operation can be maximized.

[0125] Figure 14 This is a flowchart illustrating an example of an additional method for changing the position of a lens according to an embodiment of this disclosure.

[0126] refer to Figure 14 During image sensing operation, the lens position controller 713 can additionally control the position of the lens 711.

[0127] In step S1410, the pre-operation actuator 712 can generate edge correction information based on the pixel values ​​corresponding to the second microlens among the plurality of microlenses that can be located at the edge of the test image.

[0128] Since the lens 711 can have a circular shape, it can move further based on positional correction information corresponding to the edges of the image, compared to correction information corresponding to the center of the image. When performing image sensing operations based on positional correction information corresponding to the center of the image, the pixel value difference between pixels at the edges of the image may not be the largest.

[0129] In step S1420, the lens position controller 713 may additionally move the lens 711, which has been moved to a position corresponding to the movement information, based on the edge correction information.

[0130] According to the implementation, the pre-operation actuator 712 can generate edge correction information, which is position correction information corresponding to the edges of the test image. The lens position controller 713 can additionally move the position of the lens 711 to be changed based on the position correction information based on the edge correction information.

[0131] According to another embodiment, the pre-operation actuator 712 can store position correction information corresponding to different microlenses in a storage device 714. The stored position correction information can correspond to the center and / or edge of the test image. The pre-operation actuator 712 can change the position of the lens 711 from the focus position during image sensing operation by using the average value of multiple segments of stored position correction information.

[0132] In step S1430, the image sensor 710 can perform an image sensing operation. The position correction information corresponding to the edges of the test image and the position correction information corresponding to the center of the test image can be different from each other. During the image sensing operation, the lens position controller 713 can additionally change the position of the lens 711 based on the difference between the position correction information regarding the center and edges of the test image.

[0133] Figure 15 This is a block diagram illustrating an example of an electronic device 2000 including an image sensor 2010 according to an embodiment of the present disclosure.

[0134] refer to Figure 15 The electronic device 2000 includes an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input device 2050, and an output device 2060. Although in Figure 15 As not shown, the electronic device 2000 may also include a port that can communicate with a graphics card, sound card, memory card, or USB device, or a port that can communicate with other electronic devices.

[0135] Image sensor 2010 can generate image data corresponding to incident light. Output device 2060 can display the image data. Storage device 2030 can store the image data. Processor 2020 can control the operation of at least image sensor 2010, output device 2060, and storage device 2030.

[0136] Processor 2020 can execute instructions to perform certain calculations or tasks. According to embodiments of this disclosure, processor 2020 may be a microprocessor, a central processing unit (CPU), or a controller, etc. Processor 2020 can communicate with storage device 2030, storage device 2040, and input device 2050 via address bus, control bus, and / or data bus to perform communication. According to embodiments of this disclosure, processor 2020 may also be connected to an expansion bus such as an external component interconnect (PCI) bus.

[0137] Storage device 2030 can be a non-volatile storage device, such as, for example, flash storage devices, solid-state drives (SSDs), hard disk drives (HDDs), and CD-ROMs.

[0138] Storage device 2040 can store data necessary for the operation of electronic device 2000. For example, storage device 2040 may include volatile storage devices, such as dynamic random access memory (DRAM) and static random access memory (SRAM); and non-volatile storage devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash storage devices.

[0139] Input device 2050 may include input devices such as, for example, a keyboard, a keypad, a mouse, etc. Output device 2060 may include output devices such as, for example, a printer, and a monitor, etc.

[0140] The image sensor 2010 can be connected to the processor 2020 via a bus or other communication link.

[0141] Image sensor 2010 can be implemented in various types of packages. For example, at least some configurations of image sensor 2010 can be implemented using packages such as: stacked package (PoP), ball grid array (BGA), chip-scale package (CSP), plastic leaded chip carrier (PLCC), molded dual in-line package (PDIP), die in Wozniak assembly, die in wafer form, chip on board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline integrated circuit (SOIC), shrink small outline package (SSOP), thin small outline package (TSOP), system-in-package (SIP), multi-chip package (MCP), wafer-level fabrication package (WFP), and wafer-level processing stacked package (WSP), etc.

[0142] According to the implementation, the image sensor 2010 can be integrated with the processor 2020 in a single chip, or the image sensor 2010 and the processor 2020 can be different chips.

[0143] Electronic device 2000 may be, for example, a computing system using image sensor 2010. For example, electronic device 2000 may be a digital camera, mobile phone, personal digital assistant (PDA), portable multimedia player (PMP), and smartphone, etc.

[0144] According to the implementation, the image sensor 2010 can perform preparatory operations to generate position correction information, and the position of the lens of the image sensor 2010 can be controlled during the image sensing operation. Therefore, the processor 2020 can generate a clear, high-resolution image based on the sensed image. The storage device 2030 can store the generated high-resolution image, and the output device 2060 can output the high-resolution image via an output device such as a printer or a display.

[0145] According to this disclosure, an example of an image processing system is described that obtains a recoverable image with high resolution by moving the lens to the position where the phase difference between pixels is greatest.

[0146] Examples of various embodiments are described in this disclosure for illustrative and descriptive purposes. However, this disclosure is not limited to the examples of embodiments. Rather, it will be apparent to those skilled in the art, in consideration of this disclosure, that many other embodiments are possible.

Claims

1. An image sensor, comprising: A lens is a device that receives light. A pre-operation actuator generates position correction information based on the difference between first pixel values ​​of light received through the lens in a first image, the first pixel values ​​corresponding to pixels of the first microlens; as well as A lens position controller that changes the position of the lens used for the second image based on the position correction information. The pre-operation actuator detects a focus position and a correction position. The focus position corresponds to the position of the lens when the first pixel values ​​of the first image have the same value, and the correction position corresponds to the position of the lens when there is a maximum difference between the first pixel values ​​of the first image.

2. The image sensor according to claim 1, wherein, The first microlens is one of a plurality of microlenses in the image sensor, and The first microlens includes the ability to detect adjacent pixels of the same color.

3. The image sensor according to claim 1, wherein, The pre-operation actuator detects the correction position based on the difference between the maximum and minimum values ​​in the first pixel value.

4. The image sensor according to claim 1, wherein, The pre-operation actuator detects the correction position based on the pixel values ​​of pixels located in the diagonal direction.

5. The image sensor according to claim 1, wherein, The pre-operation actuator generates position correction information about the first image, the position correction information including the movement information of the lens from the focus position to the correction position.

6. The image sensor according to claim 5, wherein, The lens position controller moves the lens from the focus position of the second image to the correction position based on the movement information.

7. The image sensor according to claim 6, wherein, The pre-operation actuator generates position correction information, including edge correction information, based on the second pixel value corresponding to the second microlens located at the edge of the first image. The second microlens includes the ability to detect adjacent pixels of the same color, and The lens position controller moves the lens based on the position correction information.

8. An image processing system, comprising: A lens, which receives light and transmits the received light to multiple microlenses included in an image sensor. The image sensor: generates position correction information about a test image received through the lens based on the phase difference between pixels corresponding to the first microlens of the plurality of microlenses; and controls the position of the lens for capturing the image based on the position correction information; and An image processor that performs a resolution restoration operation based on pixel values ​​of the captured image and outputs a high-resolution image with the restored resolution.

9. The image processing system according to claim 8, wherein, The first microlens includes the ability to detect adjacent pixels of the same color, and The image sensor includes a pre-operation actuator that detects a focus position and a correction position. The focus position corresponds to the position of the lens when the phase of the pixels in the test image has the same value, and the correction position corresponds to the position of the lens when the phase difference between the pixels in the test image is the maximum.

10. The image processing system according to claim 9, wherein, The pre-operation actuator generates position correction information about the test image, the position correction information including the movement information of the lens from the focus position to the correction position, and The image sensor includes a lens position controller that moves the lens from the focus position of the captured image to a corrected position corresponding to the movement information based on the position correction information about the test image.

11. The image processing system according to claim 10, wherein, The pre-operation actuator generates edge correction information based on the phase value of a pixel corresponding to a second microlens among the plurality of microlenses, the second microlens being located at the edge of the test image. The lens position controller moves the lens based on the edge correction information.

12. A method for operating an image processing system, the method comprising: The test image is sensed while the lens is being swung. Based on the difference between the first pixel value corresponding to the pixel of the first microlens in the image sensor, position correction information about the test image is generated; and The position of the lens is changed based on the position correction information. The generation of the position correction information includes detecting the focus position and the correction position. The focus position corresponds to the position of the lens when the first pixel values ​​have the same value. The first microlens includes adjacent pixels with the same color. The correction position corresponds to the position of the lens when the first pixel values ​​have the maximum difference.

13. The method according to claim 12, wherein, The position correction information includes the movement information of the lens from the focus position to the correction position.

14. The method of claim 13, wherein, Changing the position of the lens includes: Move the lens to the focus position relative to the captured image; and The lens is moved from the focus position of the captured image to a position corresponding to the movement information.

15. The method according to claim 14, wherein, Generating the position correction information includes: generating edge correction information based on a second pixel value corresponding to a second microlens located at the edge of the test image, and Changing the position of the lens includes: additionally moving the lens to a position corresponding to the movement information based on the edge correction information.

16. The method of claim 14, comprising: Perform a resolution restoration operation based on the pixel values ​​of the captured image; as well as Outputs a high-resolution image with restored resolution.

17. A method for operating an image processing system, the method comprising: The test image is sensed while the lens is being swung. The focus position and the correction position are detected, wherein at the focus position, the first pixel corresponding to the first microlens in the image sensor has the same phase; and at the correction position, the phase difference between the first pixels is the largest. Generate position correction information about the test image, the position correction information including information about the distance and direction of movement of the lens from the focus position to the correction position; and The position of the lens is changed to a position corresponding to the position correction information.

18. The method of claim 17, comprising: The pixel values ​​of the captured image are generated by the lens whose position has been changed; Perform a resolution restoration operation based on the pixel values ​​of the captured image; as well as Outputs a high-resolution image with restored resolution.

Images